Wednesday, 2020-04-01

mordredyeah - there's a split of raised capabilities - it gets to behave like a config project for the purposes of config, but not for the purposes of execution00:00
corvusclarkb: it's basically revisiting every "if project.source_context.trusted" in the code and turning it from a binary into a ternary :)00:00
clarkbah00:00
corvusprobably by making it some sort of generic capabilities system, so when we go from 3->4 it's not as difficult as it will be to go from 2->3 :)00:01
clarkbfwiw I'm ok with mordred's intermediate solution00:01
fungilooks like we've had another spate of oom events on lists.o.o00:01
corvusi am as well00:01
mnaseri'm ok with the expectation that it won't be anyone other than opendev00:01
fungi11 python processes killed since sunday00:01
fungii'm going to restart all the queue managers on lists.o.o00:01
corvusmordred's solution should be forward-compatible with a future tenant-config-project feature00:01
corvusfungi: thanks00:01
mnaseryou'll have a canary repo ready for that :p00:02
clarkbfungi: fwiw I wasn't really able to track it back to any specific message processing the last time I tried00:02
mnaseri'll throw up a change which modifies the acls and i guess discussion can happen there, ill summarize the idea in the commit in case someone goes back and wonders why00:02
clarkbbut its still my hunch that one of the mailman pipelines is being unhappy about particular messages (and possibly that is an intentional dos)00:02
corvusmnaser: ++00:03
clarkbmnaser: ++00:03
fungi#status log restarted queue managers for all 5 mailman sites on lists.o.o following a spate of oom conditions00:04
openstackstatusfungi: finished logging00:04
fungione thing i noticed with our etherpad puppetry... the content of /etc/apt/sources.list.d/nodesource.list doesn't seem to get updated automatically unless you blow away that file00:09
fungii guess the nodejs puppet module takes care of that, and it seems to be something we're reconsuming from puppetforge, does that sound right?00:10
clarkbfungi: yes that sounds right from memory00:10
openstackgerritMohammed Naser proposed openstack/project-config master: vexxhost: move base-jobs to config-project  https://review.opendev.org/71645900:11
mnaserclarkb, corvus, mordred: ^ did my best to summarize, moved that repo to opendev/project-config acls and added it as a config project with only jobs/secret/nodesets00:11
fungiMar 31 23:45:11 etherpad-dev01 puppet-user[28496]: (/Stage[main]/Nodejs::Repo::Nodesource::Apt/Apt::Source[nodesource]/Apt::Setting[list-nodesource]/File[/etc/apt/sources.list.d/nodesource.list]/ensure)00:11
fungiokay, so not fixing that behavior unless we fork it (or maybe there's a newer version on the forge)00:11
fungii guess we can just live with that until the containerized version is done00:12
fungihttps://etherpad-dev.openstack.org/ is back up and running again00:14
openstackgerritIan Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel  https://review.opendev.org/71644900:14
clarkbfungi: and its only necessary during the infrequent cases of updating major nodejs versions right?00:14
fungii guess00:15
ianwfungi: ReferenceError: io is not defined in https://etherpad-dev.openstack.org/javascripts/lib/ep_etherpad-lite/static/js/pad.js?callback=require.define at line 3'00:15
funginew problem:00:15
ianwfungi: i am supposed to report that to my webmaster00:15
fungiianw: beat me to it00:15
clarkbmy guess is that is related to the npm install of etherpad itself00:15
clarkb(that should pull in all the deps and build js things appropriately)00:15
clarkbperhaps we need to retrigger that with the up to date nodejs00:15
fungimaybe it needs to rerun with new node in place, yeah00:15
clarkb(assuming nodejs 6 failed to build that properly in the past)00:16
fungii did have to manually apt install nodejs to get the package to upgrade too00:16
fungii'll wind the etherpad git repo back by one commit and see what puppet does on the next pulse00:16
clarkb++00:16
fungidid a `git reset --hard HEAD^1` in it00:17
funginow behind origin/devel by 1 commit and can be fast-forwarded00:18
*** dangtrinhnt has joined #opendev00:21
openstackgerritIan Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel  https://review.opendev.org/71644900:25
*** dangtrinhnt has quit IRC00:37
mnaserok -- i'm running out of options, i'm trying to write a new job that runs a popular golang linter (golangci-lint): https://review.opendev.org/#/c/716452/00:42
mnaseri've literally repeated the same exact playbooks locally and it runs just fine, it gives: [Errno 8] Exec format error: 'golangci-lint' -- in ci/infra nodes00:42
mnaseri tried running it with verbose but that hasn't really yielded to much more success, and it runs fine in a vm too so i'm pretty confused. could i get a hold on https://review.opendev.org/#/c/716453/ for golangci-lint00:43
fungiinstalled on a different platform maybe?00:43
mnaserfungi: i mean, i tried in an ubuntu docker container and tha tworked, and i downloaded the amd64 version (double checked)00:44
*** dangtrinhnt has joined #opendev00:44
mnaserand i literally did an ANSIBLE_ROLES_PATH=./roles ansible-playbook -i localhost, playbooks/golangci-lint/pre.yaml and then run.yaml and it worked fine00:44
mnaserhmm, maybe it's because i have go in this machine and its not in upstream00:45
fungisudo zuul autohold --tenant vexxhost --project vexxhost/libvirtd_exporter --job golangci-lint-verbose --change 716453 --reason "mnaser investigating obscure go architecture error unable to reproduce elsewhere" --count 100:46
fungidoes that look right?00:46
mnaserthe reasoning is perfect00:46
mnaser:)00:46
fungiokay, the trap is set00:46
fungilmk once it fails and i can add your ssh key00:46
mnaserfungi: ok, rechecked, i'll try to figure out why it happened00:48
mnaserfungi: failed!00:50
fungiyou have ipv6 connectivity, right? this is a v6-only node00:52
fungiif not, we can release it and try again00:52
fungialso where do i find a copy of your ssh key(s)?00:53
mnaserfungi: i can find my way through a vm on our cloud00:53
mnaserfungi: https://github.com/mnaser.keys00:53
mnasermy local isp isn't wonderful when it comes for ipv6 :(00:53
mnaseroffice has ipv6, but not there00:53
fungissh root@2607:ff68:100:54:f816:3eff:fe14:5dbf00:54
fungii only just recently worked out how to get prefix delegation via dhcp6 from my current residential broadband provider, and it's still a bit fiddly00:54
fungibut functional enough for me to drop my old tunnel to hurricane electric at least00:55
mnaserfungi: what's neat is their cell phone network is running on ipv6 which is cool, but not residental00:56
clarkbmnaser: thats how I ipv6 since home hasnone00:57
clarkbI bounce through cloud vm with it00:57
clarkbI'm hoping the ziply fiber aquisition results in ipv6 eventually though00:57
*** dangtrinhnt has quit IRC00:58
mnaserclarkb: hopefully so00:58
mnasergolangci-lint returns -bash: /usr/local/bin/golangci-lint: cannot execute binary file: Exec format error00:58
mnaserfile says: /usr/local/bin/golangci-lint: Mach-O 64-bit x86_64 executable00:59
* mnaser hmms00:59
fungiso you're trying to run a binary built for macos?00:59
fungiMach-O Mach-O man, i want to be a Mach-O man! (apologies to the village people)01:00
mnasercrap01:00
mnaserthis is awkward01:00
fungii'm not sure if linux has a mach compatibility shim lkm, if you really need it to run macos binaries on linux01:02
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645201:02
mnaserfungi: sorry for that, i think we can rm that vm :(01:03
fungii think macos/darwin's microkernel has elf compat but not the other way around01:03
fungisure, will do01:03
mnaseryeah, you can run linux stuff but not macos, i assume this tool is "ultra optimized"01:03
fungithe error you were getting makes a lot more sense now, in retrospect01:04
fungiand i've released that node back into the aether from which it spawned01:04
mnaserfungi: cool, yay, now i have another type of error, but one that contains output!01:09
fungilucky you!01:09
fungipuppet updated the etherpad deployment, but i'm still getting the same javascript error from it as before01:09
fungiunlucky me01:10
fungihowever i think my evening is coming to a close so i'll take a fresh look at it in the morning if nothing new is on fire01:10
fungisince it's just the dev server01:10
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645201:11
johnsomHi there, I just got a "msg": "No viable v4 or v6 route found to opendev.org. The build node is assumed to be invalid.",01:12
johnsomhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_564/714004/6/check/neutron-ovn-provider-v2-scenario/564d705/job-output.txt01:12
johnsomJust an FYI, not blocking anything01:13
mnaserjohnsom: uh that's very strange01:14
mnaseropendev.org: Temporary failure in name resolution01:15
mnaseroh an even more interesting01:15
mnaserfetch-output reported that the remote id changed for the machine01:16
fungiso possible there's a rogue vm nova has lost track of in that cloud getting into an arp fight with a job node01:16
johnsomYeah, the remote ID thing is not unusual. I have seen that pop up every once in a while01:17
*** dangtrinhnt has joined #opendev01:17
fungiwe see it from time to time across a seemingly random selection of sorts of jobs01:17
johnsomYeah, just some zombie nova instance. We have special code in octavia to deal with them.01:18
fungihappens more often in some providers than others01:18
mnasermaybe good to report it to the provider01:18
fungiwe try to when we can correlate them01:18
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645201:23
*** xavinux has joined #opendev01:33
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645201:34
*** dangtrinhnt has quit IRC01:37
*** dangtrinhnt_ has joined #opendev01:37
*** xavinux has quit IRC01:42
*** dangtrinhnt_ has quit IRC01:45
*** dangtrinhnt has joined #opendev01:47
*** dangtrinhnt has quit IRC01:52
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645201:59
*** dangtrinhnt has joined #opendev02:01
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645202:10
*** ysandeep|away is now known as ysandeep|rover02:31
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645202:33
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645202:33
openstackgerritIan Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel  https://review.opendev.org/71644902:37
ianwi feel like something might have broken system-config :  Line 1444:02:38
ianwUnknown project openstack/project-config02:38
ianwinfra-prod-manage-projects02:40
ianwhuh ... though it looks like the tests are still running02:43
ianwcorvus / mnaser / mordred : ^ i feel like this is the intersection of all of you :)02:43
ianwhttps://review.opendev.org/#/c/716449/ is the review zuul is commenting on02:43
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645202:47
openstackgerritIan Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel  https://review.opendev.org/71644902:48
mnaserianw: oh what did i do again poop02:48
mnaserianw: wait, i wonder if thats the vexxhost tenant commenting on openstack/project-config ?02:50
ianwhrm, that might be it, and why the opendev side still runs02:51
mnaserianw: yeah, we are loading opendev/project-config indeed02:51
mnaseri think i remember maybe pushing a patch to add a note to what tenant that was sending that02:52
openstackgerritMohammed Naser proposed openstack/project-config master: vexxhost: move base-jobs to config-project  https://review.opendev.org/71645902:53
mnaserianw: i revised my patch and added opendev/project-config to it02:53
openstackgerritIan Wienand proposed opendev/system-config master: [wip] fix ansible-devel job to actually run devel  https://review.opendev.org/71644902:57
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645202:58
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645203:05
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645203:12
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage  https://review.opendev.org/71647403:13
*** dangtrinhnt has quit IRC03:25
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage  https://review.opendev.org/71647403:28
*** bolg has quit IRC03:34
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage  https://review.opendev.org/71647403:43
*** dangtrinhnt has joined #opendev04:00
*** ykarel|away is now known as ykarel04:10
openstackgerritIan Wienand proposed opendev/system-config master: Fix ansible-devel job for Ansible 2.10 changes  https://review.opendev.org/71644904:26
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing namespace collection usage  https://review.opendev.org/71647404:26
ianwmordred: i say we merge 716449, then we can even help out with testing the automatic namespace routing stuff04:28
*** sgw has quit IRC05:07
*** sgw has joined #opendev05:24
*** bolg has joined #opendev05:39
openstackgerritMerged opendev/system-config master: Use ansible debug callback plugin  https://review.opendev.org/71643305:50
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/71615906:03
*** DSpider has joined #opendev06:44
*** tobiash has quit IRC06:51
*** jhesketh has quit IRC06:52
*** jhesketh has joined #opendev06:53
*** tobiash has joined #opendev06:53
*** ysandeep|rover is now known as ysandeep|brb07:03
*** dangtrinhnt has quit IRC07:11
*** dangtrinhnt_ has joined #opendev07:12
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929207:22
*** tosky has joined #opendev07:32
*** rpittau|afk is now known as rpittau07:37
*** ysandeep|brb is now known as ysandeep07:52
*** ralonsoh has joined #opendev07:55
*** ysandeep is now known as ysandeep|rover07:57
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929208:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929208:19
*** ykarel is now known as ykarel|lunch08:30
*** dangtrinhnt has joined #opendev08:44
*** dangtrinhnt_ has quit IRC08:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds variable to toggle whether to revoke sudo  https://review.opendev.org/70624809:05
mnasiadkaMorning09:07
mnasiadkait seems CentOS 8 Extras repo is a bit stale on the mirrors, comparing http://mirror.centos.org/centos/8-stream/extras/x86_64/os/Packages/ to http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/09:07
*** mrunge has joined #opendev09:16
*** dangtrinhnt has quit IRC09:16
*** dangtrinhnt_ has joined #opendev09:17
openstackgerritSorin Sbarnea proposed openstack/diskimage-builder master: Validate virtualenv and pip  https://review.opendev.org/70710409:30
*** osmanlicilegi has quit IRC09:49
*** osmanlicilegi has joined #opendev09:52
mrungehi there, who can refresh a cache for centos-extras in zuul? We have a patch failing, because of http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/ outdated09:54
*** ykarel|lunch is now known as ykarel09:59
*** rpittau is now known as rpittau|bbl10:16
*** dangtrinhnt_ has quit IRC10:26
*** ysandeep|rover is now known as ysandeep|afk10:56
*** njohnston has quit IRC11:20
fricklerinfra-root: ^^ can't look myself currently11:20
*** ysandeep|afk is now known as ysandeep|rover11:22
*** njohnston_ has joined #opendev11:23
*** dangtrinhnt has joined #opendev11:30
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: WIP: Try to fix unicode issue when parsing tox  https://review.opendev.org/71656011:53
*** dangtrinhnt has quit IRC11:55
*** dangtrinhnt_ has joined #opendev11:56
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Ignore errors when parsing tox output  https://review.opendev.org/71656111:59
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929212:02
*** rpittau|bbl is now known as rpittau12:07
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929212:11
mrungeany infra-root able to refresh http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/ ?12:13
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Ignore errors when parsing tox output  https://review.opendev.org/71656112:28
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Ignore errors when parsing tox output  https://review.opendev.org/71656112:35
*** roman_g has joined #opendev12:42
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman  https://review.opendev.org/71657812:47
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: tox_parse_output: add no_log to tox_output  https://review.opendev.org/71657912:54
openstackgerritMerged zuul/zuul-jobs master: Ignore errors when parsing tox output  https://review.opendev.org/71656112:55
*** hashar has joined #opendev13:05
openstackgerritMerged zuul/zuul-jobs master: tox_parse_output: add no_log to tox_output  https://review.opendev.org/71657913:18
*** dangtrinhnt_ has quit IRC13:20
*** dangtrinhnt has joined #opendev13:21
*** dangtrinhnt has quit IRC13:23
*** dangtrinhnt_ has joined #opendev13:23
*** dangtrinhnt_ has quit IRC13:24
*** roman_g has quit IRC13:34
*** ykarel is now known as ykarel|afk13:43
fungimnasiadka: mrunge: frickler: we currently mirror centos 8 via rsync from https://mirror.hackingand.coffee/centos/8/extras/x86_64/os/Packages/13:47
fungiit looks like we're current with what's being served there13:47
fungiif there's a better rsync mirror we should be using, please propose an update13:47
fungihttps://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L3413:47
AJaegermordred: is everything imported from https://review.opendev.org/#/c/716159/ ? Then I'll merge13:48
mordredAJaeger: yes - I have manually imported the two repos that were missed from the upstream13:49
mordredwe're assuming there was a hiccup from when we were running things manually13:49
AJaegergreat, thanks mordred13:49
mrungefungi, can we mirror from http://mirror.centos.org/centos/8/extras/x86_64/os/Packages/ ?13:50
fungimrunge: is there a public rsync mirror for that?13:51
mrungeI'd bet13:51
fungiif you can test whether you can rsync from there, then please propose an update to the line i linked above13:51
mnasiadkafungi: the mirror being used is not really on the official mirror list - https://www.centos.org/download/mirrors/13:52
mrungefungi, where would I propose a change?13:52
fungimrunge: the same system where you're proposing the changes which are being tested with packages from our mirror network, review.opendev.org13:53
mrungefungi, I get that, but where is this in zuul or so, which repo has the info to mirror from?13:54
fungimrunge: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L3413:54
mnasiadkamrunge: it's in this file: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update - will you propose a change?13:54
fungilike i linked just above13:54
mnasiadkaoops, fungi was faster ;)13:55
mrungecurrently in a call. Will have a look later13:55
mrungeor who ever is faster than me13:55
mrungethank you mnasiadka and fungi13:55
fungicool, doesn't sound urgent but happy to review once someone has had a chance to test out and confirm they can rsync from somewhere else13:55
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Strip ansi codes from pep8 message  https://review.opendev.org/71659813:56
fungimrunge: mnasiadka: for reference, that mirror was chosen by ianw back in september, rationale is in the commit message for https://review.opendev.org/68443713:57
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929213:57
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Strip ansi codes from pep8 message  https://review.opendev.org/71659814:01
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add flag for toggling inline comments for linters  https://review.opendev.org/71659914:01
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929214:01
openstackgerritMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/71615914:01
*** ykarel|afk is now known as ykarel14:02
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment  https://review.opendev.org/71660114:16
openstackgerritMichal Nasiadka proposed opendev/system-config master: Change CentOS 8 upstream mirror  https://review.opendev.org/71660214:19
mnasiadkamrunge: ^^14:19
mrungeyou beat me, mnasiadka14:19
mnasermordred: might having a look at https://review.opendev.org/716459 ? clarkb and corvus previously +2'd but we lost them because i added project-config to the list of projects that we don't load in our tenant b/c we were reporting to that project inside gerrit14:19
mnasiadkamrunge: oh well, next time you'll be faster ;)14:21
mrungedoesn't really matter, thank you for looking into that mnasiadka :)14:22
mnasiadkamrunge: it hurts kolla (not counting other projects), so it's in my interest ;)14:22
openstackgerritMerged zuul/zuul-jobs master: Add flag for toggling inline comments for linters  https://review.opendev.org/71659914:23
openstackgerritMerged openstack/project-config master: vexxhost: move base-jobs to config-project  https://review.opendev.org/71645914:31
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645214:33
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Do not end host if correct go version is installed  https://review.opendev.org/71660714:34
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment  https://review.opendev.org/71660114:38
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645214:42
mordredAJaeger, fungi : WOOHOO! https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a37/716159/2/promote/infra-prod-manage-projects/a37619b/manage-projects.yaml.log14:45
mordredthat's from the latest manage-projects run triggered by zuul for the cleanup patch14:45
mordredhttps://review.opendev.org/#/c/716159/14:45
mordredmnaser: ^^ you can now check output of manage-projects without an infra-root14:46
mnasermordred: !!! that's awesome14:48
AJaegermordred: cool \o/14:48
*** ysandeep|rover is now known as ysandeep|away14:50
fungimnaser: mrunge: i've approved the mirror change, the next scheduled update pulse will be in a couple hours at 16:43z so we should hopefully see it take effect around then. i'm tailing the log from our cronjob so i can see when it does in case there are problems14:54
fungier, mnasiadka ^ (sorry mnaser! tab-fail on my part)14:54
mrungethank you fungi14:54
mrungemuch appreciated14:54
*** hashar has quit IRC14:55
fungithanks for spotting the stale mirror and working on updates. if this one doesn't do the trick for some reason we can try the other one you mentioned in your review comment14:55
mrungesure. I'll keep an eye on that.14:56
mnaserinfra-root: i think i broke zuul.14:57
mnaseronce my change with depends-on for golangci-lint finished14:57
mnaserit got stuck for a little bit and didn't reportt14:57
mnaserand now all the pipelines are gone and the change didn't report: https://zuul.opendev.org/t/vexxhost/status14:57
mnaserwaaaiat14:57
mnasercrap14:57
mnaserwe merged the change that made the pipelines disappear.14:57
mnaseri didnt include load pipelines in that change14:58
fungimnaser: thankfully this doesn't seem to have impacted any other tenants14:58
mnaseryes, this is because it was a vexxhost only tenant change14:58
mnaserso we just don't have a pipeline config right now14:58
fungihappy to fast-approve the fix when you push it14:58
mnaserok, let me revise the change to load pipeline as well14:58
*** lpetrut has joined #opendev14:59
mnasiadkafungi: thanks!14:59
openstackgerritMohammed Naser proposed openstack/project-config master: vexxhost: load base-jobs & project-config pipelines  https://review.opendev.org/71662015:03
*** njohnston_ is now known as njohnston15:03
mnaserfungi: here's another approach15:03
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment  https://review.opendev.org/71660115:05
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments  https://review.opendev.org/71662315:05
*** lpetrut has quit IRC15:06
mordredcorvus: don't know if you saw - but we had a successful manage-projects run with the output reported back to zuul: https://review.opendev.org/#/c/716159/ -> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a37/716159/2/promote/infra-prod-manage-projects/a37619b/manage-projects.yaml.log15:09
corvusmordred: that looks great :)15:11
mordredcorvus: I'm quite pleased. which means I think we're actually ready for https://review.opendev.org/#/c/715957/ now! :)15:13
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments  https://review.opendev.org/71662315:16
openstackgerritMerged opendev/system-config master: Change CentOS 8 upstream mirror  https://review.opendev.org/71660215:20
fungirevisiting yesterday's etherpad-dev error, love it when i paste an error into a web search and turn up one of our own conversations from 5 years ago: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-09-02.log.html#t2015-09-02T15:34:3115:20
fungii guess it's possible we need to change up how we're proxying newer etherpad15:20
mordredfungi: awesome :)15:21
mordredfungi: did you see I got started on ansible/docker for etherpad yesterday?15:21
fungii did!15:21
*** hashar has joined #opendev15:21
*** ykarel is now known as ykarel|away15:22
mordredwoot. luckily I hadn't gotten to doing apache yet - so good timing if we need to change how we proxy :)15:23
fungiweb console in ff tells me this: The resource from “https://etherpad-dev.openstack.org/socket.io/socket.io.js” was blocked due to MIME type (“text/html”) mismatch (X-Content-Type-Options: nosniff).15:25
clarkbreviewing that change is top of my list once I actually get started. Today is like tcp, slow start15:25
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman  https://review.opendev.org/71657815:26
fungiclarkb: open a large receive window15:27
fungifrom a shell on etherpad-dev, `http://localhost:9001/socket.io/socket.io.js` gives me a 40415:29
fungii'm assuming it should not15:29
fungier, passing it to wget i mean15:29
fungion etherpad.o.o it works, so i'm guessing that's a clue15:30
clarkbfungi: did it get rebuilt with newer nodejs after the git repo edit?15:31
clarkbmaybe double check build process hasnt changed tok15:31
clarkb*too15:31
fungiaha, found puppet errors15:32
fungiApr  1 00:13:51 etherpad-dev01 nodejs[32156]: #033[31m[2020-04-01 00:13:51.434] [ERROR] console - #033[39mFailed to load '/opt/etherpad-lite/etherpad-lite/node_modules/ep_etherpad-lite/node/hooks/express/socketio:expressCreateServer' for 'ep_etherpad-lite/socketio/hooks/expressCreateServer': Error: Cannot find module 'nodeify'15:32
fungilots of "Cannot find module 'nodeify'" for a variety of different libraries15:33
openstackgerritIvan Kolodyazhny proposed openstack/project-config master: Add jobs for xstatic-graphlib  https://review.opendev.org/71663015:34
fungii suppose it's looking for https://www.npmjs.com/package/nodeify15:35
fungisrc/package-lock.json does declare it's a dependency at least15:36
fungilooks like that was added by c499a08 which first appears in the 1.8.0 tag (their latest release)15:39
clarkbfungi: their readme says to run bin/run.sh which is what we do iirc15:39
clarkbfungi: https://github.com/ether/etherpad-lite/issues/3252 maybe we need to run bin/installDeps.sh?15:42
openstackgerritMonty Taylor proposed opendev/system-config master: WIP Dockerize etherpad  https://review.opendev.org/71644215:42
fungii'll give it a try manually here and see what happens15:43
clarkbpuppet does supposedly run that15:43
mordredclarkb: fwiw - in that ^^ I started from their upstream image, so once we're on it we should be able to stop caring about how to build etherpad ;)15:43
clarkboh! except that we have a creates on the puppet15:43
clarkbso it will only run if the dir it creates isn't there15:43
clarkbso ya I bet tahts it, need to run that15:44
clarkbmordred: are we basically going to get that -1 on all system-config jobs until we move it into the opendev tenant?15:50
mordredyup15:50
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663615:50
mordredclarkb: maybe we should exclude project from the zuul config entry for it15:51
clarkb++ if that makes the noise go away15:51
mordredoh - we already do15:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663615:52
fungiclarkb: and for some reason it complains "Error: EACCES: permission denied, mkdir '/opt/etherpad-lite/etherpad-lite/src/node_modules/wd/build'" even when run as root15:53
clarkbmordred: left note on the conatiner for etherpad change15:53
fungilooks like it blows away the node_modules dir completely and then chokes when trying to create a subdir under it15:54
mordredclarkb: yes - I was thinking a similar thing15:54
clarkbfungi: if you look in the script it seems to do a bunch of stuff with an || rm node_modules15:56
clarkbfungi: implying something is failing in that bunch of stuff?15:56
fungiahh, i think it may also have wanted this run as eplite15:56
fungiokay, that seems to have worked better15:56
fungichowned the tree to eplite, then used `sudo -H -u eplite bin/installDeps.sh`15:57
openstackgerritMerged openstack/project-config master: vexxhost: load base-jobs & project-config pipelines  https://review.opendev.org/71662015:57
clarkbfungi: looks like puppet runs it as that user15:57
fungiyep15:57
fungiadded 1003 packages from 1155 contributors and audited 13804 packages in 24.216s15:57
clarkbremember when people thought we had a lot of dependencies in openstack?15:57
fungiif only we'd written it all in javascript15:58
fungijust think of the bragging rights!15:58
fungiokay, https://etherpad-dev.openstack.org/ seems to be up and running15:59
clarkbhttps://etherpad-dev.openstack.org/p/clarkb-test2 loads for me with expected content too16:00
clarkbclarkb-test still fails to load16:00
clarkb(also expected)16:00
corvusthat looks very much like the current etherpad16:00
corvusi thought it was sposed to be way different?16:00
clarkbcorvus: the etherpad-lite readme gif indicates things look the same still16:00
clarkbnot sure if that is just stale image though16:00
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663616:01
fungiyeah, i thought they had changed up the default theme16:02
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663616:03
clarkbhttps://github.com/ether/etherpad-lite/issues/3441#issuecomment-56289644716:04
clarkbit is a settings.json thing, that is a file we manage with puppet. Possible we are undoing the "default" by supplying the file directly?16:04
clarkbyup we don't set skinName16:05
clarkband that results in the empty value old school theme16:06
clarkb(according to settings.json.template)16:06
fungithere's a message about that when starting up too16:06
mordredclarkb: we have support for openid in the puppet etherpad - but we're not using that right?16:06
fungior maybe it was in the puppet stdout16:06
clarkbmordred: correct, I ended up using it elsewhere but not in the openstack/opendev deployment16:07
mordredk16:07
clarkbmordred: I think it can be safely ignored for dockering16:07
fungiyeah, puppet16:07
fungiNo "skinName" parameter found. Please check out settings.json.template and update your settings.json. Falling back to the default "no-skin".16:08
clarkbso they didn't really change the default16:08
clarkbthey provide a default example config with a different value16:08
clarkbservice default remains unchanged16:09
fungiseems that way16:09
fungii can flip it in the config real quick if folks want to see16:09
clarkbfungi: if nothing else it will help confirm we're running the new thing16:10
funginew theme temporarily in place until puppet undoes it16:13
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663616:14
openstackgerritMonty Taylor proposed opendev/system-config master: WIP Dockerize etherpad  https://review.opendev.org/71644216:14
clarkbthe LOADING.. banner doesn't ever seem to go away even after content has loaded16:14
clarkbthough maybe it is trying to load a top banner that isn't coming in?16:14
mordredclarkb: ok - now with database and apache16:14
fungiclarkb: maybe16:14
clarkbmy immediate reaction is I really don't like this :)16:14
clarkbits quite a bit less dense16:14
fungialso as i noted in that gh issue, it seems they've "optimized" it for portrait orientation screens (phones and tablets?)16:15
clarkbfungi: I think that may just be to limit width of text since people have an easier time reading shorter lines16:15
*** rpittau is now known as rpittau|afk16:15
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Add tox_envlist to the inline comment  https://review.opendev.org/71660116:16
clarkbfor drafting actual documents this is probably better, but for brainstorm scratch pad I think the original theme is likely better16:16
clarkbgives you much more room to work with16:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663616:17
fungii suspect it's not all loading, as evidenced by the perpetual "loading" message you mentioned16:18
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Rename install- roles to ensure- for consistency  https://review.opendev.org/71663616:19
fungiprobably more urls we need to whitelist in our proxy config16:19
fungii'll check it out with a js debugging console shortly16:19
clarkbfungi: probably files under the theme/ dir or something16:20
clarkbsrc/static/skins is the source repo path16:20
openstackgerritsebastian marcet proposed opendev/system-config master: Openstack Id production deploy v3.0.9  https://review.opendev.org/71665316:22
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments  https://review.opendev.org/71662316:32
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Require a / in a file path  https://review.opendev.org/71665516:32
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Use os.path.exists  https://review.opendev.org/71665716:36
openstackgerritMerged zuul/zuul-jobs master: Strip ansi codes from pep8 message  https://review.opendev.org/71659816:37
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments  https://review.opendev.org/71665516:38
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: WIP Add testing for inline tox comments  https://review.opendev.org/71662316:38
clarkbre using etherpad-dev with jitsi, the ssl cert on that server is currently self signed. I wonder how that will interact with jitsis nested loading. Maybe you have to navigate to the self signed location first, accept the cert, then load jitsi?16:38
* clarkb makes change and we can test that16:38
openstackgerritClark Boylan proposed opendev/system-config master: Switch meetpad to etherpad-dev  https://review.opendev.org/71666016:40
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments  https://review.opendev.org/71665516:40
fungiclarkb: the other thing i've been wondering about is whether having the meetpad and etherpad dns names in different domains is crossing a line from a browser tracking paranoia standpoint16:42
clarkbfungi: ya that was another thought that came up16:43
clarkbwe can probably test that with etherpad-dev too fi we're adding ssl verification exceptions anyway16:43
clarkb(add a etherpad-dev.opendev.org cname and tell brwoser to add exception for that verification error too)16:43
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments  https://review.opendev.org/71665516:43
openstackgerritMerged zuul/zuul-jobs master: Add tox_envlist to the inline comment  https://review.opendev.org/71660116:44
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666316:45
mordredclarkb: yah - and if we get happy with the docker patch above, we should likely take the opportunity to spin up etherpad01.opendev.org, roll out new code on it, LE it, and make etherpad.openstack.org a redirect like we do with gerrit16:46
clarkbmordred: ++16:47
mordredclarkb: in fact, why don't I make that patch for etherpad.opendev and not for etherpad.openstack - that way we can land it and spin up a new server and see how it goes16:47
mordredin parallel to the other stuff16:47
openstackgerritClark Boylan proposed opendev/zone-opendev.org master: Add an etherpad-dev CNAME to openstack.org  https://review.opendev.org/71666516:49
clarkbmordred: we can just do ^ too16:49
clarkb(don't need a new server yet)16:49
clarkboh in parallel ++16:50
fungiwfm16:50
mordredyeah16:51
mordredlike - might as well roll out docker on a clean host in this case16:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666716:53
fungithe great dockering seems like a fine fulcrum for switching over to using etherpad.opendev.org as the official service name anyway16:53
fungi(after -dev of course)16:54
clarkbfungi: we may not need -dev anymore with the system-config-run-etherpad job16:55
clarkbthats one of the great things about proper end to end testing there. We can treat it as the dev platform16:55
fungiyep16:56
mordred++16:56
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: go-jobs: improve testing  https://review.opendev.org/71666816:56
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Check that a file exists for inline comments  https://review.opendev.org/71665516:58
openstackgerritMerged opendev/zone-opendev.org master: Add an etherpad-dev CNAME to openstack.org  https://review.opendev.org/71666517:02
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666717:06
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency  https://review.opendev.org/71667517:06
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644217:09
fungimnasiadka: mrunge: our centos 8 mirror seems to have updated as of a little while ago, please see if things are still working for you17:11
fungi(or are back to working, i guess)17:12
mnaserhmm17:12
mnaserQueue lengths: 2173 events, 0 management events, 11 results.17:12
mnaserthat seems unusual, no?17:13
mnaseroh, it's zero now17:13
fungithey tend to back up during reconfigure events17:13
clarkband no not very unusual17:13
clarkbour zuul is busy17:13
fungiseems like even a global pandemic doesn't slow down our workload17:14
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644217:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666317:19
mnaseroh yay17:19
mnaseri have my pipelines back.17:19
openstackgerritMerged opendev/system-config master: Openstack Id production deploy v3.0.9  https://review.opendev.org/71665317:20
mordredmnaser: \o/17:23
fungieveryone loves their pipelines17:23
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Generalize parse tox output  https://review.opendev.org/71626317:29
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626417:29
mordredfungi: in soviet russia, pipeline loves everyone17:29
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666317:31
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666717:31
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency  https://review.opendev.org/71667517:31
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency  https://review.opendev.org/71668217:31
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Do not end host if correct go version is installed  https://review.opendev.org/71660717:34
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929217:36
openstackgerritMerged zuul/zuul-jobs master: Check that a file exists for inline comments  https://review.opendev.org/71665517:38
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645217:39
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency  https://review.opendev.org/71668517:41
mordredclarkb: so - I'm happy with how manage-projects is running - I'd like to land the base/bridge one - whatcha thing?17:42
mordredthink?17:42
mordredhttps://review.opendev.org/#/c/715957/ (it's already got 2x+2 - just checking in before I pull the trigger)17:42
clarkbmordred: ya I think we can do that next. Do we need to set allowed projects for those jobs too?17:43
clarkbwe shouldn't need the semaphore because only system-config is running them in periodic and promote17:44
fungitesting the colibris skin on etherpad-dev with the js debugging console open, the only error it's reporting is "Error: setAuthorInfo: author (undefined) is not a string"17:44
clarkbbut maybe we explicitly restrict it to system-config for now to avoid it growing outside?17:44
mordredclarkb: yeah - I thnik we don't need to17:44
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency  https://review.opendev.org/71668717:44
clarkbfungi: look in the network debugger to see if its failing to load files17:44
fungiahh17:44
clarkbthey should show up as 404 or similar17:44
mordredclarkb: well - it won't work from other repos17:45
mordredclarkb: becuase it depends on the per-project ssh key17:45
clarkbmordred: it will work from project-cofig17:45
clarkb(we have that key on bridge too iirc)17:45
mordredyeah - but only from project-config17:45
mordredyeah17:45
mordredclarkb: if I add allowed-projects to the infra-prod-apply base job - that should let us just override it on child jobs yes?17:46
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Generalize parse tox output  https://review.opendev.org/71626317:46
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626417:46
mordredclarkb: actually - it's pointless17:47
mordredclarkb: allowed-projects is ignored by config projects :)17:47
clarkboh heh17:47
clarkbok17:47
mordredk. pulling the trigger17:48
mordredassuming we're happy with this one, I'll make the patches to do the same with everything else :)17:48
fungiclarkb: no error codes on any of the requests, though it did eventually stick a js error in the pad itself. trying to recreate now17:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-go to ensure-go for consistency  https://review.opendev.org/71668917:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency  https://review.opendev.org/71668717:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency  https://review.opendev.org/71669217:54
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency  https://review.opendev.org/71669317:58
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency  https://review.opendev.org/71669518:01
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency  https://review.opendev.org/71669818:04
openstackgerritMerged opendev/system-config master: Run manage-projects/base/bridge on system-config changes  https://review.opendev.org/71595718:11
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666318:14
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666718:15
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency  https://review.opendev.org/71667518:16
fungiclarkb: so, interesting to note, if i start a new pad with the colibris theme set, it all finishes loading fine. might have just been something about the state of the clarkb-test2 pad?18:17
fungihttps://etherpad-dev.openstack.org/p/NHKtmbuTD5MbrkqFHIp_18:18
clarkbfungi: possibly18:18
clarkbfungi: maybe because I have things cached in the browser too18:18
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency  https://review.opendev.org/71668518:19
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency  https://review.opendev.org/71668718:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency  https://review.opendev.org/71669218:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency  https://review.opendev.org/71669318:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency  https://review.opendev.org/71669518:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency  https://review.opendev.org/71669818:20
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666318:21
*** diablo_rojo has quit IRC18:32
*** diablo_rojo has joined #opendev18:33
*** hashar is now known as hasharBreak18:34
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message  https://review.opendev.org/71672218:40
mordredinfra-root: fwiw, zuul is currently running the base playbook18:41
mordredand previously successfully ran update-system-config18:42
mordredso our new zuul overlords are working18:42
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-docker to ensure-docker for consistency  https://review.opendev.org/71666318:42
clarkbexciting18:45
mordred\o/18:46
openstackgerritMonty Taylor proposed opendev/system-config master: Make a new dockerized etherpad.opendev.org  https://review.opendev.org/71644218:50
mordredwow. the base playbook takes 42 minutes19:00
corvusmordred: yeah, i think in the long run we wanted to just do that daily or something?19:00
corvusmordred: now that i look closer at https://review.opendev.org/715957 i'm not sure i understand that second paragraph fully19:02
*** xavinux has joined #opendev19:03
openstackgerritMonty Taylor proposed opendev/system-config master: Run base playbook with 50 forks  https://review.opendev.org/71672719:03
mordredcorvus: ^^ also - we ran it differently in zuul than we did in run_all19:04
corvusmordred: why does manage-projects need bridge and base?19:04
mordredit needs bridge because bridge applies any changes to ansible settings (and before it would get those natually as a result of sequencing)19:05
corvusmordred: ideally the answer to that is, it doesn't.  but if it does, can we look at putting those tasks in the service playbook?  that seems more appropriate.  that way they are automatically limited to the hosts involved.19:05
mordredit might not actually need base come to think of it though19:05
mordredcorvus: yeah- I think that's a great idea19:05
mordredoh - wait - your second thing I read wrong19:06
corvusso then base is just something that runs infrequently on everything (to maintain stasis) and on new node bringup.  then each service playbook encapsulates what's needed to operate that service from start to finish.  but if we have lots of playbooks adjusting settings on bridge, that could be a problem (that could be a problem with the current approach too)19:07
mordredbut - the first thing is the more telling - I dont think we need base - I think we do need bridge19:07
mordredyeah - I think we could put in a depend on bridge easily - it's a very short playbook19:07
mordredand make it soft - so if nothing touched the trigger files for bridge, it doesn't run and all is good19:07
mordredbut if it does, we run bridge real quick then manage-projects, yeah?19:08
corvusthat sounds good19:08
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645219:08
* corvus lunches19:08
* mordred makes change19:08
mordredcorvus: actually - it's just a bad commit message19:09
mordredcorvus: we don't need base for manage-projects - or bridge - no do we depend on them19:09
mordredwe need update-system-config for manage-projects, and we do depend on that19:09
xavinuxhi, hope everyone is well here19:10
clarkbmordred: that assumes ansible and all that is already in place, which is probably fine (but I think bridge does that technically?)19:10
xavinuxhave been taking a look at these links https://docs.openstack.org/infra/system-config/ and https://docs.opendev.org/opendev/infra-manual/latest/19:11
xavinuxas i would like to contribute to the infra team19:12
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-kubernetes to ensure-kubernetes for consistency  https://review.opendev.org/71666719:15
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-openshift to ensure-openshift for consistency  https://review.opendev.org/71667519:15
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-podman to ensure-podman for consistency  https://review.opendev.org/71668219:15
openstackgerritMonty Taylor proposed opendev/system-config master: Rename bridge.yaml to install-ansible.yaml  https://review.opendev.org/71673119:15
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-devstack to ensure-devstack for consistency  https://review.opendev.org/71668519:15
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-javascript-packages to ensure-javascript-packages for consistency  https://review.opendev.org/71668719:16
mordredclarkb: yeah - it does - ^^ I just renamed that because it's actually really unclear19:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency  https://review.opendev.org/71669219:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency  https://review.opendev.org/71669319:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency  https://review.opendev.org/71669519:16
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency  https://review.opendev.org/71669819:16
clarkbmordred: one small suggestion maybe we should order the jobs in rough "chronological" execution order to show the dependencies that way too?19:19
mordredclarkb: that's a good idea19:19
clarkb-base comes after -ansible at this point basically19:20
* clarkb finds lunch now too19:21
clarkboh wait cathcing up on scrollback xavinux has questions19:21
clarkbxavinux: welcome! sorry my stomach is hungry so I am distracted :)19:21
clarkbxavinux: currently we've got a few things in progress to give you an idea of the sorts of things happening right now. We are deploying our Gerrit with ansible and docker (migrating from puppet), we are starting to drive our ansible deployments from Zuul (our CI/CD tool) rather than cron, we are deploying a new jitsi meet server to integrate with existing etherpad services (to enable remote meetings and19:23
clarkbcollaboration), and we are adding new fedora 31 test nodes (which involves adding mirrors and diskimage-builder image builds)19:23
clarkbxavinux: if any of that is of interest feel free to dive in and start doing reviews or offer to help write changes, and if you need help knowing where to do that you can ask in here (and I'm also happy to help on a more 1:1 basis)19:24
clarkbxavinux: there is typically quite a lot happening so finding something interesting to you that you can focus on is probably a good way to get started19:24
openstackgerritMonty Taylor proposed opendev/system-config master: Reorder jobs in job list "chronologicaly" and add files  https://review.opendev.org/71673419:26
mordredclarkb: how's that look now?19:26
xavinuxclarkb: no problem! and thanks for your welcome!19:26
mordredoh yay! yeah- welcome xavinux !19:26
mordredI'm about to be producing a pile of changes for the replace-cron-with-zuul thing he mentioned - eyeballs definitely appreciated!19:27
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Support multiple matchers when parsing tox output  https://review.opendev.org/71626319:27
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626419:27
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message  https://review.opendev.org/71672219:27
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-nodejs to ensure-nodejs for consistency  https://review.opendev.org/71669219:28
*** ralonsoh has quit IRC19:28
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-yarn to ensure-yarn for consistency  https://review.opendev.org/71669319:29
xavinuxmordred thanks for your welcome!19:31
clarkbmordred: the reorder change lgtm and the file matchers on base should speed things up quite a bit19:32
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Keep error status in tox run  https://review.opendev.org/71673619:33
xavinuxonly a question, what kind of servers do the openstack infra team support? are those server for the openstack website or also other services behind the openstack project?19:33
clarkbxavinux: we run development and collaboration tools to produce the software. code review, ci, wiki and communications tools and so on19:34
mordredclarkb: ++19:37
xavinuxclarkd good, so this infra is used to produce the code behind the openstack services like nova, glance, neutron, etc?19:37
clarkbxavinux: yes19:37
xavinuxnice19:38
xavinuxto let the team know more about me, i live in argentina, work as a cloud engineer, recently obtain the aws solution architect certification and work as a Linux sysadmin for the last 10 years in an internet service provider19:40
xavinuxteach about openstack in an it institute for the las 3 years, where i used to have a small lab with packstack and openstack running an all-in-one version on linux centos19:41
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Keep error status in tox run  https://review.opendev.org/71673619:42
*** xavinux has quit IRC19:42
*** xavinux has joined #opendev19:47
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626419:51
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message  https://review.opendev.org/71672219:51
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-pdk-dependencies to ensure-pdk-dependencies for consistency  https://review.opendev.org/71669519:52
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Rename install-if-python to ensure-if-python for consistency  https://review.opendev.org/71669819:52
openstackgerritMerged zuul/zuul-jobs master: Keep error status in tox run  https://review.opendev.org/71673619:56
*** hasharBreak is now known as hashar19:57
openstackgerritMonty Taylor proposed opendev/system-config master: Run service-bridge in zuul  https://review.opendev.org/71674520:02
openstackgerritMonty Taylor proposed opendev/system-config master: Migrate gitea-lb to zuul  https://review.opendev.org/71674620:02
openstackgerritMonty Taylor proposed opendev/system-config master: Run letsencrypt in zuul  https://review.opendev.org/71674720:02
mordredclarkb: ^^ starting to peel services off20:04
mordredxavinux: nice! well - most of us here (other than mnaser) don't spend much time actually running openstack clouds - but we do certainly use them heavily. all of our servers here run as VMs in openstack public clouds20:06
mordredxavinux: we have a static inventory at the moment: https://opendev.org/opendev/system-config/src/branch/master/inventory/openstack.yaml which has all of the servers listed - as well as what region of what cloud they are in20:06
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Update registry test to use ensure-podman and ensure-docker  https://review.opendev.org/71675220:06
mordredthat repo is the root repo that drives our gitops - which is currently transitioning from being puppet run by ansible to being just ansible running things that are increasingly installed via containers20:08
mordredhappy to have more friends in channel with linux sysadmin background - you'll fit right in :)20:08
clarkbmordred: the LE one might be awkward because we need it to always run before the cron (until we get services out of cron). We may just want to give people a heads up that ordering there may be weird until we've transitioned20:11
clarkbits eventually consistent which is good20:11
mordredyeah20:12
mordredclarkb: also - I think I can have the rest of these done today20:12
mordredthey're not hard patches to write20:12
mordredclarkb: do we want to run letsencrypt early and add soft depends on it from other service playbooks?20:13
mordredI could put it as a soft-depend in that base job and we could just make everything soft-dep on it20:13
clarkbmordred: I think it may need to go after base but before everything else technically20:14
mordredok. why don't I make that change20:14
*** sgw has quit IRC20:14
clarkbbut if it doesn't strictly need base it could be the root too20:14
xavinuxmordred good! will take a look at that inventory20:16
openstackgerritMonty Taylor proposed opendev/system-config master: Run letsencrypt in zuul  https://review.opendev.org/71674720:17
mordredclarkb: that puts it soft after base - so if we're gonna do base, cool, we'll wait (which is honestly not a bad idea anyway given how many forks base wants)20:17
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645220:18
openstackgerritMonty Taylor proposed opendev/system-config master: Run nameserver in zuul  https://review.opendev.org/71676420:21
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626420:23
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message  https://review.opendev.org/71672220:23
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output  https://review.opendev.org/71676620:23
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Support multiple matchers when parsing tox output  https://review.opendev.org/71626320:32
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output  https://review.opendev.org/71676620:32
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626420:32
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message  https://review.opendev.org/71672220:32
openstackgerritMonty Taylor proposed opendev/system-config master: Run nodepool in zuul  https://review.opendev.org/71677020:34
openstackgerritMonty Taylor proposed opendev/system-config master: Run meetpad in zuul  https://review.opendev.org/71677120:34
clarkbmordred: is there a chagne yet to not load project from system-config in opendev tenant? I Think that is what we need there right?20:35
openstackgerritMonty Taylor proposed opendev/system-config master: Run mirror-update in zuul  https://review.opendev.org/71677220:38
mordredclarkb: no - we already do that20:38
clarkbhrm why do we keep getting those -1s20:38
mordredclarkb: I *think* it's more complicated - because I think the opendev tenant has a config error20:39
clarkbah20:39
mordredhttps://zuul.opendev.org/t/opendev/config-errors20:39
mordredoh. duh. it's because we reference openstack/project-config - but that's unknown to the opendev tenant20:41
mordredit's the half-transition to opendev issue20:41
clarkbya we could fix that by not lodaing jobs but I think that is why we have system-config in opendev tenant (for the jobs)20:41
mordredyeah20:41
mordredand this is blocked  on the puppet jobs being legacy jobs20:41
clarkbwhat are we pulling from system-config in opendev specifically?20:42
clarkbmaybe we can split that out?20:42
mordredI don't know?20:42
mordredI mean - to be honest, there is very little in the opendev tenant right now20:43
mordred          - opendev/gear20:44
mordred          - opendev/lodgeit20:44
mordred          - openinfralabs/contrib20:44
mordredthose are the only "meaningful" repos in there20:44
mordredother than, of course, the inaugust repos20:44
clarkbI41b345246f2012d15d969524b5879c9da32b708d20:44
clarkb"Specifically, we're looking to run third-party-check builds of Gerrit and Zuul on changes to upstream Gerrit repos."20:44
mordredoh ... waiut20:45
mordredyeah20:45
clarkbmaybe we should run those out of the openstack tenant for now?20:45
mordredwe shoudl really redo the puppet jobs in ozj20:45
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626420:46
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: DNM: Debug sphinx message  https://review.opendev.org/71672220:46
*** xavinux has quit IRC20:46
clarkbanother option is to pull in project-config but load only job from it?20:46
clarkbhrm that might not fix the pupept error20:47
mordredyeah20:47
clarkband the allowed-projects error we could fix that by allowing nothing (just defining the repo in the tenant but nothing else)20:47
clarkbmordred: ya I think we can add openstack/project-config with nothing included and add ozj with just job included?20:48
clarkbor remove system-config from opendev until we can migrate it (and run gerrit integration jobs out of openstack tenant if necessary)20:49
corvusclarkb, mordred: we have suspended the third-party-check builds of gerrit right now; we can ignore them for a bit20:51
clarkbcorvus: meaning its safe to undo the system-config inclusion into the opendev tenant (the reason this comes up is we get a -1 from opendev tenant every time we make system-cofnig changes due to project-config not being defined in that tenant)20:52
mordredyou know...20:53
corvusclarkb: probably?  all i know at this level of involvement is that i disabled the upstream checker config20:53
mordredI think let's disable it for now - since also we have a zuul there now, so us building gerrit with our zuul is less important20:54
mordredwe're not even running those from that tenant20:56
mordredwe already are running them from openstack20:56
mordredor - hrm. I have no idea where we're running them if we are20:57
openstackgerritClark Boylan proposed openstack/project-config master: Revert "Add system-config and zuul to OpenDev tenant for jobs"  https://review.opendev.org/71677520:57
clarkbI think ^ may be all we need. That was prepatory but then we didn't run things from there in the end?20:57
clarkbor it got removed as corvus said20:57
mordredclarkb: I'm game to try it20:58
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645221:05
mordredinfra-root: https://review.opendev.org/#/q/topic:infra-prod-zuul <-- I'm slogging through those - which is just making a lovely nice pile of reviews for you all21:07
ianwmordred: not sure if you saw but i got the devel job working; see https://review.opendev.org/#/c/716449/.  imo we should merge it because we'll be in a position to validate the ongoing work for automatically routing the old names to their new fully-qualified modes21:07
mordredianw: yes indeed! I was actually going to follow up with you on that ... I think we could probably take some of it and even start doing it for real now - like the install of the collectinos we are using - and the updating our use of them to being the qualified version21:10
mordredianw: but I thik what's there is a great step for now21:10
*** xavinux has joined #opendev21:10
mordredianw: (once ansible-base is out, I'm pretty sure we'll want to just install ansible-base and a couple of collections and call it done)21:11
ianwyeah, if someone feels keen it's just a matter of updating zuul to know about the collections repos and we can do speculative changes for them too, which would be cool21:11
mordredyeah21:12
mordredianw: in other news - I hope you enjoy the giant pile of infra-prod-zuul topic patches :)21:12
mordredI didn't want you to be bored when you woke up21:12
ianwanyway, would be also be nice to be able to jump into https://github.com/ansible/ansible/pull/67684 when it's later and say "hey, working great for us :)"21:12
clarkbmordred: see comment on https://review.opendev.org/#/c/716764/121:12
ianwhaha yes my client scrollback overflowed so i knew something was up21:13
clarkbinfra-root can we also land and test https://review.opendev.org/#/c/716660/ ? then if that doesn't work we can switch meetpad to etherpad-dev.opendev.org to see if the domains are at fault21:13
mordredclarkb: oh - the zone repos are all in the openstack tenant right now, right/21:13
mordred?21:13
fungiianw: not sure if you saw, but just a heads up that the centos 8 rsync mirror we're pulling from was changed to one which is being updated with more regularity21:13
mordredor did we put zone-zuul in the zuul tenant?21:13
fungimordred: we've kept them all together afaik21:14
clarkbmordred: they are in openstack and opendev and zuul is also in zuul21:14
fungioh, got it, so they're all in more than one tenant21:14
clarkbmordred: I think the real trick is that we'd need to add their keys to bridge to have them trigger that job21:15
mordredwell - we can't trigger patches in system-config on patches in another tenant no matter how much we wait21:15
clarkband to do that we basically have to assert that all the zone files are opendev owned21:15
mordredI don't think that'll work atm21:15
mordredwe'd have to re-revert the patch you just reverted :)21:15
clarkbmordred: I don't think that is an issue, they are all in openstack/21:15
openstackgerritMerged openstack/project-config master: Revert "Add system-config and zuul to OpenDev tenant for jobs"  https://review.opendev.org/71677521:15
clarkbmordred: so we can drive it from openstack/project-config21:15
mordredI though we just said they were in multiple tenants21:16
clarkbthe bigger issue is project ssh keys21:16
clarkbmordred: they are, but I don't think that is an issue?21:16
ianwfungi: cool, any manual interventions required?21:16
fungiianw: not so far at least21:17
clarkbmordred: I think in openstack/project-config/zuul.d/projects.yaml we can add entries for zone* that run the opendev/system-config nameservers playbook job21:17
fungiianw: seems to have updated fine21:17
clarkbmordred: and since that config is entirely in the openstack tenant it won't complain21:17
clarkbthe bigger issue is the project ssh keys aiui21:17
mordredyes. I agree21:17
mordred(just went and checked)21:17
ianwfungi: the new model of sshing and running vos release under localauth on the server seems to have been effective, i'm not sure we've seen any of our locked volumes issues since?21:17
mordredand yes - I agree, the bigger issue would be the project ssh keys - we'd need to be ok with those repos having the ability to run playbooks on bridge21:18
clarkbI don't think corvus wanted to remove https://review.opendev.org/#/admin/groups/2030,members essentially21:18
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645221:18
mordredclarkb: so - I'd say yeah we should just wait - but since there's no actual blockers for any of these, maybe it's time to just make a decision as to how we want that to work?21:19
fungiianw: i haven't seen anyone raise issues related to it, at least21:19
mordredclarkb: like - it's been a theoretical problem until now since we haven't been doing this :)21:19
corvusclarkb, mordred: i'm having a really hard time keeping up with what you're talking about because i'm working on various other things21:20
corvusclarkb, mordred: do i need to drop that and just hang out with you?21:20
mordredcorvus: it's ok - I'm having a tough time keeping up with it too21:20
mordredcorvus: nope21:20
clarkbcorvus: I don't think its urgent21:20
clarkbmordred: I think the ideal here is to have a repo trigger a job within the context of another repo21:20
fungimordred: clarkb: looking at the state of the zone repos, they're included in the opendev and zuul tenants without reading any config, they're regular untrusted projects in the openstack tenant right now21:20
clarkbrather than within its own context21:20
clarkbfungi: yup we would use openstack/ to drive things I think21:21
mordredclarkb: yeah - I don't think that triggering concept exists today - it's sort of the very undefined "subscribe to project" feature I keep daydreaming21:21
clarkbmordred: because the real gap here is giving anyone with merge rights to repo A ability to get access to what repo B has access to. What we want to express is that repo A can tell repo B it should do its predefined things21:21
corvusmordred, clarkb: still not sure i'm up to speed but https://review.opendev.org/671637 is relevant21:21
mordredcorvus: yes!21:22
mordredcorvus: cool. thanks21:22
mordredclarkb: ok - so let's stick that patch at the end to give ourselves time to think on this topic further21:22
clarkbya  I think that change is one way of expressing what I just said21:22
openstackgerritMerged openstack/project-config master: Add Fedora 31 builds to nb04  https://review.opendev.org/71612721:22
mordredbut it seems like there is a thing that's made some progress21:23
clarkbmordred: ++21:23
corvuswell, that patch may be dead in the water21:23
corvusi have not performed the followup analysis it needs after logan- raised that point21:23
clarkbcorvus: ya I think what I head in mind was more of a trigger mechanism internal to zuul21:24
clarkband that may be less of a security concern (its basically run this predefined thign with no updates)21:24
corvusnot sure that's different :)21:25
openstackgerritMonty Taylor proposed opendev/system-config master: Run nodepool in zuul  https://review.opendev.org/71677021:25
openstackgerritMonty Taylor proposed opendev/system-config master: Run meetpad in zuul  https://review.opendev.org/71677121:25
openstackgerritMonty Taylor proposed opendev/system-config master: Run mirror-update in zuul  https://review.opendev.org/71677221:25
mordredclarkb: rebased nameserver out of the stack21:26
fungioh, the thing we were discussing at the shanghai ptg21:26
funginow i remember21:26
mordredclarkb: I left a comment on the nameserver patch with a link to the zuul patch and marked it WIP fo rnow21:27
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645221:27
clarkbcorvus: I think the key difference is running the job in its original context. Eg this isn't running nameserver-playbook job with zone-zuul's context and a borrowed ssh key. Its zone-zuul emits an event saying "I merged", system-config can then say "run nameserver-playbook in my context on this pipeline after that event"21:27
clarkbits possible that that doesn't chagne things internally21:28
corvusclarkb: yeah, i think we all want the same thing to happen.  i think we've established it's a hard change.21:28
corvusclarkb: i think a literal implementation of what you're suggesting would be "allow projects to attach their own jobs to other project pipelines" which is also a *big* change with many pitfalls.21:29
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645221:31
clarkbI keep thinking of it as a new trigger thing. Where maybe we can say its a change-merged trigger in pipeline config, then in project config we say promote subscribes to project 1 project 2 project 3. Then when it runs we get the change info for what merged but we still execute as if it were the "hosting" project21:32
clarkb(and ya thats a big change to things like mergers and executors)21:32
mordredyeah - I keep thinking of it like a new trigger as well - because putting a job into another project's pipeline isn't the thing we want to do - we want to start running a job in our own pipeline but triggered when an external event happens ... and I totally agree, it's likely a huge and spidering change that I can't even begin to comprehend the ramifications of21:33
clarkbin this specific example we don't actaully care about teh triggering project at all21:33
corvusokay i will task switch to talking about this problem21:33
clarkball we care is that something we know we care about merged and we should run the job21:33
clarkb(which gets tricky in zuul because zuul does care about that state)21:34
*** xavinux has quit IRC21:34
mnaserbtw -- thanks for the whole repo with secrets, i am able to centralize it for uploading docker image ssuccessfully!21:34
mordredmnaser: woot!21:34
corvusclarkb: i don't understand the following things you wrote at 21:32: "pipeline config" "project config" "promote subscribes"21:34
clarkbcorvus: let me try and mock it up in etherpad. Will be a few minutes21:35
mnaserand we can confirm that the acls were updated too: https://review.opendev.org/#/admin/projects/vexxhost/base-jobs,access21:35
corvusclarkb: i think just using more words there would help.21:35
clarkbcorvus: the first bit of it is either a new trigger or modifying existing triggers to apply more broadly than a project:project match21:36
mordredcorvus: in my brain this is akin to the SF trigger plugin - there is a trigger source that something can hit that causes one or more jobs to run for a project. except in this case it's not an external party hitting a rest endpoint - it's an internal zuul event - although honest to god it could be a post job in a zone repo pinging a rest endpoint to say "please run this job"21:37
corvusclarkb, mordred: zuul is driven by git events.  this is a git event.  the event is that a change in project A merged.  we want it to run a job which is defined in project B.  that is straightforward and there is a model for that.21:37
clarkbcorvus: then in your system-config zuul.yaml project: config you can say under promote: job listings that "this job subscribes to this trigger event happening for projects x y and z"21:37
corvusit's not an internal zuul event21:37
corvusthe trigger is literally that a commit to a git repo that zuul manages has merged.21:37
clarkbcorvus: yes, the main change is updating how we map those unto project: job listings in system-config zuul.yaml21:38
corvuszuul understands things in terms of git commits to projects.  to try to bend this into an arbitrary external trigger is the wrong approach.21:38
clarkbI don't know if that needs a new expression or not on the pipeline trigger side21:38
corvusclarkb: yeah, i grok that.  it's hard, probably not impossible.21:38
clarkbbut then you can basically say "I know this event is for another project but I don't care, run this job in the context of the current "host" sate"21:39
corvuswhat's a "host" ?21:39
clarkbcorvus: in this case system-config's promote: pipeline entry for playbook-nameserver job saying "I want to run when zone-foo mergers"21:39
clarkbcorvus: the executors and mergers would prep the job for system-config repos and not use speculative state from zone-foo21:39
corvusi still don't understand what a host is21:39
clarkbcorvus: its the context for which the job is running in. We aren't switching that like we would in a setup today21:40
corvusclarkb: running the job is the easy part21:40
fungihost as in the project hosting the job?21:40
corvusclarkb: i get and totally agree that "run this job without the ability for zone-foo to muck it up" is desirable.  that's why my change has a WIP-1 on it.21:41
clarkbfungi: yes21:41
clarkbif I understand the security concerns in that job the issue is we're running the job in the context of the "other"21:42
corvusclarkb: let's try this approach.  you're suggesting that a merge event on zone-foo should enqueue an item into that pipeline.  it also sounds like you think that the project attached to that item should be "system-config", not "zone-foo".  what ref would you attach to that item?21:42
clarkbit decides when the job runs21:43
clarkband all of the zuul dict data in the job is tied to that repo21:43
mordredI thnik the ref should be head of master of system-config21:43
clarkbcorvus: yes that is what I'm suggesting21:43
mordredlike if it was the periodic pipeline21:43
clarkbmordred: yup21:43
clarkbexactly that21:43
clarkbbecause what is important here is that the job get run, but not any new or speculative state21:44
corvuswhy?  given that the triggering event is "a change to stable/blarg on zone-foo has merged" why should we enqueue a system-config@master change into a pipeline?21:44
corvusit seems to me that we're really focused on "how to we most expeditiously get this bit of code to run" and not at all thinking about "what is the right way to fit this into the zuul data model"21:44
clarkbcorvus: because system-config knows to grab latest zone-foo and apply it, it doesn't need the specifics. I expect this would be a fairly common scenario for more CD type operations21:44
corvusclarkb: sure, but why do we have to break the zuul model for that?  why can't we take a minute and see if we can actually encode what's really going on?21:45
clarkbcorvus: we don't necessarily need to21:45
corvusbecause what's really going on is that a change to some branch of zone-foo has merged. *that* is the item that should be enqueued into the pipeline21:45
corvusyes, there's a piece missing because we can't do what we want21:46
corvusbut let's see if we can figure out that missing piece in the existing model21:46
clarkbcorvus: I agree with the first part. I think the desired end result is that a specific job be run. I'm not sure that the second bit is necessary to achieve the use case. But it would be more consistent with zuul's existing model21:46
clarkbif we can solve the problem within that existing model thats great and I have no objections21:47
corvushow urgent is this?21:49
clarkbI don't think it is critically urgent21:50
clarkbwe can either keep nameserver updates happening in cron or add a new periodic pipeline with shorter cycle period and run it there too21:50
clarkbneither option is a regression compared to the current setup and the second allows us to push it into zuul anyway21:50
mordredyeah - this came up because I've been pushing up patches to zuul-ify run_all and clarkb brought up that we should wait on doing the nameserver playbook21:51
corvusre-reading the change, i think logan-'s concern may be fixable with the same fix we made to secrets: allow project-config repos to attach jobs to other projects, and drop the idea of doing this with allowed-projects21:51
corvusclarkb, mordred: let's say we come to agreement that we should have some sort of "trigger on other project" facility that lets zone-foo enqueue a system-config item of some kind21:52
corvuswhat pipeline should that go into, and where should zuul report the results for that?21:52
mordredcorvus: I would imagine it would potentially need to be its own pipeline, and I'd imagine the results would just go to the dashboard21:53
clarkbcorvus: to me I think that goes to system-config. The context of the job is there, its where people will look for why the dns update didn't happen etc. All that zone-foo is providing is the trigger that this happen21:53
corvus(i think that it should report the results on the zone-foo change, that way we can look back at that change, and see what happened when it merged; that's a big driver for why i think the current model is the correct one).21:53
corvusclarkb: how does it "go to system-config"?  what change in system-config does it report on?21:54
clarkbcorvus: right thats where it breaks the zuul model21:54
corvusmordred: earlier you linked to a change which changed manage-projects, and ran manage-projects, and the results were all there.  i thought that was cool.21:54
corvusi'd like to keep doing that.21:55
mordredyes - I thnk that's GREAT21:55
mordredbut I think that, for some reason, we are not comfortable with that model here21:55
mordredbecause we don't trust the zone repos the same way we trust the project-config repo21:55
corvusmordred: why are we not comfortable?21:55
clarkbI think where things get weird here is system-config is affecting the state chagne so that is where I expect to observe the state change21:55
mordredcorvus: I have no clue21:55
corvusmordred: no i don't think that's it21:55
mordredit's not?21:55
corvusmordred: i'm the one not comfortable with a thing, let me explain it21:55
mordredcool. I think I've misunderstood something fundamental21:56
corvusmordred: the thing from last july that we're not comfortable with (which i raised) was that we wanted to allow the zuul project to maintain its own zone repo, and without doing something that gave them access to prod servers21:57
corvusmordred: it's definitely not the case that we're not comfortable reporting run results on changes to the zuul zone repo.  i think that is very desirable.21:58
clarkbright the concern is adding zuul zone's ssh key to bridge21:58
mordredoh - no - that's not the thing that I thought we were uncomfortable with21:58
corvusclarkb: yeah, that's it exactly21:58
mordredright21:58
clarkbwhile also allowing the zuul project to manage that directly rather than limiting it to infra-root21:58
mordredthat's what I meant by "we don't trust the zone repo enough"21:58
clarkbwe could do that if we didn't let zuul maange the zone file direclty21:58
mordredwe dont' have the same trust relationship between system-config and zone as we do between system-config and project-config - because we ARE willing to put project-config key on bridge, but not zone21:59
corvusmordred: okay. so from a high-level POV, i think we probably all agree that if we can have the zone-foo trigger a system-config job and report the results on zone-foo, that's what we want.  *how* to accomplish that is up in the air and probably complicated.21:59
mordredyah.22:02
mordredthat said ...22:02
corvusbut if we agree that's the ideal result, then i think it's worth looking at something like 671637, and exhausting that solution space before we take the other approach.22:02
mordredfor now, the zuul zone repo is not managed by the zuul team22:02
*** frickler_ has joined #opendev22:03
mordredso I think it is safe at the moment to do the job manage-projects-style from an opendev POV - and the work that we're talking about is actually work to enable to use give maint to the zuul team of the zuul-zone repo, right?22:03
corvusmordred: then maybe we should just put its keys on bridge and call it done?  :)22:03
mordredyeah. I'm thinking all of this may have just been a long conversation to get us to that point ;)22:03
clarkbit does technically have its own group22:03
clarkbits just the only members of that group are us22:03
mordredit does - but the onlymembers are infra-core22:03
mordredyeah22:03
mordredso - I think we'd have to be the ones to change the membership22:03
clarkbI do think this conversation is an important one to keep thinking on though22:04
mordredso it's still fully withing our trust domain22:04
mordredYES22:04
mordredit's _very_ important22:04
clarkbbecause this is likely to be fundamental to zuul's cd useablity22:04
mordredbut I think we can unblock ourselves on this particular task because of where we happen to be currently22:04
corvusclarkb: in the original zuulv3 design, there's supposed to be a tenant ssh key22:05
corvusclarkb: just doing that would probably solve this for 98% of the world22:05
corvusclarkb: leaving our little corner of "projects that share a tenant in a publicly accessible zuul but don't trust each other" needing a solution22:06
corvus(like if you're in a corp, and your team has a zuul tenant with all your microservices or whatever, just add the tenant ssh key to your prod system)22:07
clarkbya22:07
*** frickler has quit IRC22:08
openstackgerritMonty Taylor proposed opendev/system-config master: Add zone keys to zuulcd user  https://review.opendev.org/71678122:20
mordredclarkb, corvus : ^^22:20
clarkbmordred: should we also update the acl for the zone files to explicitly make it infra-root for now?22:21
mordredclarkb: I mean - we could - but since the only member is infra-root - I don't know that it's that much different?22:21
* mordred does not have an opinion one way or the other22:21
clarkbmordred: I think it would mostly just be a reminder that "this repo is in the class of things with root access"22:22
mordredseems reasonable22:22
*** DSpider has quit IRC22:29
openstackgerritIan Wienand proposed openstack/project-config master: zuul-worker element: use python3-libselinux for fedora  https://review.opendev.org/71678322:34
ianwinfra-root: ^ if we could look at that one, it will stop a broken fedora31 build loop -- i'm going to propose we remove it all in a follow-on anyway22:34
openstackgerritMerged zuul/zuul-jobs master: local-log-download : role with script to download all log files  https://review.opendev.org/71575622:49
openstackgerritMerged openstack/project-config master: zuul-worker element: use python3-libselinux for fedora  https://review.opendev.org/71678322:50
openstackgerritIan Wienand proposed openstack/project-config master: zuul-worker: remove python-apt & libselinux deps  https://review.opendev.org/71678522:58
*** hashar has quit IRC22:59
openstackgerritIan Wienand proposed openstack/project-config master: zuul-worker: remove python-apt & libselinux deps  https://review.opendev.org/71678523:01
*** tosky has quit IRC23:03
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Debuntu: add apt-transport-https  https://review.opendev.org/71678823:11
openstackgerritIan Wienand proposed openstack/project-config master: zuul-worker: remove additional install of apt-transport-https  https://review.opendev.org/71678923:14
clarkbmordred: ianw fungi corvus https://review.opendev.org/#/c/716660/ any chance you want to give that a try on meetpad (use etherpad-dev.openstack.org)23:20
clarkbalso  Ithink we can remove meetpad from the emergency file?23:20
clarkbI want to say the chagne corvus said it needed has merged23:20
mordredclarkb: did you want to switch that to point to the etherpad-dev.opendev.org ?23:21
mordred(didn't you make a cname for that?)23:21
fungishould i switch ep-dev back to the "no-skin" undefault or did people want to play with the colibris skin some more?23:21
clarkbmordred: I think it would probably be helpful to do it one at a time23:21
mordredkk23:21
clarkbmordred: so that we can identify which issue it is if this or that fixes it23:21
clarkbfungi: I prefer the old one personally23:21
fungias do i23:21
clarkbI think the old one will be better for meetpad too23:21
mordred+2 from me23:22
clarkbmordred: if new etherpad doesn't work we'll switch to opendev.org and see if domain change fixes it23:22
fungiprobably so. i'll yank that line back out of its config and take it back out of the emergency list then23:22
clarkbmordred: but if we land both we won't know which is the thing :)23:22
mordredgood point :)23:22
mordredalso - if y'all feel like doing more reviews of https://review.opendev.org/#/q/topic:infra-prod-zuul - I can start landing them in the morning23:22
fungiokay, ep-dev is back to no-skin23:23
fungiclarkb: mordred: one problem to note with using etherpad-dev.opendev.org is that it's going to need a vhost change23:25
clarkbfungi: oh do we not * it there?23:26
fungiRewriteRule ^/+(.+)$ https://etherpad-dev.openstack.org/p/$1 [NC,L,R=301]23:26
clarkbI was hoping we'd just need to override the ssl cert validation23:26
clarkbaha23:27
clarkbok if we get to that point we can maybe just ninja that for a few minutes and test23:27
clarkb(we can probably just add a second vhost and s/openstack/opendev/)23:27
fungiwe could probably drop the https://etherpad-dev.openstack.org from that and just rewrite to /p/$1 with no hostname?23:27
corvusclarkb: i will remove meetpad from emergency23:27
clarkbcorvus: I'm approving https://review.opendev.org/#/c/716660/ then. thanks!23:28
corvusdone23:28
fungii'll readd etherpad-dev to the emergency file now though in case folks want to fiddle with the vhost config locally on it23:28
clarkbmordred: I suddenly had a worry that the emregency file would stop working with the zuul jobs driving things, but then I remembered we are still running ansible on bridge so that is all fine.23:29
clarkb(this is me talking out loud so that other reviewers can either confirm or reject that statement :) )23:29
clarkbmordred: re https://review.opendev.org/#/c/716771/2/.zuul.yaml similar to dns, how do we get that to run when docker images have updated. I think this is much simpler since it is all in the same repo23:30
ianwmordred: q inline for https://review.opendev.org/#/c/716745/1 ... why have a semaphore of 10 for service-bridge?23:32
clarkbianw: its for all the jobs that parent to that base job iirc. That way we limit the number of ansible processes on bridge23:32
clarkbthey won't directly conflict with each other but resource consumption may be something to keep in mind23:33
ianwclarkb: hrm, so do we have a lock to ensure service-bridge won't run ontop of itself?23:34
clarkbianw: the jobs run in supercedent promote pipeline. Though now that you've asked they also run in periodic and we may need to keep promote and periodic from overlapping23:35
fungiit's only triggered from one branchless repo in a supercedent pipeline, right?23:35
clarkbmordred: ^ we may need the per job semaphore afterall23:35
fungioh, indeed, if periodic is also in use23:35
ianwyeah, i feel like if it can happen, it will :)23:36
clarkbif we only ran in promote it would be fine23:36
clarkbdue to supercedent pipeline behvaior23:37
fungithat said, what's the actual risk if it runs twice at the same time?23:37
fungiwhat's it going to break?23:38
fungior is it just that we don't know, effectively undefined behavior, better safe than spend a weekend indoors?23:38
clarkbfungi: I think the big risk is in things like service restarts23:38
clarkbwith gitea in particular we ensure that we stop a single backend at at time and do the stops in a very coordinated fashion23:38
fungiahh, yeah, strict ordering23:39
clarkbwe could break replication again because we are trying to stop the same backend multiple times (breaking the careful stop start ordering)23:39
ianwalso in the later changes, letsencrypt really doesn't want to run over itself .. they would race to update acme.opendev.org records23:41
clarkbI expect what we want to do is add a semaphore per job then it can run in periodic and promote23:44
clarkband that sempahore will have a count of 1 making it a mutex23:44
clarkbthen we can have a second semaphore with a large limit to reduce too many jobs running at once on bridge23:44
ianwyep23:46
ianwas fungi points out many would be pretty close to very idempotent, but still i bet there's plenty of corner cases23:47
clarkbya and I expect they'll be difficult to debug23:48
clarkbwe'll just get really weird behaviors occasionally and not be able to track them back to anything specific23:48
openstackgerritMerged opendev/system-config master: Switch meetpad to etherpad-dev  https://review.opendev.org/71666023:51
openstackgerritMerged zuul/zuul-jobs master: golangci-lint: add job  https://review.opendev.org/71645223:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!