Friday, 2021-04-30

*** mlavalle has quit IRC00:04
ianwclarkb: i'll drop all those autoholds now? (and remove that testing host?)00:11
*** SWAT has joined #opendev00:24
openstackgerritIan Wienand proposed opendev/system-config master: nodepool clouds: adds stats prefix  https://review.opendev.org/c/opendev/system-config/+/78885701:10
openstackgerritIan Wienand proposed opendev/system-config master: nodepool clouds: adds stats prefix  https://review.opendev.org/c/opendev/system-config/+/78885701:12
*** hamalq has quit IRC01:30
kevinzianw: Good to hear that :-)01:48
ianwyay, i'll see if we can get the dib/glean gate moving :)03:01
clarkbianw: ya I don't think we need to catch anymore at this point03:22
ianw++03:39
*** redrobot6 has joined #opendev03:52
*** redrobot has quit IRC03:55
*** redrobot6 is now known as redrobot03:55
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing using gr-date-formatter in zuul plugin  https://review.opendev.org/c/opendev/system-config/+/78888204:20
openstackgerritMerged openstack/diskimage-builder master: Remove dib-nodepool-functional-openstack-ubuntu-bionic  https://review.opendev.org/c/openstack/diskimage-builder/+/78840404:26
openstackgerritMerged openstack/diskimage-builder master: Fix centos stream set mirror  https://review.opendev.org/c/openstack/diskimage-builder/+/78453005:21
*** ysandeep|away is now known as ysandeep05:30
*** ralonsoh has joined #opendev05:37
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] testing using gr-date-formatter in zuul plugin  https://review.opendev.org/c/opendev/system-config/+/78888205:54
openstackgerritMerged openstack/diskimage-builder master: Collect openstack logs  https://review.opendev.org/c/openstack/diskimage-builder/+/78855305:58
openstackgerritMerged openstack/diskimage-builder master: debian-minimal: Set bullseye version  https://review.opendev.org/c/openstack/diskimage-builder/+/78766505:59
*** slaweq has joined #opendev06:00
hrwmorning06:20
hrwso, with above merged, we need to generate new images and then bullseye nodes should work?06:20
fricklerhrw: ianw: I'd assume we need a fresh dib release in order for our nodepool to pick up these fixes?06:51
*** fressi has joined #opendev06:54
*** sboyron has joined #opendev07:05
*** eolivare has joined #opendev07:09
*** hashar has joined #opendev07:17
ianwfrickler: yes, dib release, then bump nodepool so we build new docker images.  there is one outstanding atm though07:23
ianwhttps://review.opendev.org/75599207:23
ianwno, not that one sorry07:23
ianwhttps://review.opendev.org/c/openstack/diskimage-builder/+/78745407:24
*** rpittau|afk is now known as rpittau07:24
ianwdib-functests-bionic-python3 TIMED_OUT07:24
ianwsigh07:24
ianwi'll try to pop back later and see07:24
*** DSpider has joined #opendev07:26
*** jpena has joined #opendev07:35
*** amoralej has joined #opendev07:45
hrwthx08:05
*** andrewbonney has joined #opendev08:14
*** tosky has joined #opendev08:21
*** elod is now known as elod_afk08:47
*** hashar has quit IRC08:47
openstackgerritMerged openstack/diskimage-builder master: debian-minimal: bullseye: /updates -> -security  https://review.opendev.org/c/openstack/diskimage-builder/+/78745409:25
fricklerianw: if you are still around/awake ^^09:34
*** sboyron has quit IRC09:37
cennewhat is zuul's frequency?09:59
cennelike after how many hours can i expect it to re-run tests/build09:59
*** dtantsur|afk is now known as dtantsur10:01
*** sboyron has joined #opendev10:14
*** brinzhang has quit IRC10:21
fricklercenne: except for special periodic jobs (of which we have hourly, daily and weekly ones), zuul doesn't re-run jobs automatically. if things failed for a patch in gerrit and you assume that the failure has been fixed, you can trigger new jobs by leaving a "recheck" comment10:28
fungiwell, also zuul tests everything again when it's approved10:47
fungiwhich addresses the risks non-gating ci systems try to solve with occasional retesting10:48
*** dtantsur is now known as dtantsur|brb11:05
ianwi'll tag a dib 3.11.0 which should be bullseye-capable :)11:24
*** jpena is now known as jpena|lunch11:30
fricklerinfra-root: do we have an opendev group registered here on freenode? I've been thinking for some time on getting an "openstack" cloak, but maybe "opendev" would be even nicer11:30
fricklermordred: on a related note, do you still know who made that openstack cloak for you? or are you yourself the group contact still?11:31
ianwBump dib to 3.11.0  https://review.opendev.org/c/zuul/nodepool/+/78893711:33
ianwshould get us going with bullseye on all platforms.  i'm out for now, ttyl11:33
fricklerianw: ack, I'll approve once tests pass11:35
* frickler found http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-01-05.log.html about cloaks and puts hope onto corvus 11:45
*** sshnaidm is now known as sshnaidm|off11:59
*** amoralej is now known as amoralej|lunch12:19
*** dtantsur|brb is now known as dtantsur12:28
*** hashar has joined #opendev12:31
*** jpena|lunch is now known as jpena12:34
*** dhellmann_ has joined #opendev12:41
*** dhellmann has quit IRC12:41
*** dhellmann_ is now known as dhellmann12:42
*** amoralej|lunch is now known as amoralej13:07
*** ysandeep is now known as ysandeep|away13:11
*** dhellmann has quit IRC13:24
*** dhellmann has joined #opendev13:35
*** owalsh has quit IRC13:54
*** mlavalle has joined #opendev13:58
*** nicolasbock has joined #opendev14:03
*** whoami-rajat has joined #opendev14:04
*** fressi has quit IRC14:09
*** rpittau is now known as rpittau|afk14:10
*** owalsh has joined #opendev14:13
clarkbfrickler: I want to say that corvus set that up when we added the new channel a while back14:15
clarkboh ya last line logged says you found scrollback with that info14:15
*** dhellmann has quit IRC14:16
corvusi've never done a cloak; is that something we want to get into the business of?14:23
corvusi don't really want to handle it as one-off, so if we do, we should establish a procedure14:23
clarkbI use a dedicated host for my freenode conenction so haven't really bothered. I know a lot of people like having them though14:24
fungiany clue what the implications are? like if we were to hand them out to just anybody are we responsible for havoc they wreak?14:25
corvusoh yeah, no judgement there.  anyone can get a freenode cloak if they're worried about that.14:25
corvusi sort of understood the desire for an openstack/opendev cloak as a badge14:26
corvusso if we're going to hand out badges, we should have a badge policy :)14:26
clarkbgot it14:26
*** avass has quit IRC14:27
*** avass has joined #opendev14:28
corvus(i'm now visualising a cloak with a badge sewn on to it; this is getting weird)14:28
corvus(it really clashes with fungi's shirt)14:29
fungii'll stitch my merit badges onto my backpack instead14:31
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (12)  https://review.opendev.org/c/openstack/project-config/+/78857214:34
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (13)  https://review.opendev.org/c/openstack/project-config/+/78857314:34
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (14)  https://review.opendev.org/c/openstack/project-config/+/78857414:34
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (15)  https://review.opendev.org/c/openstack/project-config/+/78857714:35
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (17)  https://review.opendev.org/c/openstack/project-config/+/78858014:46
openstackgerritElod Illes proposed openstack/project-config master: Move projects under meta-config acl (16)  https://review.opendev.org/c/openstack/project-config/+/78857814:50
fungielod_afk: thanks, once we get those last few in i'll start on a cleanup change for acls which are in the wrong namespaces relative to their corresponding repositories15:08
clarkbfungi: were there more than just the starlingx ones?15:08
fungiroughly a dozen, yes15:08
clarkbinteresting15:09
fungii cheated and used some python to find them15:09
fungia bit one is the xstatic repos, half of which are in x/ but share an acl in openstack/ with others15:09
fungis/bit/big/15:09
clarkbah15:09
fungicases like that, i'll just copy the acl15:09
fungirather than moving15:10
clarkb++15:10
fungialso tempted to move the openstack/project-config acl into opendev/ or something for clarity15:10
clarkbor even a these-projects-are-weird-dir/project-config15:15
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder master: Update the ironic jobs  https://review.opendev.org/c/openstack/diskimage-builder/+/78897215:15
mordredfrickler: my cloak came from Rick Clark (so it's very very very old) and pre-dates corvus having the org contact role15:22
mordreds/Rick Clark/dendrobates/ for anyone who happened to only know him in IRC15:23
mordredI agree with corvus though - it seems like a thing where we should have some policy/structure around handing out/managing15:23
mordredan opendev cloak would also be neat :)15:24
*** hashar has quit IRC15:36
*** mlavalle has quit IRC15:37
elod_afkfungi: thanks, too! :)15:40
yoctozeptomorning folks15:45
yoctozeptois there a way to make opendev apt mirrors *signed*?15:46
yoctozeptoI mean, we have this image production-gradiness discussion on kolla and we now disable key verifications15:47
fungiyoctozepto: probably. how they're signed would differ by distro though15:47
fungiyoctozepto: but keep in mind that the mirrors we run aren't official, if we signed them we'd sign them with an unofficial key which wouldn't be valid for official distro mirrors15:47
yoctozeptoyeah, but can't we have the exact repository info with signature from the original source?15:48
yoctozeptoassume I am apt packaging noob15:48
clarkbfor ubuntu and debian you can't unless you accept broken mirrors as the compromise15:49
clarkbthe reason our mirrors are not signed is that we generate our own indexes to fix that brokeness and don't bother to resign with our own keys (we could do that, but that wouldn't be the same key as the upstream)15:49
fungiwe could in theory work around that by testing the mirrors before we vos release them, and delay updating until they're coherent, but someone would need to write that automation15:50
yoctozeptoall right, thank you for the answers! now I understand15:50
clarkbthats a good point. You run the risk of not updating as frequently, but could release an always functional state with the appropriate checks beforehand15:50
yoctozeptoit also explains how we end up in those weird situations in CI15:51
yoctozeptothat we have working repo, yet it lacks required packages15:51
yoctozepto(e.g. app updated but its libs not)15:51
yoctozeptoand repo is happy15:51
yoctozeptowell, we don't have the manpower to offer for that particular task15:52
clarkbfungi: though I don't think you can address the stale update on client side that way?15:52
fungiyoctozepto: also we've treated the lack of signatures on our mirrors as a feature, to dissuade people from configuring their production servers to use our ci system mirrors15:52
clarkbfungi: one of the things we do is keep old package around for a cycle or too so that you don't break if you lose the race between apt-get update, repo update, and package pull/install15:53
fungiclarkb: stale update on the client side would be addressed the same way we do now: delay deletions15:53
clarkbfungi: ya so its multiple pieces we'd need though right? one to validate index against all packages and another to track old packages and delete them after some time delta?15:53
fungiwe'd mirror with something like rsync, and only delete on the next pass after a successful vos release, or something along those lines15:54
fungiand yeah, coax rsync into dumping a list of proposed deletions to a file or something like that15:54
fungibut have it not actually delete15:55
clarkbanother option is to convince debuntu to do per package signing >_>15:56
clarkb(I don't think that is a serious option though)15:57
yoctozeptobut the cleanest one nonetheless ;-)15:57
fungitechnically all uploaded packages are signed, but those signatures aren't used by the client, they're for tracking provenance of the builds15:57
*** mlavalle has joined #opendev16:03
mordredyeah - I think if we could figure out the delay-deletion-rsync thing - we could apply that in a few different places16:10
mordredbut definitely non-trivial - we'd wind up needing to keep a local db of file states probably, so that we could attach a timestamp to the first time a file was flagged as needing deletion16:11
mordredand then for deletion passes we could get the list of files whose deletion flag date was > $period16:12
mordredoh! you know what would be a clever way to do that that wouldn't require an extra db? make a hardlink tree for each sync operation just containing the files that rsync wants - then periodically delete old hardlink trees - so you'd have the hard link reference count actually take care of removing files no longer referenced16:14
openstackgerritMerged openstack/project-config master: Add glance-tempest-plugin to publish-to-pypi job  https://review.opendev.org/c/openstack/project-config/+/78825016:14
clarkbdoes afs do hardlinks?16:14
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (18)  https://review.opendev.org/c/openstack/project-config/+/78858116:14
mordredoh - I think it doesn't16:16
mordrednevermind :)16:16
fungiprobably the way to do it would be to run rsync once without --delete, then with --delete and --dry-run together and output the results to a timestamped file, then look for any timestamped lists at least x seconds older the most recent vos release and explicitly rm the files they say would have been deleted at those times16:17
fungimordred: also, technically a filesystem *is* a database, so... ;)16:17
mordredfungi: :)16:17
mordredfungi: have I told you about the time I accidentally facilitated a debate between krow and linus about filesystems vs databases?16:18
fungihah, that sounds frightening16:18
clarkbfungi: but worth watching16:18
mordredkrow was pro-filesystem and linux was pro-db :)16:18
mordredlinus16:18
JayFDid the venue supply the plate armor or did you have to bring your own :D16:21
mordred:D16:21
mordredJayF: it turns out, this is the consequence of writing a talk with an inflammatory title and then not having enough content to actually fill the talk slot. so the plate armor was actually withheld purposefully16:22
JayFoh god that's like, conference fear #116:22
fungimine is that they supply the audience with crates of old produce16:23
fungiand a working trebuchet, because those are always fun16:24
*** dtantsur is now known as dtantsur|afk16:27
yoctozeptolol16:32
*** jpena is now known as jpena|off16:38
mordredfungi: now I really want that to be a conference feature16:47
fungijust warn me not to submit to that cfp16:51
*** hamalq has joined #opendev17:19
*** hamalq has quit IRC17:20
*** hamalq has joined #opendev17:21
*** stephenfin is now known as stephenfin|PTOin17:27
*** stephenfin|PTOin is now known as stephenfin|PTO17:28
clarkbfungi: we also want to clean up the all projects acl once we're happy with the state of openstack repos ya?17:44
clarkbfungi: does the new meta project acl already cover everything in all-projects or does it just add deletion?17:45
fungiall17:47
fungiso once it's in place we should be able to clean that up and make sure everything's still working for the openstack release team17:47
clarkbcool so really the only thing we need tis to be satisfied all the necessary project acls are up to date to point at the meta project17:48
clarkb++17:48
*** andrewbonney has quit IRC17:48
fungithere are 2 still not merged: 788578 78858417:48
clarkblooking17:48
fungiworth noting, the openstack/meta-config acl actually grants create and delete on refs/* instead of refs/heads/* because that's what the all-projects config had, though we can probably tighten it up later17:49
fungii did at least switch to pushsignedtag instead of pushtag though17:49
clarkbI've approved 788578 and +2'd 788584. I didn't approve 788584 as that is the largest update and want that to go in separately just to minimize chances it has a sad17:51
clarkb(I believe we'll always use master project-config on those updates so landing them together would add to the list of updates)17:51
fungiyep, i agree17:58
openstackgerritMerged openstack/project-config master: Move projects under meta-config acl (16)  https://review.opendev.org/c/openstack/project-config/+/78857818:01
*** owalsh has quit IRC18:04
*** eolivare has quit IRC18:04
*** jaicaa has quit IRC18:04
*** lbragstad_ has quit IRC18:04
*** fdegir has quit IRC18:04
*** priteau has quit IRC18:04
*** stand has quit IRC18:04
*** calcmandan has quit IRC18:04
*** elod_afk has quit IRC18:04
*** akahat|ruck has quit IRC18:04
*** icey has quit IRC18:04
*** owalsh has joined #opendev18:07
*** eolivare has joined #opendev18:07
*** jaicaa has joined #opendev18:07
*** lbragstad_ has joined #opendev18:07
*** fdegir has joined #opendev18:07
*** priteau has joined #opendev18:07
*** stand has joined #opendev18:07
*** calcmandan has joined #opendev18:07
*** elod_afk has joined #opendev18:07
*** akahat|ruck has joined #opendev18:07
*** icey has joined #opendev18:07
*** gouthamr has quit IRC18:08
*** gouthamr has joined #opendev18:09
fungiwith all but 788584 now merged, i can safely base my cleanup change on top of that one18:25
fungiany idea if vexxhost/base-jobs using gerrit/acls/opendev/project-config.config was intentional?18:29
*** sboyron has quit IRC18:39
*** eolivare has quit IRC18:40
clarkbfungi: I think it started in opendev/ at first maybe?18:45
clarkband then we split out a tenant for them18:45
openstackgerritJeremy Stanley proposed openstack/project-config master: Move some Gerrit ACL files into better namespaces  https://review.opendev.org/c/openstack/project-config/+/78901618:47
fungiwell, those ^ are the easy half18:47
fungithe other half i turned up are all in sticky splits of repos between namespaces18:47
fungimostly cases where one team maintains some repos which are part of openstack and some others which aren't18:48
fungiaside from the vexxhost/base-jobs mentioned above, these seem to be team split situations: charms, monasca, openstack-ansible, puppet-modules and xstatic repos18:51
fungiso would all need a second copy of the acl to associate stuff from the other repo with18:51
fungier, from the other namespace with18:51
clarkband trim out the openstack meta project parentage18:57
fungiyep18:57
fungido we want the opendev sysadmins retaining control over vexxhost/base-jobs?18:57
clarkbI want to say there may have been reasons for that at the time18:59
clarkbbut I'm not remembering them18:59
*** slaweq has quit IRC19:18
corvusi'd like to restart zuul, how's that sound?19:37
corvusgreat, i'll get started then :)19:39
corvus(release queue looks empty; load is around 50%)19:40
corvus#status log restarted zuul on commit b9a6190a452a428da43dc4ff3e6e388d4df41e8b19:41
openstackstatuscorvus: finished logging19:41
clarkbcorvus: no objections (seems I'm late anyway) I'm popping out for some exercise though19:42
corvusrestart may be a smidge slower as the fleet is starting with cold git repo caches19:47
fungiyeah, a belated thumbs up and thanks for the restart!19:51
fungisorry, dinner prep has distracted me19:51
corvusgrafana says we're about 1/2 through the merge jobs19:54
corvusand we will take a brief pause as several mergers all clone nova around the same time :)19:55
corvusre-enqueing20:04
corvusinfra-root: this moves another chunk of functionality into zk, so worth being aware of that in case of performance impacts (build results are returned via zk instead of gearman)20:05
fungithanks for the heads up!20:15
*** ralonsoh has quit IRC20:37
*** auristor has quit IRC20:38
*** auristor has joined #opendev20:39
clarkbcorvus: a "brief" 7 minute pause21:36
corvusit actually didn't slow down that much21:37
*** erbarr has quit IRC21:37
corvusif you look at the graph, it's mostly a constant slope down21:37
corvusi guess the number of mergers is sufficiently greater than the number of nova branches that the nova cat jobs weren't a big speed bump21:38
*** erbarr has joined #opendev21:40
clarkbnice21:49
*** DSpider has quit IRC22:56
*** tosky has quit IRC23:31
TheJuliaQuestion, has anyone tried turning off the #openstack-unregistered stuff?23:31
clarkbI don't think I'm in the unregistered channel anymore due to a recent upgrade of my bouncer host but prior to that it would still get ~weekly spams assulats23:32
* clarkb looks at eavesdrop23:32
TheJulia:(23:35
clarkbhttp://eavesdrop.openstack.org/irclogs/%23openstack-unregistered/%23openstack-unregistered.2021-04-20.log.html#t2021-04-20T14:44:04 and http://eavesdrop.openstack.org/irclogs/%23openstack-unregistered/%23openstack-unregistered.2021-04-09.log.html#t2021-04-09T09:47:23 (though that second one may be legit, I'm not going to go trying it)23:36
clarkbthats not too bad considering what it was like. Also looks like we lost logging this week23:36
clarkbthe openstack bot is in that channel so why are there no logs23:37
clarkbfungi: ^ other channels (like this one) seem to continue to log just fine. Any ideas?23:38
clarkbTheJulia: given there appears to have only been ~1.5 spam incidents in the last several weeks I'd be inclined to try undoing the registration requirement. Will let others chime in though23:39
TheJuliaI'm just thinking it could be a barrier for easiest entry since few grow up with IRC these days :)23:41
clarkbTheJulia: maybe add it to the infra meeting agenda and we can bring it up there? or send a note to service-discuss@lists.opendev.org23:42
clarkbI'll try to remember to bring it up at the meeting either way but I've got 5 year olds that want outdoor time now23:43
TheJuliaenjoy :)23:52

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!