*** mlavalle has quit IRC | 00:04 | |
ianw | clarkb: i'll drop all those autoholds now? (and remove that testing host?) | 00:11 |
---|---|---|
*** SWAT has joined #opendev | 00:24 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool clouds: adds stats prefix https://review.opendev.org/c/opendev/system-config/+/788857 | 01:10 |
openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool clouds: adds stats prefix https://review.opendev.org/c/opendev/system-config/+/788857 | 01:12 |
*** hamalq has quit IRC | 01:30 | |
kevinz | ianw: Good to hear that :-) | 01:48 |
ianw | yay, i'll see if we can get the dib/glean gate moving :) | 03:01 |
clarkb | ianw: ya I don't think we need to catch anymore at this point | 03:22 |
ianw | ++ | 03:39 |
*** redrobot6 has joined #opendev | 03:52 | |
*** redrobot has quit IRC | 03:55 | |
*** redrobot6 is now known as redrobot | 03:55 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing using gr-date-formatter in zuul plugin https://review.opendev.org/c/opendev/system-config/+/788882 | 04:20 |
openstackgerrit | Merged openstack/diskimage-builder master: Remove dib-nodepool-functional-openstack-ubuntu-bionic https://review.opendev.org/c/openstack/diskimage-builder/+/788404 | 04:26 |
openstackgerrit | Merged openstack/diskimage-builder master: Fix centos stream set mirror https://review.opendev.org/c/openstack/diskimage-builder/+/784530 | 05:21 |
*** ysandeep|away is now known as ysandeep | 05:30 | |
*** ralonsoh has joined #opendev | 05:37 | |
openstackgerrit | Ian Wienand proposed opendev/system-config master: [dnm] testing using gr-date-formatter in zuul plugin https://review.opendev.org/c/opendev/system-config/+/788882 | 05:54 |
openstackgerrit | Merged openstack/diskimage-builder master: Collect openstack logs https://review.opendev.org/c/openstack/diskimage-builder/+/788553 | 05:58 |
openstackgerrit | Merged openstack/diskimage-builder master: debian-minimal: Set bullseye version https://review.opendev.org/c/openstack/diskimage-builder/+/787665 | 05:59 |
*** slaweq has joined #opendev | 06:00 | |
hrw | morning | 06:20 |
hrw | so, with above merged, we need to generate new images and then bullseye nodes should work? | 06:20 |
frickler | hrw: ianw: I'd assume we need a fresh dib release in order for our nodepool to pick up these fixes? | 06:51 |
*** fressi has joined #opendev | 06:54 | |
*** sboyron has joined #opendev | 07:05 | |
*** eolivare has joined #opendev | 07:09 | |
*** hashar has joined #opendev | 07:17 | |
ianw | frickler: yes, dib release, then bump nodepool so we build new docker images. there is one outstanding atm though | 07:23 |
ianw | https://review.opendev.org/755992 | 07:23 |
ianw | no, not that one sorry | 07:23 |
ianw | https://review.opendev.org/c/openstack/diskimage-builder/+/787454 | 07:24 |
*** rpittau|afk is now known as rpittau | 07:24 | |
ianw | dib-functests-bionic-python3 TIMED_OUT | 07:24 |
ianw | sigh | 07:24 |
ianw | i'll try to pop back later and see | 07:24 |
*** DSpider has joined #opendev | 07:26 | |
*** jpena has joined #opendev | 07:35 | |
*** amoralej has joined #opendev | 07:45 | |
hrw | thx | 08:05 |
*** andrewbonney has joined #opendev | 08:14 | |
*** tosky has joined #opendev | 08:21 | |
*** elod is now known as elod_afk | 08:47 | |
*** hashar has quit IRC | 08:47 | |
openstackgerrit | Merged openstack/diskimage-builder master: debian-minimal: bullseye: /updates -> -security https://review.opendev.org/c/openstack/diskimage-builder/+/787454 | 09:25 |
frickler | ianw: if you are still around/awake ^^ | 09:34 |
*** sboyron has quit IRC | 09:37 | |
cenne | what is zuul's frequency? | 09:59 |
cenne | like after how many hours can i expect it to re-run tests/build | 09:59 |
*** dtantsur|afk is now known as dtantsur | 10:01 | |
*** sboyron has joined #opendev | 10:14 | |
*** brinzhang has quit IRC | 10:21 | |
frickler | cenne: except for special periodic jobs (of which we have hourly, daily and weekly ones), zuul doesn't re-run jobs automatically. if things failed for a patch in gerrit and you assume that the failure has been fixed, you can trigger new jobs by leaving a "recheck" comment | 10:28 |
fungi | well, also zuul tests everything again when it's approved | 10:47 |
fungi | which addresses the risks non-gating ci systems try to solve with occasional retesting | 10:48 |
*** dtantsur is now known as dtantsur|brb | 11:05 | |
ianw | i'll tag a dib 3.11.0 which should be bullseye-capable :) | 11:24 |
*** jpena is now known as jpena|lunch | 11:30 | |
frickler | infra-root: do we have an opendev group registered here on freenode? I've been thinking for some time on getting an "openstack" cloak, but maybe "opendev" would be even nicer | 11:30 |
frickler | mordred: on a related note, do you still know who made that openstack cloak for you? or are you yourself the group contact still? | 11:31 |
ianw | Bump dib to 3.11.0 https://review.opendev.org/c/zuul/nodepool/+/788937 | 11:33 |
ianw | should get us going with bullseye on all platforms. i'm out for now, ttyl | 11:33 |
frickler | ianw: ack, I'll approve once tests pass | 11:35 |
* frickler found http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-01-05.log.html about cloaks and puts hope onto corvus | 11:45 | |
*** sshnaidm is now known as sshnaidm|off | 11:59 | |
*** amoralej is now known as amoralej|lunch | 12:19 | |
*** dtantsur|brb is now known as dtantsur | 12:28 | |
*** hashar has joined #opendev | 12:31 | |
*** jpena|lunch is now known as jpena | 12:34 | |
*** dhellmann_ has joined #opendev | 12:41 | |
*** dhellmann has quit IRC | 12:41 | |
*** dhellmann_ is now known as dhellmann | 12:42 | |
*** amoralej|lunch is now known as amoralej | 13:07 | |
*** ysandeep is now known as ysandeep|away | 13:11 | |
*** dhellmann has quit IRC | 13:24 | |
*** dhellmann has joined #opendev | 13:35 | |
*** owalsh has quit IRC | 13:54 | |
*** mlavalle has joined #opendev | 13:58 | |
*** nicolasbock has joined #opendev | 14:03 | |
*** whoami-rajat has joined #opendev | 14:04 | |
*** fressi has quit IRC | 14:09 | |
*** rpittau is now known as rpittau|afk | 14:10 | |
*** owalsh has joined #opendev | 14:13 | |
clarkb | frickler: I want to say that corvus set that up when we added the new channel a while back | 14:15 |
clarkb | oh ya last line logged says you found scrollback with that info | 14:15 |
*** dhellmann has quit IRC | 14:16 | |
corvus | i've never done a cloak; is that something we want to get into the business of? | 14:23 |
corvus | i don't really want to handle it as one-off, so if we do, we should establish a procedure | 14:23 |
clarkb | I use a dedicated host for my freenode conenction so haven't really bothered. I know a lot of people like having them though | 14:24 |
fungi | any clue what the implications are? like if we were to hand them out to just anybody are we responsible for havoc they wreak? | 14:25 |
corvus | oh yeah, no judgement there. anyone can get a freenode cloak if they're worried about that. | 14:25 |
corvus | i sort of understood the desire for an openstack/opendev cloak as a badge | 14:26 |
corvus | so if we're going to hand out badges, we should have a badge policy :) | 14:26 |
clarkb | got it | 14:26 |
*** avass has quit IRC | 14:27 | |
*** avass has joined #opendev | 14:28 | |
corvus | (i'm now visualising a cloak with a badge sewn on to it; this is getting weird) | 14:28 |
corvus | (it really clashes with fungi's shirt) | 14:29 |
fungi | i'll stitch my merit badges onto my backpack instead | 14:31 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (12) https://review.opendev.org/c/openstack/project-config/+/788572 | 14:34 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (13) https://review.opendev.org/c/openstack/project-config/+/788573 | 14:34 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (14) https://review.opendev.org/c/openstack/project-config/+/788574 | 14:34 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (15) https://review.opendev.org/c/openstack/project-config/+/788577 | 14:35 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (17) https://review.opendev.org/c/openstack/project-config/+/788580 | 14:46 |
openstackgerrit | Elod Illes proposed openstack/project-config master: Move projects under meta-config acl (16) https://review.opendev.org/c/openstack/project-config/+/788578 | 14:50 |
fungi | elod_afk: thanks, once we get those last few in i'll start on a cleanup change for acls which are in the wrong namespaces relative to their corresponding repositories | 15:08 |
clarkb | fungi: were there more than just the starlingx ones? | 15:08 |
fungi | roughly a dozen, yes | 15:08 |
clarkb | interesting | 15:09 |
fungi | i cheated and used some python to find them | 15:09 |
fungi | a bit one is the xstatic repos, half of which are in x/ but share an acl in openstack/ with others | 15:09 |
fungi | s/bit/big/ | 15:09 |
clarkb | ah | 15:09 |
fungi | cases like that, i'll just copy the acl | 15:09 |
fungi | rather than moving | 15:10 |
clarkb | ++ | 15:10 |
fungi | also tempted to move the openstack/project-config acl into opendev/ or something for clarity | 15:10 |
clarkb | or even a these-projects-are-weird-dir/project-config | 15:15 |
openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder master: Update the ironic jobs https://review.opendev.org/c/openstack/diskimage-builder/+/788972 | 15:15 |
mordred | frickler: my cloak came from Rick Clark (so it's very very very old) and pre-dates corvus having the org contact role | 15:22 |
mordred | s/Rick Clark/dendrobates/ for anyone who happened to only know him in IRC | 15:23 |
mordred | I agree with corvus though - it seems like a thing where we should have some policy/structure around handing out/managing | 15:23 |
mordred | an opendev cloak would also be neat :) | 15:24 |
*** hashar has quit IRC | 15:36 | |
*** mlavalle has quit IRC | 15:37 | |
elod_afk | fungi: thanks, too! :) | 15:40 |
yoctozepto | morning folks | 15:45 |
yoctozepto | is there a way to make opendev apt mirrors *signed*? | 15:46 |
yoctozepto | I mean, we have this image production-gradiness discussion on kolla and we now disable key verifications | 15:47 |
fungi | yoctozepto: probably. how they're signed would differ by distro though | 15:47 |
fungi | yoctozepto: but keep in mind that the mirrors we run aren't official, if we signed them we'd sign them with an unofficial key which wouldn't be valid for official distro mirrors | 15:47 |
yoctozepto | yeah, but can't we have the exact repository info with signature from the original source? | 15:48 |
yoctozepto | assume I am apt packaging noob | 15:48 |
clarkb | for ubuntu and debian you can't unless you accept broken mirrors as the compromise | 15:49 |
clarkb | the reason our mirrors are not signed is that we generate our own indexes to fix that brokeness and don't bother to resign with our own keys (we could do that, but that wouldn't be the same key as the upstream) | 15:49 |
fungi | we could in theory work around that by testing the mirrors before we vos release them, and delay updating until they're coherent, but someone would need to write that automation | 15:50 |
yoctozepto | all right, thank you for the answers! now I understand | 15:50 |
clarkb | thats a good point. You run the risk of not updating as frequently, but could release an always functional state with the appropriate checks beforehand | 15:50 |
yoctozepto | it also explains how we end up in those weird situations in CI | 15:51 |
yoctozepto | that we have working repo, yet it lacks required packages | 15:51 |
yoctozepto | (e.g. app updated but its libs not) | 15:51 |
yoctozepto | and repo is happy | 15:51 |
yoctozepto | well, we don't have the manpower to offer for that particular task | 15:52 |
clarkb | fungi: though I don't think you can address the stale update on client side that way? | 15:52 |
fungi | yoctozepto: also we've treated the lack of signatures on our mirrors as a feature, to dissuade people from configuring their production servers to use our ci system mirrors | 15:52 |
clarkb | fungi: one of the things we do is keep old package around for a cycle or too so that you don't break if you lose the race between apt-get update, repo update, and package pull/install | 15:53 |
fungi | clarkb: stale update on the client side would be addressed the same way we do now: delay deletions | 15:53 |
clarkb | fungi: ya so its multiple pieces we'd need though right? one to validate index against all packages and another to track old packages and delete them after some time delta? | 15:53 |
fungi | we'd mirror with something like rsync, and only delete on the next pass after a successful vos release, or something along those lines | 15:54 |
fungi | and yeah, coax rsync into dumping a list of proposed deletions to a file or something like that | 15:54 |
fungi | but have it not actually delete | 15:55 |
clarkb | another option is to convince debuntu to do per package signing >_> | 15:56 |
clarkb | (I don't think that is a serious option though) | 15:57 |
yoctozepto | but the cleanest one nonetheless ;-) | 15:57 |
fungi | technically all uploaded packages are signed, but those signatures aren't used by the client, they're for tracking provenance of the builds | 15:57 |
*** mlavalle has joined #opendev | 16:03 | |
mordred | yeah - I think if we could figure out the delay-deletion-rsync thing - we could apply that in a few different places | 16:10 |
mordred | but definitely non-trivial - we'd wind up needing to keep a local db of file states probably, so that we could attach a timestamp to the first time a file was flagged as needing deletion | 16:11 |
mordred | and then for deletion passes we could get the list of files whose deletion flag date was > $period | 16:12 |
mordred | oh! you know what would be a clever way to do that that wouldn't require an extra db? make a hardlink tree for each sync operation just containing the files that rsync wants - then periodically delete old hardlink trees - so you'd have the hard link reference count actually take care of removing files no longer referenced | 16:14 |
openstackgerrit | Merged openstack/project-config master: Add glance-tempest-plugin to publish-to-pypi job https://review.opendev.org/c/openstack/project-config/+/788250 | 16:14 |
clarkb | does afs do hardlinks? | 16:14 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (18) https://review.opendev.org/c/openstack/project-config/+/788581 | 16:14 |
mordred | oh - I think it doesn't | 16:16 |
mordred | nevermind :) | 16:16 |
fungi | probably the way to do it would be to run rsync once without --delete, then with --delete and --dry-run together and output the results to a timestamped file, then look for any timestamped lists at least x seconds older the most recent vos release and explicitly rm the files they say would have been deleted at those times | 16:17 |
fungi | mordred: also, technically a filesystem *is* a database, so... ;) | 16:17 |
mordred | fungi: :) | 16:17 |
mordred | fungi: have I told you about the time I accidentally facilitated a debate between krow and linus about filesystems vs databases? | 16:18 |
fungi | hah, that sounds frightening | 16:18 |
clarkb | fungi: but worth watching | 16:18 |
mordred | krow was pro-filesystem and linux was pro-db :) | 16:18 |
mordred | linus | 16:18 |
JayF | Did the venue supply the plate armor or did you have to bring your own :D | 16:21 |
mordred | :D | 16:21 |
mordred | JayF: it turns out, this is the consequence of writing a talk with an inflammatory title and then not having enough content to actually fill the talk slot. so the plate armor was actually withheld purposefully | 16:22 |
JayF | oh god that's like, conference fear #1 | 16:22 |
fungi | mine is that they supply the audience with crates of old produce | 16:23 |
fungi | and a working trebuchet, because those are always fun | 16:24 |
*** dtantsur is now known as dtantsur|afk | 16:27 | |
yoctozepto | lol | 16:32 |
*** jpena is now known as jpena|off | 16:38 | |
mordred | fungi: now I really want that to be a conference feature | 16:47 |
fungi | just warn me not to submit to that cfp | 16:51 |
*** hamalq has joined #opendev | 17:19 | |
*** hamalq has quit IRC | 17:20 | |
*** hamalq has joined #opendev | 17:21 | |
*** stephenfin is now known as stephenfin|PTOin | 17:27 | |
*** stephenfin|PTOin is now known as stephenfin|PTO | 17:28 | |
clarkb | fungi: we also want to clean up the all projects acl once we're happy with the state of openstack repos ya? | 17:44 |
clarkb | fungi: does the new meta project acl already cover everything in all-projects or does it just add deletion? | 17:45 |
fungi | all | 17:47 |
fungi | so once it's in place we should be able to clean that up and make sure everything's still working for the openstack release team | 17:47 |
clarkb | cool so really the only thing we need tis to be satisfied all the necessary project acls are up to date to point at the meta project | 17:48 |
clarkb | ++ | 17:48 |
*** andrewbonney has quit IRC | 17:48 | |
fungi | there are 2 still not merged: 788578 788584 | 17:48 |
clarkb | looking | 17:48 |
fungi | worth noting, the openstack/meta-config acl actually grants create and delete on refs/* instead of refs/heads/* because that's what the all-projects config had, though we can probably tighten it up later | 17:49 |
fungi | i did at least switch to pushsignedtag instead of pushtag though | 17:49 |
clarkb | I've approved 788578 and +2'd 788584. I didn't approve 788584 as that is the largest update and want that to go in separately just to minimize chances it has a sad | 17:51 |
clarkb | (I believe we'll always use master project-config on those updates so landing them together would add to the list of updates) | 17:51 |
fungi | yep, i agree | 17:58 |
openstackgerrit | Merged openstack/project-config master: Move projects under meta-config acl (16) https://review.opendev.org/c/openstack/project-config/+/788578 | 18:01 |
*** owalsh has quit IRC | 18:04 | |
*** eolivare has quit IRC | 18:04 | |
*** jaicaa has quit IRC | 18:04 | |
*** lbragstad_ has quit IRC | 18:04 | |
*** fdegir has quit IRC | 18:04 | |
*** priteau has quit IRC | 18:04 | |
*** stand has quit IRC | 18:04 | |
*** calcmandan has quit IRC | 18:04 | |
*** elod_afk has quit IRC | 18:04 | |
*** akahat|ruck has quit IRC | 18:04 | |
*** icey has quit IRC | 18:04 | |
*** owalsh has joined #opendev | 18:07 | |
*** eolivare has joined #opendev | 18:07 | |
*** jaicaa has joined #opendev | 18:07 | |
*** lbragstad_ has joined #opendev | 18:07 | |
*** fdegir has joined #opendev | 18:07 | |
*** priteau has joined #opendev | 18:07 | |
*** stand has joined #opendev | 18:07 | |
*** calcmandan has joined #opendev | 18:07 | |
*** elod_afk has joined #opendev | 18:07 | |
*** akahat|ruck has joined #opendev | 18:07 | |
*** icey has joined #opendev | 18:07 | |
*** gouthamr has quit IRC | 18:08 | |
*** gouthamr has joined #opendev | 18:09 | |
fungi | with all but 788584 now merged, i can safely base my cleanup change on top of that one | 18:25 |
fungi | any idea if vexxhost/base-jobs using gerrit/acls/opendev/project-config.config was intentional? | 18:29 |
*** sboyron has quit IRC | 18:39 | |
*** eolivare has quit IRC | 18:40 | |
clarkb | fungi: I think it started in opendev/ at first maybe? | 18:45 |
clarkb | and then we split out a tenant for them | 18:45 |
openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Move some Gerrit ACL files into better namespaces https://review.opendev.org/c/openstack/project-config/+/789016 | 18:47 |
fungi | well, those ^ are the easy half | 18:47 |
fungi | the other half i turned up are all in sticky splits of repos between namespaces | 18:47 |
fungi | mostly cases where one team maintains some repos which are part of openstack and some others which aren't | 18:48 |
fungi | aside from the vexxhost/base-jobs mentioned above, these seem to be team split situations: charms, monasca, openstack-ansible, puppet-modules and xstatic repos | 18:51 |
fungi | so would all need a second copy of the acl to associate stuff from the other repo with | 18:51 |
fungi | er, from the other namespace with | 18:51 |
clarkb | and trim out the openstack meta project parentage | 18:57 |
fungi | yep | 18:57 |
fungi | do we want the opendev sysadmins retaining control over vexxhost/base-jobs? | 18:57 |
clarkb | I want to say there may have been reasons for that at the time | 18:59 |
clarkb | but I'm not remembering them | 18:59 |
*** slaweq has quit IRC | 19:18 | |
corvus | i'd like to restart zuul, how's that sound? | 19:37 |
corvus | great, i'll get started then :) | 19:39 |
corvus | (release queue looks empty; load is around 50%) | 19:40 |
corvus | #status log restarted zuul on commit b9a6190a452a428da43dc4ff3e6e388d4df41e8b | 19:41 |
openstackstatus | corvus: finished logging | 19:41 |
clarkb | corvus: no objections (seems I'm late anyway) I'm popping out for some exercise though | 19:42 |
corvus | restart may be a smidge slower as the fleet is starting with cold git repo caches | 19:47 |
fungi | yeah, a belated thumbs up and thanks for the restart! | 19:51 |
fungi | sorry, dinner prep has distracted me | 19:51 |
corvus | grafana says we're about 1/2 through the merge jobs | 19:54 |
corvus | and we will take a brief pause as several mergers all clone nova around the same time :) | 19:55 |
corvus | re-enqueing | 20:04 |
corvus | infra-root: this moves another chunk of functionality into zk, so worth being aware of that in case of performance impacts (build results are returned via zk instead of gearman) | 20:05 |
fungi | thanks for the heads up! | 20:15 |
*** ralonsoh has quit IRC | 20:37 | |
*** auristor has quit IRC | 20:38 | |
*** auristor has joined #opendev | 20:39 | |
clarkb | corvus: a "brief" 7 minute pause | 21:36 |
corvus | it actually didn't slow down that much | 21:37 |
*** erbarr has quit IRC | 21:37 | |
corvus | if you look at the graph, it's mostly a constant slope down | 21:37 |
corvus | i guess the number of mergers is sufficiently greater than the number of nova branches that the nova cat jobs weren't a big speed bump | 21:38 |
*** erbarr has joined #opendev | 21:40 | |
clarkb | nice | 21:49 |
*** DSpider has quit IRC | 22:56 | |
*** tosky has quit IRC | 23:31 | |
TheJulia | Question, has anyone tried turning off the #openstack-unregistered stuff? | 23:31 |
clarkb | I don't think I'm in the unregistered channel anymore due to a recent upgrade of my bouncer host but prior to that it would still get ~weekly spams assulats | 23:32 |
* clarkb looks at eavesdrop | 23:32 | |
TheJulia | :( | 23:35 |
clarkb | http://eavesdrop.openstack.org/irclogs/%23openstack-unregistered/%23openstack-unregistered.2021-04-20.log.html#t2021-04-20T14:44:04 and http://eavesdrop.openstack.org/irclogs/%23openstack-unregistered/%23openstack-unregistered.2021-04-09.log.html#t2021-04-09T09:47:23 (though that second one may be legit, I'm not going to go trying it) | 23:36 |
clarkb | thats not too bad considering what it was like. Also looks like we lost logging this week | 23:36 |
clarkb | the openstack bot is in that channel so why are there no logs | 23:37 |
clarkb | fungi: ^ other channels (like this one) seem to continue to log just fine. Any ideas? | 23:38 |
clarkb | TheJulia: given there appears to have only been ~1.5 spam incidents in the last several weeks I'd be inclined to try undoing the registration requirement. Will let others chime in though | 23:39 |
TheJulia | I'm just thinking it could be a barrier for easiest entry since few grow up with IRC these days :) | 23:41 |
clarkb | TheJulia: maybe add it to the infra meeting agenda and we can bring it up there? or send a note to service-discuss@lists.opendev.org | 23:42 |
clarkb | I'll try to remember to bring it up at the meeting either way but I've got 5 year olds that want outdoor time now | 23:43 |
TheJulia | enjoy :) | 23:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!