Friday, 2020-04-03

*** sgw has joined #opendev00:05
*** ysandeep|away is now known as ysandeep|rover01:44
*** ysandeep|rover is now known as ysandeep|rover|b02:36
*** ykarel|away is now known as ykarel03:54
openstackgerritIan Wienand proposed opendev/system-config master: [dnm] test with plain nodes  https://review.opendev.org/71281904:02
*** ysandeep|rover|b is now known as ysandeep|rover04:19
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds  https://review.opendev.org/71717705:52
*** sgw has quit IRC06:01
*** dpawlik has joined #opendev06:06
*** DSpider has joined #opendev06:35
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds  https://review.opendev.org/71717706:56
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Adds roles to install and run hashicorp packer  https://review.opendev.org/70929206:57
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds  https://review.opendev.org/71717707:01
*** mrunge_ is now known as mrunge07:05
*** ykarel is now known as ykarel|afk07:21
yoctozeptomorning folks07:34
yoctozeptoI have a weird issue with logs in swift: https://07bda34aab9549eeca55-002c6183c0f1ab9234471cb74705bd70.ssl.cf5.rackcdn.com/717073/2/check/kolla-ansible-centos8-source-upgrade/7b9a5ae/primary/logs/kolla/libvirt/07:35
yoctozeptothe file has non-0 size in listing and really should have some content07:35
yoctozeptoyet downloads as a 0-length file ;/07:35
yoctozeptoother files seem fine07:36
*** ykarel|afk is now known as ykarel07:37
*** tosky has joined #opendev07:44
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: [DNM] - testing updated software-factory images  https://review.opendev.org/71721007:48
*** ykarel is now known as ykarel|afk07:55
*** rpittau|afk is now known as rpittau07:56
*** ralonsoh has joined #opendev07:57
frickleryoctozepto: iirc we had some reports of incomplete multipart-uploads, maybe the same is happening for single-part uploads too, sometimes07:59
*** ysandeep|rover is now known as ysandeep|rover|l08:03
yoctozeptofrickler: ack, sad to lose logs08:04
*** ykarel|afk is now known as ykarel|lunch08:09
*** dpawlik has quit IRC08:11
*** dpawlik has joined #opendev08:11
*** ysandeep|rover|l is now known as ysandeep|lunch08:32
openstackgerritMerged openstack/project-config master: Add hourly periodic pipeline  https://review.opendev.org/71706308:44
openstackgerritMerged openstack/project-config master: Be clear that zone repos are owned by infra-core  https://review.opendev.org/71712108:44
*** rmart04 has joined #opendev08:48
*** hashar has joined #opendev09:03
*** ysandeep|lunch is now known as ysandeep|rover09:09
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman  https://review.opendev.org/71657809:14
*** ykarel|lunch is now known as ykarel09:26
openstackgerritThierry Carrez proposed openstack/project-config master: release-approval pipeline: fix zuul-excluding regexp  https://review.opendev.org/71727710:06
*** rpittau is now known as rpittau|bbl10:29
*** ysandeep|rover is now known as ysandeep|break11:05
*** ysandeep|break is now known as ysandeep|rover11:48
*** rpittau|bbl is now known as rpitau12:08
*** rpitau is now known as rpittau12:09
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: Add phoronix-test-suite job  https://review.opendev.org/67908213:03
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005713:11
fungiyoctozepto: does that file usually get served okay from other build results?13:20
openstackgerritMerged openstack/project-config master: release-approval pipeline: fix zuul-excluding regexp  https://review.opendev.org/71727713:20
fungiinteresting, certcheck says the https cert for mirror02.mtl01.inap.opendev.org isn't getting renewed13:24
fungiahh, it's the stale apache worker problem again13:26
fungiopenssl says "Not After : Jun 22 03:04:36 2020 GMT" there for me13:28
fungii've restart apache on it for good measure13:29
fungier, restarted13:29
fungi#status log restarted apache on mirror02.mtl01.inap since certcheck spotted a stale apache worker13:30
openstackstatusfungi: finished logging13:30
yoctozeptofungi: yeah, it does13:37
fungiokay, so it was just in that one build where it's corrupted, not a file which is consistently corrupted across builds. in that case frickler's theory is a good onw13:40
fungigood one13:40
openstackgerritMerged opendev/system-config master: Run service-bridge in zuul and semaphore everything  https://review.opendev.org/71674513:42
openstackgerritMerged opendev/system-config master: Migrate gitea-lb to zuul  https://review.opendev.org/71674613:44
openstackgerritMerged opendev/system-config master: Run letsencrypt in zuul  https://review.opendev.org/71674713:47
openstackgerritMerged opendev/system-config master: Run nodepool in zuul  https://review.opendev.org/71677013:47
*** ysandeep|rover is now known as ysandeep|away13:59
openstackgerritMonty Taylor proposed openstack/project-config master: zuul-worker: remove python-apt & libselinux deps  https://review.opendev.org/71678514:20
openstackgerritMonty Taylor proposed openstack/project-config master: zuul-worker: remove additional install of apt-transport-https  https://review.opendev.org/71678914:20
openstackgerritMerged openstack/project-config master: Trigger infra-prod-service-nodepool on nodepool changes  https://review.opendev.org/71713514:24
mordredwoot. we are now triggering nodepool bulder ansible when we land nodepool things in project-config14:29
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: Add support for RedHat platforms on install-podman  https://review.opendev.org/71657814:35
clarkbmordred: part of me wants to merge silly changes and just watch it work. Disable gitea04, reenable gitea04 etc14:48
*** sgw has joined #opendev14:49
*** ykarel is now known as ykarel|away14:52
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: tox: allow tox to be upgraded  https://review.opendev.org/69005715:01
*** yoctozepto has quit IRC15:17
*** yoctozepto8 has joined #opendev15:18
mordredclarkb: yeah. that's actually why I pushed up updates to ianw's project-config patches above15:24
mordredclarkb: https://review.opendev.org/#/c/716785/ should be safe to land and we can watch the ansible run15:24
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds  https://review.opendev.org/71717715:36
mordredclarkb, fungi : oh - we need to do this: https://review.opendev.org/#/c/717112/15:41
mordred(this is gerrit followup, not infra-prod-zuul stuff)15:42
clarkbmordred: I don't see where ${git_cmd} is defined in those crons?15:43
*** rpittau is now known as rpittau|afk15:43
mordredclarkb: oh! good point15:45
openstackgerritMonty Taylor proposed opendev/system-config master: Add cron jobs that were managed by puppet  https://review.opendev.org/71711215:46
mordredclarkb: also - I don't thnk we need to escape those " do we?15:47
openstackgerritMonty Taylor proposed opendev/system-config master: Add cron jobs that were managed by puppet  https://review.opendev.org/71711215:48
*** ysandeep|away is now known as ysandeep15:49
clarkbmordred: nope, thats from puppet string interpolation15:49
clarkbmordred: because we have to use " in puppet to substitute git_cmd15:49
mordredyah15:50
corvusmordred: i want to jump in here; where do i start?15:50
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: use stage3 instead of stage4 for gentoo builds  https://review.opendev.org/71717715:50
mordredcorvus: which thing do you want to jump in with?15:50
corvusmordred: the fun, easy thing?  or maybe the zuul cd stuff15:50
mordredcorvus: the zuul cd is all fun and easy!15:51
corvusmordred: should i just start reviewing 716771?15:51
mordredif you review and land https://review.opendev.org/#/c/716785/15:51
clarkbmordred: one minor string thing related to the "s15:51
clarkb(inlined on the change)15:51
mordredcorvus: you can watch zuul run nodepool playbooks when it's done15:51
corvusmordred: ah cool didn't see that15:51
mordredcorvus: because we landed that!15:51
mordredclarkb: we're sure we don't need to double \ in ansible?15:52
clarkbmordred: you don't have it on line 32615:52
mordredgood point15:53
clarkband I think the yaml ' quoting will preserve a \15:53
clarkbbut now I'm double checking15:53
mordredalthough I should put a ' on 32615:53
openstackgerritMonty Taylor proposed opendev/system-config master: Add cron jobs that were managed by puppet  https://review.opendev.org/71711215:53
clarkbyup yaml single quotes preserves the \15:53
mordredcorvus: https://review.opendev.org/#/q/status:open+topic:infra-prod-zuul is the remaining zuul-cd stack15:54
mordredcorvus: paused at landing it on the nodepool one so that we could watch that work before doing more of them15:54
*** JayF is now known as JasonF15:55
corvusmordred: something is still rubbing me a little wrong about 716771 -- why do we have so many jobs instead of one playbook that does all those things?15:55
*** JasonF is now known as JayF15:55
corvusmordred: like the system-config, install-ansible, base, letsencrypt....15:55
corvusshouldn't there be a 'meetpad' playbook that does all of those if necessary?15:56
corvusmordred: oh, most of those are soft depends...15:56
*** ysandeep is now known as ysandeep|away15:56
mordredcorvus: oh - well, I mean - that would be a rearchitecture of our existing playbooks which I wasn't really doing ... yeah15:56
corvusso they won't normally run?15:56
mordredyeah15:56
mordredthey've all got file matchers on them15:56
mordredso we should be running what we need to run when we need to run it and not running if we don't15:57
mordred(except for in periodic, where we'll run everything)15:57
corvuswell, we started with 0 playbooks, and i thought we wanted a service-based architecture, so i wasn't thinking there should be a rearchitecture.  :)15:57
corvus(like, i'm thinking all the way back to when we started planning this :)15:57
corvusi always thought it was supposed to be "run the etherpad playbook"15:57
mordredyeah. I think there's also a reason for some of the base separation (except for base itself)15:58
corvusanyway, i guess the soft deps make it moot for now15:58
mordredwhich is that we need to run install-ansible separately so that it gets picked up in the next ansible runs15:58
mordredand we need to update system-config separately15:58
clarkbI think what we are doing is mapping the current shell script of run this playbook run that playbook, onto zuul15:58
mordredbase I *totally* agree with15:58
clarkbfrom there we can continue to split things15:58
mordredbase should probably go away and be things we run in each service playbook15:58
corvusmordred: why does meetpad need system-config updated?15:59
clarkbthe only realy difference so far as been knowing when we need to run playbooks and not running them if we don't need to (but the old shell script ran them all regardless)15:59
mordredbecause system-config is where the meetpad playbook is15:59
corvusmordred: and we're not syncing repos from zuul onto bridge15:59
mordredso if we update the meetpad playbook we need to pull the latest system-config to get the updated playbook15:59
mordredright15:59
mordredwe can probably move to that15:59
mordredand remove update-system-config15:59
mordredbut it'll be easier to do that once _everything_ is a zuul job and run_all is gone15:59
corvusmordred: ack; i don't have an opinion on which is better yet :)16:00
mordredme either :)16:00
mordredcorvus: one downside to including base in service playbooks is that it's a LOT of tasks that are mostly no ops most of the time but still take a decent amount of time to run16:00
mordredby keeping it separate and triggering with files matchers we can avoid nooping the install of all the infra sysadmins each time and only run the thing that updated16:01
mordredwe'd also wind up with a larger surface area of triggering if we triggered on all of the base files for every service16:01
mordredbut ... I think we could also go to single-playbook and get rid of base and that would be valid too16:02
mordred(I think we're still learning here)16:02
clarkband for right now mapping the shell script onto zuul makes the reviews nice and symmetric16:02
clarkbits clearer when things are correct16:02
corvusmordred: yeah.  letsencrypt might be the more interesting thing to fold in first.16:02
mordredcorvus: yeah16:02
mordredcorvus: and would be the one that would make more important logical service sense16:03
corvusmordred: +2 with comment on 71677116:03
mordredcorvus: I like that comment16:03
openstackgerritMerged openstack/project-config master: zuul-worker: remove python-apt & libselinux deps  https://review.opendev.org/71678516:03
mordredcorvus: I think I might do a followup that goes in and follows that pattern across all the jobs16:03
mordred(where appropriate)16:03
corvus(i'm confident the first half is a good idea; the second one may have unintended consequences)16:04
clarkbdon't forget to do similar on the group_vars path too :)16:05
clarkbthough amybe thats what the second suggestion gets us globally16:05
corvusmordred: why doesn't 716772 have a dep on base, etc?16:05
mordredcorvus: yay! the nodepool project-config change failed in promote -16:05
mordredfix coming16:05
mordredcorvus: because they're dep'd in infra-prod-service-base16:06
corvusmordred: then shouldn't that be true for meetpad?16:06
mordredcorvus: meetpad adds a dep16:06
clarkbya I think we decided that you can't merge deps, that changing deps requires them to be relisted16:07
mordredand I thought deps were overrides rather than adds - so I've been copying the full dep list when we need to add one16:07
corvusi believe that's correct16:07
openstackgerritMonty Taylor proposed openstack/project-config master: Add update-system-config to promote list for project-config  https://review.opendev.org/71732816:10
mordredcorvus, clarkb : that should fix the error from https://review.opendev.org/#/c/716785/16:10
mordredcorvus: this is fun isn't it?16:11
corvusmordred: :)16:12
mordred(we could alternately give it a dependencies: [] - since it's a project-config change it doesn't ACTUALLY need to update system-config)16:12
mordredcorvus, clarkb : once 717328 lands, we can land https://review.opendev.org/#/c/716789/ as a second test16:15
clarkbk16:16
clarkbI'm about to enter tech support for remote learning/school stuff. Its fun because we get to use zoom!16:16
fungiyou're just secretly planning to zoom-bomb your preschoolers' virtual classrooms, aren't you?16:17
clarkbfungi: well not sure if you know about the behavior of 4 year old but they basically zoom bomb each other already16:17
clarkbthe teachers have discovered they can manage mute toggle of participants though so heopfully things will get better16:18
fungichances are the students showed them where16:20
corvusclarkb: https://www.nytimes.com/2020/03/30/technology/new-york-attorney-general-zoom-privacy.html16:20
*** mlavalle has quit IRC16:20
clarkbcorvus: yes I'm well aware16:20
clarkbironically if I want to read the nytimes I have to use the tabs with google account enabled16:21
corvusclarkb: k, wasn't sure if you saw that article yet -- it's in a pretty accessible form that non-tech-experts might be comfortable digesting16:21
corvusmordred: we can't land 789, it has a depends-on, and i think we'd need a new nodepool builder image16:23
*** mlavalle has joined #opendev16:23
openstackgerritDirk Mueller proposed openstack/diskimage-builder master: opensuse: fix python 2.x install  https://review.opendev.org/71643716:23
mordredcorvus: ah - oh, good point16:23
clarkb(by the way firefox tab containers are great! you discover fun things like that when your default has nothign signed in, website doesn't work, switch to container group with google anv oila!)16:23
corvusmordred:  want to make a no-op change to test, or just leave it for the next time an element changes?16:24
fungiclarkb: i use ff tab containers extensively on a few of my machines, but have slowly warmed to the idea of just setting my browser permanently incognito16:25
openstackgerritMonty Taylor proposed openstack/project-config master: Trigger nodepool run  https://review.opendev.org/71733016:26
mordredcorvus: there's a noop16:27
mordredcorvus: IIRC - noop changes match all file matchers so all the jobs should get run16:27
openstackgerritDirk Mueller proposed openstack/diskimage-builder master: Build openSUSE images on opensuse nodepool image  https://review.opendev.org/71733116:28
openstackgerritMerged openstack/project-config master: Add update-system-config to promote list for project-config  https://review.opendev.org/71732816:37
fungiclarkb: just got around to looking at the flock the opensuse-15 builds on nb02 are complaining about, lsof says one of the processes with an open file handle on it is:16:42
funginodepool 24024  0.0  0.0  26960  3236 ?        S    Apr01   0:00 /bin/bash /opt/dib_tmp/dib_build.DFvCP7NK/hooks/extra-data.d/98-source-repositories16:42
fungithat seems like it's been running longer than i would expect16:42
mordredclarkb, fungi, corvus : quick +A on https://review.opendev.org/#/c/717330 ?16:42
fungimordred: looking16:42
mordredfungi: I agree, that seems a little longer than we'd want16:42
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733916:42
*** yoctozepto8 is now known as yoctozepto16:43
clarkbfungi: oh I was looking at nb01 not nb02, but nb01 definitely has stale processes16:43
clarkbps -elf | grep disk-image-create16:43
clarkbmordred: and we expect that to match all file matchers/ I guess we'll find out?16:44
fungiit seems like a harmless experiment at the very least16:46
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Support multiple matchers when parsing tox output  https://review.opendev.org/71626316:46
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output  https://review.opendev.org/71676616:47
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626416:47
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Ignore absolute paths after stripping work dir  https://review.opendev.org/71704216:47
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Add test cases for tox line comment parsing  https://review.opendev.org/71734116:47
clarkbfungi: my hunch is that certain things crash in such a way that the processes don't clean up nicely16:47
fungithough long term i wonder if we really want to be littering our repos with a bunch of empty commits just to manually trigger things16:47
clarkbfungi: but I have no idea what that may be16:47
fungiclarkb: looking at ps tree view suggests it's stalled out installing distro packages16:47
fungiyum install install dnf dnf-plugins-core curl libcurl glibc-minimal-langpack glibc-langpack-en coreutils16:48
fungis/install //16:48
clarkboh hrm thats fedora16:48
clarkbor does centos 8 do dnf now too?16:48
fungioh, yep, i was looking at the wrong tree16:48
clarkb(fedora should be on nb04 only fwiw)16:48
fungiyeah, so 24024 on nb02 has no children16:50
fungistrace says it's stuck at: write(1, "Updating cache of https://opende"..., 16016:50
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733916:50
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733916:52
fungilooks like AJaeger and i are the only subscribers on service-discuss so far16:53
clarkbthanks for the reminder!16:54
fungidon't forget to sign up if you want to use our shiny new ml! http://lists.opendev.org/cgi-bin/mailman/listinfo/service-discuss16:54
fungiclarkb: also i'm happy to co-admin the list if you let me know what you wind up resetting the listadmin pw to16:54
clarkbfungi: k, I'm sorting that out now16:55
fungithere's no hurry16:55
fungiwhile i'm poking around, it looks like we should do a better job of promoting the service-announce ml too... it currently has 5 subscribers (4 of whom are on the osf staff, i don't recognize the 5th one)16:56
fungiand probably want to make sure service-announce is set to moderate all posts with followup to service-discuss16:57
fungii'm happy to help admin that one too if you like16:57
openstackgerritMerged openstack/project-config master: Trigger nodepool run  https://review.opendev.org/71733016:58
corvusfungi, clarkb: i think a "please subscribe to service-discuss" email sent to openstack-infra might be warranted?16:59
fungicorvus: absolutely16:59
fungididn't know if we wanted to wait for next tuesday, but i don't see a compelling reason to hold off17:00
fungion a related note, there are only 5 non-bots in http://eavesdrop.openstack.org/irclogs/%23opendev-meeting/17:00
fungibut it's ready for service too17:00
fungimaybe we should do a quick pass of the two service-.* ml configs and then suggest subscribing to both when we e-mail the old ml?17:01
clarkbfungi: sound sgood17:03
clarkbfungi: what do you mean by "with followup to service-discuss"?17:05
clarkbforward to service-discuss?17:05
fungireply-to17:05
clarkbah17:05
clarkbok I'll give that a go then you can double check what I've done17:06
fungiso that if someone replies to an announcement from the service-announce ml their reply goes to service-discuss17:06
mordredcorvus: https://review.opendev.org/#/c/717330/ ... we're getting a graph issue because we didn't define infra-prod-letsencrypt, which is a soft-dep from nodepoo17:08
clarkbfungi: I've got the reply to set. Digging around for forcing moderation on all emails17:09
corvusmordred: that suggests you'll need the image promote job too, yeah?17:10
clarkbfungi: is hat "set everyone's moderation bit, including those not visiable" option that I want?17:11
mordredcorvus: should we maybe instead re-define the deps list there in project-config17:11
corvusmordred: so maybe the dependencies:[] is the better approach?17:11
corvusmordred: yeah that17:11
clarkbI'm taking the secrets file lock in order to add these passwds there as others may need to interact with the lists from time to time17:11
*** mlavalle has quit IRC17:11
fungiclarkb: i think that's a one-time thing to set the moderation bit on all current subscribers, but i'll look in just a sec17:12
mordredcorvus: should we keep update-system-config?17:13
mordredcorvus: my only concern would be de-duplication of the nodepool job - but project is part of the uniqifier right?17:13
clarkbdone with the lock17:14
corvusmordred: deduplication?  like in the supercedent pipeline?17:14
mordredyeah17:14
fungiclarkb: almost on the reply-to... see the option immediately above the field you filled in17:14
corvusmordred: pipelines deduplicate items, not jobs.  so a change to project-config followed by a change to system-config will always run the jobs for both.17:15
clarkbfungi: I want to set that to explicit address and remove the one I set?17:15
openstackgerritMonty Taylor proposed openstack/project-config master: Update deps list for nodepool job  https://review.opendev.org/71735117:15
mordredcorvus: cool - then that ^^ should do what we want17:15
fungiclarkb: pretty sure you need both, set reply_goes_to_list to explicit address radio button, then specify the explicit address in the reply_to_address option immediately below it17:16
corvusmordred: (the only thing that would be potentially deduplicated is two changes to the same branch of project-config in a row.  note that means there's actually a pretty high probability that we may not run the jobs we need to.17:16
clarkbfungi: gotcha17:16
clarkbfungi: done, if you want ot check it17:16
mordredcorvus: yeah - I was wondering how file matchers and supercedent pipeline would work out17:16
corvusmordred: i think promote with supercedent probably works best without file matchers17:16
fungiclarkb: afaik reply_to_address is only ever used *if* reply_goes_to_list is set to explicit reply address17:17
corvusmordred: we may want to take AJaeger's advice and make a new pipeline17:17
mordredyeah. so ... that's unfortunate :(17:17
mordredyeah17:17
corvusmordred: make an independent change-merged pipeline, so there's no deduplication17:17
fungiclarkb: yep, that looks right17:17
mordredcorvus: ++17:17
corvusmordred: call it 'deploy'?17:17
mordredcorvus: I'm starting to be sad that we're not in the opendev tenant already17:17
clarkbfungi: I'm not seeing anything other forced moderation options unless we se tallowed message size to 017:18
fungiclarkb: default_member_moderation under Privacy options...Sender filetrs17:20
fungifilters17:20
fungithe way to do it is set that to yes and then also manually (individually or mass) set the moderation bit on the handful of existing subscribers17:20
openstackgerritMonty Taylor proposed openstack/project-config master: Add a deploy pipeline  https://review.opendev.org/71735317:21
openstackgerritMonty Taylor proposed openstack/project-config master: Use deploy pipeline for project-config deployments  https://review.opendev.org/71735417:21
mordredcorvus: ^^17:21
fungiclarkb: and unset the moderation bit for any subscribers you want to be able to post announcements directly without being reviewed17:21
fungiso what will happen then is any new subscribers will automatically be set to moderated17:22
mordredcorvus: here's another fun edge case ...17:22
clarkbfungi: ok I've moderated everyone but you and I and changed that default setting17:22
corvusmordred: why not squash those 2?17:22
clarkbfungi: thanks for the elp17:22
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Add test cases for tox line comment parsing  https://review.opendev.org/71734117:22
mordredcorvus: that'll work? I was thinking the deploy would be a config error until we landed it - but now that I say that, of course it won't be :)17:23
clarkbfungi: though the set everyones moderation bit is flipped. I don't think I did that, did you?17:23
mordredcorvus: while we're talking about this ...17:23
fungiclarkb: my pleasure, we can also untick moderation on our other sysadmins once they subscribe, if there's a consensus that's how we want to operate the announcements list17:23
fungiclarkb: i didn't, as far as i know17:23
clarkbfungi: perhaps that was toggled by the default changing on the privacy page17:23
clarkbfungi: I think that must've been it17:23
mordredcorvus: some of the deploy jobs depend on an image being published - but that's a promote pipeline thing. should we move those image publications to deploy too, even if it means no de-dupe for them?17:23
fungiclarkb: oh, i see what you're talking about, at the bottom of the member list view. that's an action masquerading as a config option ;)17:24
clarkbfungi: I thought it was set to no before17:24
corvusmordred: yeah, they'll all need to be in the same pipeline17:25
clarkbbut no wI see the "set" button is seprate fro mthe list update17:25
clarkbso ya all good17:25
openstackgerritMonty Taylor proposed openstack/project-config master: Add and use a deploy pipeline  https://review.opendev.org/71735317:25
fungiif you hit the set button while the radio toggle is at "on" it will set all current subscribers to moderated. if you hit the set button while the radio toggle is at "off" it'll unset everyone's moderation flag. it's not really a config option on its own17:25
mordredcorvus: kk.17:25
clarkbfungi: yup17:25
clarkbI missed the "set" button at first17:25
clarkbfungi: on nb01 what do you think of my idea of stopping the builder service there and checking if dib cleans up at that point?17:26
clarkbfungi: then we can start the builder up again17:26
fungiclarkb: one potential topic for next week's meeting could be whether we want to adopt the same dmarc avoidance policies on service-discuss as we've got applied on openstack-discuss and zuul-discuss17:26
clarkbfungi: I want to say we did apply them to -infra for testing?17:27
clarkbso maybe we should for consistency, but can discuss it there17:27
fungiyeah, i mean i'm in favor, but picking a stance on that and stating it clearly while the list is fresh is a good time to do so17:27
openstackgerritMonty Taylor proposed opendev/system-config master: Switch to deploy pipeline for deployments  https://review.opendev.org/71735617:28
mordredcorvus: k. there's the other side of that then17:28
fungiclarkb: also we should set some good description and info text for these17:28
fungii'm happy to take a stab at that17:29
clarkbfungi: go for it17:29
fungiclarkb: yeah, stopping the builder can't hurt and would be good to find out what actually happens17:29
*** mlavalle has joined #opendev17:29
fungithough getting to the bottom of the stuck processes might be good to if we have time to do so17:30
clarkbok I'll give that a go on nb01 now and ee if it cleans up those processes (i expect they will get zombied and reparented to init17:30
mordredclarkb, AJaeger: https://review.opendev.org/#/c/717353 could use eyeballs - and then https://review.opendev.org/71735617:30
mordred(because of the issue identitied with promote + files matchers)17:30
*** dpawlik has quit IRC17:30
clarkbyup all the dib processes went away when nodepool-builder stopped17:30
clarkbmaybe we just need nodepool to waitpid harder?17:31
clarkbI mean if init can clean them up we should be able to?17:31
openstackgerritTobias Henkel proposed zuul/zuul-jobs master: Add test cases for tox line comment parsing  https://review.opendev.org/71734117:34
clarkbI think I see the nodepool bug17:35
*** mlavalle has quit IRC17:37
corvusbiab17:37
*** mlavalle has joined #opendev17:39
clarkbhttps://review.opendev.org/717357 pushed to help with nodepool dib process leaks17:41
*** mlavalle has quit IRC17:42
mordredclarkb: lgtm17:44
fungiclarkb: okay, i've drafted description and info text for both mailing lists, take a look and see if you want to tweak anything17:48
fungiinspiration taken from openstack-announce and openstack-discuss17:48
clarkbfungi: lgtm17:50
fungii guessed at the scope we ultimately want for the announcements list, we can go over that in the meeting too17:52
fungiand we might want to consider forklifting the netiquette wiki article into the infra manual17:53
fungithen link to that in the discuss ml info instead of to wiki.openstack.org17:53
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Fixed permissions issues on SpammerProcess  https://review.opendev.org/71735917:54
*** hashar is now known as hasharDinner17:55
clarkbmordred: actually I think that change isn't necssary. There are two loops happening there and I thought there was just one18:11
clarkbmordred: we fall into the subprocess checking via the outer loop18:12
clarkbso now I think we want ot look for logged timeouts18:12
clarkband maybe we need a p.wait() after the kill instead18:12
*** mlavalle has joined #opendev18:13
clarkbwe never log a build timeout though18:16
clarkbsubprocess does say that process.wait() can deadlock if using PIPE for stdout and stderr. We do use a PIPE I wonder if that is what is happening here18:17
clarkbthat would also explain why we don't hit the timeout case because we'd be stuck on the wait condition18:18
clarkbI *think* what we want to do is get a thread dump and see if we are deadlocked in subprocess wait routines18:18
clarkbfungi: ^18:18
fungithat sounds likely18:19
fungii'm about to disappear on our weekly outing to pick up our grocery order and some takeout pizza though18:19
corvusclarkb, mordred: do we want to: 1) merge the 'deploy' pipeline change and rebase everything in system-config?  2) ignore the deploy pipeline for now and keep merging system-config changes?18:25
clarkbcorvus: mordred: I'm thinking that maybe we should switch to deploy to start as it will avoid any oddness that might demand debugging later18:27
clarkbalso whay does https://review.opendev.org/#/c/717356/1/.zuul.yaml switch the image builds to deploy from promote18:27
corvusclarkb: because of job dependencies18:27
clarkbah right bceause we consume those images in deploy18:27
corvusyep18:27
clarkbI've acced that change and its parent but not approved in case consensus is to switch after landing things18:28
clarkb*acked18:28
corvusclarkb: i think maybe we should go ahead and merge it; i guess we could probably still move system-config to deploy later18:33
corvuser scratch that, nm18:33
corvuslet's merge that and rebase the system-config stack18:33
*** ralonsoh has quit IRC18:38
*** rmart04 has quit IRC18:38
*** hasharDinner has quit IRC18:55
*** hashar has joined #opendev18:57
mordredclarkb, corvus : yes - I agree- let's merge it19:04
mordredalso - I don't think we'll need to rebase the system-config stack - I think the patches in the stack will merge onto it just fine19:05
clarkbmordred: oh because we edit the lines above ya19:05
mordred(I might be wrong - but I think it'll just work)19:05
mordredyeah19:05
clarkbgit may just do the right thing, I agree19:05
mordredthere's enough separation19:05
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733919:12
corvusfingers crossed :)19:17
corvusclarkb, mordred: gah, i did not notice https://review.opendev.org/71735119:17
clarkbwhy do we not need the deps in that case?19:19
clarkbdon't we still want system-config playbooks to be updated and LE to run first etc?19:19
corvusclarkb: not on on project-config where we're only running that to install new nodepool elements19:19
clarkbits also installing the nodepool confg which could depend on other bits?19:20
corvusi dont think it needs to run le19:21
clarkbya I think system-config playbooks are what I'm most concerned about19:22
clarkbspecifically we write out clouds.yaml for nodepool in system-config but consume that in the nodepool configs in project-config19:22
corvusclarkb: but they can't be updated in this change19:22
corvuser19:22
corvusclarkb: but they can't be updated in a change to project-config19:22
clarkbso you might add a new cloud and get that written out without the properly clouds.yaml on disk19:22
clarkbcorvus: its the consumption that is updated in project-config19:22
clarkbhttps://opendev.org/openstack/project-config/src/branch/master/nodepool/nl01.openstack.org.yaml that stuff19:23
corvusclarkb: i don't think we have any testing that the nodepool config file works with the clouds.yaml contents from system-config19:24
corvusclarkb: so even if we did run it, zuul isn't going to stop us from merging a change that adds "cloud-that-does-not-exist" to nodepool.yaml19:24
clarkbcorrect its the side effects on production I'm worried about19:25
clarkbbasically even if we do all the things properly with depends on and merge things in order we could apply them in the wrong order an dmake things unhappy19:25
corvusclarkb: however, if we do the process correctly, then we will add a new cloud to system-config first, when that change merges, the system-config deploy job will put it on disk.  then a project-config change to use it can rely that it will be there.19:25
clarkbgranted it will likely only be unhappy for a short time19:25
corvusclarkb: i don't think we can do it in the wrong order, especially with a depends-on19:25
clarkbcorvus: they can merge concurrently19:25
clarkbor at least one after the other19:25
clarkbcouldn't the project-config deploy job run before the system-config deploy job in that case?19:26
clarkb(bceause zuuls scheduling isn't strictly ordered? though maybe that isn't true for empty nodeset jobs)19:26
corvusclarkb: it generally shouldn't but could occasionally run in the wrong order if they merged close together19:27
corvusclarkb: don't we have a semaphore though?19:27
clarkbcorvus: we do have a semaphore does that enforce order or just locking?19:27
corvusclarkb: just locking, but the only way they could run out of order is if an executor failed or was slow.  so if they share a semaphore, they'll run in order.19:28
clarkbok, I think the risk in this case then is that changes merged together could write invalid configs until the other change's deploy job ran. In most cases that would be a short period of invalid config.19:28
clarkbdo we think it is worthwhile to properly express that dependency?19:29
clarkbMaybe I'm missing the motivation for removing it in the first place19:29
corvusi think if there's a depends-on, then we can't break it19:29
corvusand i think the motivation is so we don't put a whole bunch of the system-config deploy jobs in the project-config pipeline19:30
clarkbthe file filter would've restricted it to when nodepool was updated19:31
corvusclarkb: yes, but we'd have to add the letsencrypt and base and system-config update when it's not actually possible to make a change to project-config which can do anything that would require those jobs to run19:31
clarkbok maybe thats what I'm missing19:31
clarkbwe can't run the system-config update without running base and le?19:32
mordredwe can ... but the deps list for the job currently has them19:32
corvusyeah, any change to things that those playbooks touch are in the system-config project; when changes to system-config to touch those things merge, system-config will run the whole set of jobs19:32
mordredwe could also override the deps list to only have the system-config job19:32
mordredinstead of overiding to []19:32
clarkbmordred: I think thats what I'm trying t oget at :)19:33
corvusbut the update-system-config job is not necessary19:33
mordredbut I agree with corvus19:33
mordredthat the system-config is not necessary19:33
clarkbI'm still not sure I understand that19:33
mordredbecause the system-config change that would go in woudl trigger it19:33
mordredso when system-config changes, it'll run the update-system-config19:33
mordredand it'll update system-config19:33
clarkbright but didn't corvus say they could run out of order in some cases?19:33
corvusno, i revised that to say, with a semaphore, that can't happen19:34
clarkband expressing the dep explicitly would avoid that19:34
mordredyeah19:34
mordredthe semaphore gets us here19:34
clarkbok thats no thow I parsed that. I still parsed that as "in some cases wecan run out of order" and those cases would still be addressed by explicit dep19:35
clarkb*not how19:35
corvuspipeline change queues are ordered; so even though these projects don't share a change queue (because it's an independent pipeline), they're still created in order19:35
clarkbcorvus: but not necessarily executed in order (which was my qusteion about empty node set jobs)19:35
corvusi said that before we started talking about semaphores.  i'm sorry i didn't remember we had semaphores at first, but as soon as i did, i attempted to correct myself.19:36
clarkbok19:36
clarkbI think I'm still of the opinion that being explicit is worthwhile if it makes the system correct and more intuitively understandable. But if the setup as proposed is correct I'll go with it.19:37
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-yarn: add coverage for all platforms  https://review.opendev.org/71737519:38
corvusclarkb: if we're worried about sequencing there, i'd rather do more work to make sure the jobs only run in the right order (ie, if we relax the semaphore restriction, make sure they still run in the right order).  i don't think we should ever need to run the update-system-config on a change to project-config, and i think it's more correct for us not to.19:39
clarkbalso for anyone else following along here I think the lightbulb just went off a bit brighter. deploy peipeline is not supercedent19:40
clarkbthis means we'd be running redundant jobs earlier in the queue19:40
clarkbor later depending on which way you look at it I suppose19:40
clarkbcorvus: I think my problem is I'm still thinking of it from a supercedent perspective19:42
clarkbcorvus: where you want any one run to ensure all of its deps run in that pipeline item19:42
clarkbwhat we've done is split that out into running jobs for every change in the pipeline (in order via semaphore)19:43
clarkband so don't need each thing to always have its deps (they'll have run ahead)19:43
corvusclarkb: yes, we can (due to semaphore) guarantee that19:44
corvusclarkb, mordred: we may want to experiment with having a dependent pipeline with a defined shared queue19:44
AJaegeranybody to review updating openstack-zuul-jobs hacking version, please? https://review.opendev.org/71626119:45
corvusit's unorthodox, but could be interesting19:45
mordredcorvus: hrm. that could be interesting19:45
openstackgerritMerged zuul/zuul-jobs master: Add phoronix-test-suite job  https://review.opendev.org/67908219:46
mordredI"m not 100% sure we need it given the semaphore thing ... but it also might not be crazy19:46
*** dpawlik has joined #opendev19:48
openstackgerritMerged openstack/project-config master: Update deps list for nodepool job  https://review.opendev.org/71735119:51
openstackgerritMerged openstack/project-config master: Add and use a deploy pipeline  https://review.opendev.org/71735319:51
corvusmordred: it'll look really good19:52
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-yarn: always install  https://review.opendev.org/71737519:56
openstackgerritMerged zuul/zuul-jobs master: Support multiple matchers when parsing tox output  https://review.opendev.org/71626320:00
openstackgerritMerged zuul/zuul-jobs master: Don't silently ignore exceptions when parsing tox output  https://review.opendev.org/71676620:00
openstackgerritMerged zuul/zuul-jobs master: Strip source dir from file comments  https://review.opendev.org/71626420:00
openstackgerritMerged zuul/zuul-jobs master: Ignore absolute paths after stripping work dir  https://review.opendev.org/71704220:00
openstackgerritMerged zuul/zuul-jobs master: Add test cases for tox line comment parsing  https://review.opendev.org/71734120:02
mordredcorvus: well that's reason enough20:03
mnaseri dont see more than 1 log for bionic on nb0120:08
fungithis discussion reminds me, i guess it's dangerous to use file filters on jobs in supercedent pipelines, since the deduplicated builds might run different sets of jobs20:08
mnaserbut https://nb01.openstack.org/ubuntu-bionic-plain-0000000031.log from a couple days ago installed gnupg https://nb01.openstack.org/ubuntu-bionic-plain-0000000043.log the most recent one does not show it being installed20:09
corvussee scrollback in #zuul for info about a possible image problem20:09
corvuswhat's ubuntu-bionic vs ubuntu-bionic-plain?20:09
fungi-plain is the version without the pip-and-virtualenv element20:09
clarkbit is what we are using to test that removing pip doesn't completely break the owrld20:09
clarkb(because we'll have zuul jobs cover the delta)20:10
corvusok, so 'ubuntu-bionic' is mostly what we're concerned about, but '-plain' may give us extra data points20:10
mnaserright, but i think what was hinted about the relation to the zuul-worker role change20:10
fungiyeah, eventually that will disappear when we fold that change into the real images20:10
mnaserand nb01 doesnt have logs except for 1 single bionic build20:10
mnaserso this one provided some contextual history20:10
clarkbI know mordred had made changes to things to not install suggests and recommends but I believe that was our docker images only20:10
clarkband gpg-agent is a requires not suggests or recomments of gnupg anyway20:10
corvusoh i thought we applied it elsewhere too?20:11
mnaserbasically 2020-04-02 05:30 had gnupg and 2020-04-03 06:57 did not20:11
corvus(i thought i recalled a sort of "we're doing it here, let's do it in the other place" commit message)20:11
clarkbcorvus: ah that could be20:11
mordredclarkb: yeah - and that was actually beause we already do that in our dib images20:12
corvusmordred: was that a change to dib rather than a change to p-c?20:12
fungilast dib release was monday20:12
mordredit wasn't either - it was a change to python-base docker to match what p-c was already doing in dib images20:12
fungiand we're installing dib from tagged releases20:12
corvusmordred: okay, so nothing about that changed recently in our disk images20:13
mordrednope20:13
fungiso unless something delayed image deployment by a couple days the dib release seems unlikely to have brought this behavior in20:13
corvusfungi: if nb04 is implicated, that's a possibility, but i think only nb01/nb02 are right now20:14
openstackgerritMerged zuul/zuul-jobs master: helm: collect kubernetes logs in post  https://review.opendev.org/71570920:14
corvus(i mean, it's still a possibility on nb01/2 but it's remote)20:14
fungiyeah, nb04 is still only building fedora-31 i think?20:14
clarkbfwiw python-apt does hard dep gnupg20:14
mnaserhmm20:14
clarkbso I bet removing python-apt did cause us to stop installing gnupg ( and gpg-agent)20:14
fungier, 30 and 3120:14
fungihttps://nb04.opendev.org/20:14
mnaserit looks like install-packages element didnt run at all20:15
clarkbwhat I don't understand is how we have a `gpg` utility on the server without gpg-agent20:15
clarkbmnaser: it had to otherwise we wouldn't have an ssh server20:15
fungidib may run apt installing without recommends in some places20:15
mnaserctrl+f on the log which doesnt include gnupg doesnt show `package-installs-v2 --phase install.d` running20:15
clarkbfungi: they aren't recommends they are requires20:15
clarkbfungi: that is why I'm confused gnupg requires gpg-agent20:15
fungidepends, yes20:16
fungii just checked20:16
fungi(not requires, that's a python thing)20:16
mordredmnaser: I was about to say the same thing20:16
corvusyeah, they seem surprisingly different20:16
mnasermaybe comparing the 'plain' isn't teh right idea but yeah20:17
mordredmaybe we were getting a transtive dep on install-packages from pip-and-virtualenv20:17
corvusmnaser: even comparing https://nb02.openstack.org/ubuntu-bionic-0000104262.log and https://nb01.openstack.org/ubuntu-bionic-0000104264.log shows that difference20:17
mordredinfra-package-needs has a dep on package-installs20:18
mordrednot on install-packages20:18
mnaserit looks like ifnra-package-needs is not running20:18
mnaseror at least it's not doing the whole mapping thing20:18
corvusoh 4262 failed20:18
clarkbmnaser: thats what installs sshd though20:18
corvusmaybe it just didn't get far enough20:18
clarkbmnaser: so how did we get an sshd if that didn't run?20:18
mordredoh - nm. the element is package-installs20:19
mnaserthat's a very good question but i'm just trying to put things out20:19
clarkbI think its running20:19
mordredpackage-installs-v2 --phase install.d didn't run in the bad log20:19
mnaserMap install for infra-package-needs: wget, rsync, ntpdate, git, traceroute, lvm2, ntp, tcpdump, iptables, tox, at, redhat-lsb-core, iproute2, dnsutils, build-essential, cron, gentoolkit, strace, curl, util-linux, rsyslog, redhat-rpm-config, haveged, python3-dev, python-dev, parted, uuid-runtime, coreutils, acpid, iputils-ping20:19
mnaseri dont see that in https://nb01.openstack.org/ubuntu-bionic-0000104264.log20:19
clarkbsearch DPKG_MANIFEST_NAME20:20
mnaserclarkb: openssh comes in from another element20:20
mnaserMap install for openssh-server: openssh-server20:20
mnaserso i think the issue is that somehow infra-package-needs list of packages didnt get "appended" inside package-installs20:20
clarkbhrm Ithought that was in infra pakacage needs (I always added it to my own image builds for that reason) but maybe we can get it from multiple places now20:20
fungiokay, so here's how we can have gpg and no gpg-agent. the gpg and gpg-agent packages are a dependens entries for the gnupg metapackage, but you an install the gpg package which merely recommends gnupg20:21
mnaserwait, im blind, sorry, infra-package-needs is there20:21
clarkb2020-04-03 17:51:51.325 | Map install for infra-package-needs: rsync, tcpdump, cron, tox, git, haveged, ntpdate, traceroute, util-linux, dnsutils, redhat-lsb-core, build-essential, ntp, python-dev, gentoolkit, acpid, redhat-rpm-config, python3-dev, parted, lvm2, coreutils, rsyslog, iptables, curl, strace, iputils-ping, wget, at, iproute2, uuid-runtime20:22
* fungi will retype that if people can't read his pizza-writing20:22
clarkbfungi: aha20:22
clarkbso ya I bet something is installing gpg but not gnupg20:22
clarkbbefore this dind't matter becase python-apt installed gnupg20:22
clarkbto fix this we should add gnupg to infra-package-needs20:22
clarkb(or revert https://review.opendev.org/#/c/716785/3 )20:23
fungiyeah, i concur, adding gnupg to infra-package-needs ought to solve it20:23
mordred++20:23
mnaserthat seems most reasonable to me too20:23
fungibut who knows what other package we might also be missing beyond that20:23
fungii guess odds are not many20:23
clarkbfungi: well its just things that python-apt and libselinx-python would pull in20:23
fungiso better to play whack-a-mole with any we find than roll back20:23
clarkboh and libselinux-python wasn't a thing on debuntu20:24
clarkbso just python-apt20:24
fungithe upshot of this is that we're not getting images updated, right?20:24
clarkbhttps://packages.ubuntu.com/bionic/python-apt20:24
fungiit's not breaking any jobs?20:24
clarkbfungi: no we found this because jobs broke20:24
corvusfungi: no, we have broken images breaking jobs20:24
fungiahh20:24
corvusmostly ones that install things (like docker or yarn) from apt repos20:24
mnaser^ and need to add apt-keys20:25
fungioh, more specifically things which try to use gpg to load third-party repository keys with apt-key?20:25
corvusyep20:25
fungimakes sense why we wouldn't have spotted that sooner20:25
corvusfungi: we spotted it pretty quickly :)20:26
corvuschange merged 4 hours ago, the image is 2 hours old20:26
corvusbut yeah, it didn't break *everything*20:27
corvuswhich probably means there aren't too many moles to whack20:27
fungineat20:27
corvusis someone going to push up a fix we can merge quickly, or should i push up a change to pause bionic builds?20:27
mnaseri can do that quickly20:27
fungithanks mnaser!20:27
fungistanding by to review in that case20:28
clarkbcorvus: working on it20:28
clarkbjust double checking package names on various systems20:28
clarkbI think 'gnupg2' will work universally20:28
fungiyeah, gnupg2 in newer debian/ubuntu is just a transitional package but pulls in gnupg which in turn depends on gpg and gpg-agent and more20:28
mnaseroh we need to play "figure out the package name"20:28
corvusi'll go ahead and delete the current bionic image; we might end up building one more before the cleanup lands, but we can roll the dice on that20:28
mnaserok clarkb is doing that20:29
clarkbjust trying to confirm on suse now20:29
fungithanks corvus!20:29
corvus#status log deleted ubuntu-bionic-0000104264 image due to missing gpg-related packages20:29
openstackstatuscorvus: finished logging20:29
mnaserdoesnt seem to be gnupg2 or gnupg20:30
mnaserbut do we need it on suse-based system anyways?20:30
openstackgerritClark Boylan proposed openstack/project-config master: Install gpg tooling on dib images  https://review.opendev.org/71738620:31
clarkbthat should do it I think20:31
clarkbmnaser: we want to be consistent I think20:31
clarkbmnaser: ya I saved suse for lsat bceause it is what I run20:32
clarkbmnaser: I just have to run zypper search locally :)20:32
mnaserclarkb: oh cool, i enjoy using docker for these type of things lately20:32
mnaserdocker run -it --rm opensuse/tumbleweed /bin/bash and go20:32
mordredmnaser: same20:35
mordredclarkb: well - amusingly enough this should test our new update to the deps list :)20:39
clarkbindeed20:39
corvusmnaser: do i recall correctly that all the machines using an image in vexxhost have to be deleted before the image itself can be completely deleted?20:39
clarkbcorvus: ya due to boot from volume + ceph behavior20:40
corvusk.  no problem; i just noticed that the bionic image was sticking around in delete state and wanted to confirm that's expected20:40
corvusshould clear up eventually20:40
clarkbI've just noticed that ubuntu gnupg2 is actually a transitional package20:41
clarkbfungi: ^ should I push a followon change that will switch to gnupg on debuntu?20:41
clarkb(the change that is approved should work, mostly thinking about future distro releases)20:42
fungiclarkb: it's probably fine for now20:42
fungiwhen that stops working (which won't be for a year or two probably) we'll start getting an error in the image builds anyway20:43
fungiso we can always wait to fix that until we actually have to20:43
clarkbk20:44
fungiclarkb: especially since part of what's transitional about the gnupg2 package is that it ships symlinks from /usr/bin/gpg2 to /usr/bin/gpg and the like20:45
fungiwhich folks are probably going to want for a long time to come20:45
clarkboh ya those are useful20:45
fungiso there's no indication as to when (if ever) that package would go away20:45
fungiso many custom scripts and tutorials still reference `gpg2`20:46
fungiand from the package maintainers' point of view, continuing to provide that package costs basically nothing20:47
fungiso there's little incentive to try to get rid of it20:47
openstackgerritMerged openstack/project-config master: Install gpg tooling on dib images  https://review.opendev.org/71738620:59
mordredinfra-root: infra-prod-service-nodepool is running in the new deploy pipeline :)21:00
mordredand it has run21:04
clarkbnow we need to trigger an image build but there is probably already one running that we need to complete first (and when that one completes deleting its images will trigger the rebuild for us)21:05
mordredyeah21:05
clarkbhrm I don't think the files updated on nb01 at least21:05
clarkbmordred: /etc/nodepool/elements/infra-package-needs/ doesn't seem to have updated on nb01 or nb0221:06
clarkbdid that deploy job run properly?21:07
mordredclarkb: it looks like it - I mean- it ran the playbook - lemme go read the playbook21:07
clarkboh wai21:08
clarkbI know21:08
clarkbthe nodepool playbook is only for nb04 I bet21:08
clarkbnb01-03 rely on puppet21:08
fungithat's it exactly21:08
mordredah. wow. yeah. so ... yeah21:09
mordredso we'll get it in the next pulse21:09
fungido we have a way to kill the in progress builds?21:09
clarkbya and nb04 is up to date21:09
mordredI think ianw was planning on replacing 1-3 with ansible/docker soon21:09
clarkbfungi: it will just restart21:09
fungior are we better off stopping and restarting the builders once puppet runs21:09
openstackgerritMerged opendev/system-config master: Switch to deploy pipeline for deployments  https://review.opendev.org/71735621:09
clarkbfungi: ya I think that otherwise nodepool will immediately start a new build21:10
clarkbthe good news here is nb04 updated as epxceted so that side of the system is working well :)21:10
fungii mean, this change is going to affect all the images regardless, so we might as well wait for it to land and build everything anew21:10
fungii'll go stop the nodepool-builder service on 01-03 unless there are objections21:11
mordredsounds fine to me21:11
corvus++21:11
clarkbyup that should be fine21:11
mordredso - for things like nodepool now ...21:12
fungiokay, term signals sent to all via initscripts21:12
mordredshould we remove nb* from remote_puppet_else and add a puppet play to the service-nodepool playbook?21:12
mordredthat way we'll properly trigger all of nodepool on those project-config patches21:12
clarkbmordred: maybe check with ianw monday and see? since I know ianw said moving to docker nodepool builder was one of his next things21:13
fungisome still have some long-running subprocesses underway, like qemu-immg invocatins21:13
clarkband if thats done the old servers just go away21:13
openstackgerritMonty Taylor proposed opendev/system-config master: Run puppet on old nb0[1-3] in nodepool playbook  https://review.opendev.org/71739621:16
mordredclarkb: ^^ I think we can wait for ianw to land it- but I think we should do that until the other builders are replaced - otherwise we might miss project-config changes and have to wait for the nightly21:16
mordredclarkb: but also - yeah - just spinning up nb01-3.opendev.org and deleting the old ones will also be nice21:18
clarkbmordred: the cron is still running though right?21:18
clarkbso puppet is happening hourly21:18
mordredclarkb: well - it is now until we land the remote_puppet_else change :)21:18
clarkboh gotcha21:18
fungithe builders have gone idle now, wrapped up pending child processes21:19
mordredalthough we coudl also just add a trigger21:19
corvusmordred: what should we be doing now?  merge meetpad change?21:19
mordredyeah.21:20
mordredclarkb: I forgot - we have a catchall for running remote_puppet_else hourly at the end of the stack21:20
corvusclarkb: https://review.opendev.org/71677121:20
mordredhttps://review.opendev.org/#/c/717064/321:21
mordred(just - if we land through there we're back to hourly catch-all runs)21:21
corvusmordred: do you want to proceed one at a time through that, or do a batch?21:21
mordredcorvus: at this point I'm pretty happy with how it's running21:21
mordredI think we can do a batch - main thing is to make sure we're not missing triggering from something else on any of them21:21
mordred(of course, we can always add that as a followup as needed)21:22
mordredso - from my pov - I don't think there's any reason to hold off on any of them21:22
clarkbI think they've got the reviews they need now21:23
mordredwell - we need to pause on https://review.opendev.org/#/c/717058 for at least a deploy21:23
clarkbmordred: if you want to +A what you are comfortable with21:23
clarkbfwiw batches are nice simply if something goes wrong its less things to debug21:24
clarkbbut also take longer when things are going well :)21:24
mordredyeah. well - I clicked +A up to https://review.opendev.org/#/c/717053 and then stopped because it's voteless21:24
mordredso let's call that a natural pausing place :)21:24
corvusi'm going to stop at 717053; that seems like a really good place to stop21:24
corvusmordred: heh, yeah :)21:25
corvusnot opposed to doing that on a friday afternoon (or saturday), but that seems like a good one to merge only if everything else is stable21:26
mordredcorvus: re: comment - it was mostly because it kept confusing my eyeballs - and zuul-registry is running in the registry service, so preview felt similarly like an independent service21:28
mordredcorvus: but- I don't have strong feelings on it in either direction21:28
corvusmordred: yeah, it's a +2 comment :)21:29
clarkbhttps://etherpad.openstack.org/p/IBLWO1WBBc email draft for getting people on mailing lists21:29
clarkbI worry sending that on friday afternoon might end up leading to it being ignored21:30
corvusclarkb: yeah, next week ++21:30
clarkbbut can either send that nowish or monday morning21:30
corvus1) sign up for service discuss.  2) sign up for service-announce.  3) wash hands.21:30
mordredcorvus: I feel like wash hands should be in the list more21:31
corvusclarkb: looks good, made some suggestions21:35
clarkbcorvus: those edits look great, thanks21:36
fungiclarkb: yes, monday will in theory get more eyeballs21:38
fungiclarkb: do we want to also use service-discuss to answer service usage questions? if so, the announcement doesn't give that impression with "plan changes to services, notify of meetings, and otherwise communicate about OpenDev" (though i guess it could technically fall into the "otherwise communicate about" category)21:43
clarkbya I was trying to give examples that set it apart from announce21:44
clarkbanswering service usage questions would fit under otherwise communicate but we should call it out21:44
fungidefinitely asking for help sets it apart from the scope of the announce ml21:44
fungiif it's a place we expect users and collaborators on these services to go for help with them, then i do think it probably merits direct mention21:45
clarkbfungi: how does that look?21:45
openstackgerritMatthew Thode proposed opendev/glean master: write one resolv config  https://review.opendev.org/71733921:45
fungiclarkb: perfect!21:46
fungii tried to capture that in the ml description/info texts too21:46
openstackgerritMerged opendev/system-config master: Run meetpad in zuul  https://review.opendev.org/71677121:51
openstackgerritMerged opendev/system-config master: Run mirror-update in zuul  https://review.opendev.org/71677221:55
openstackgerritMerged opendev/system-config master: Run nameserver in zuul  https://review.opendev.org/71676421:55
*** DSpider has quit IRC21:56
*** hashar has quit IRC22:05
openstackgerritMerged opendev/system-config master: Run mirror in zuul  https://review.opendev.org/71704822:08
openstackgerritMerged opendev/system-config master: Run static in zuul  https://review.opendev.org/71704922:08
openstackgerritMerged opendev/system-config master: Add file matchers for roles used via include_role  https://review.opendev.org/71705022:08
openstackgerritMerged opendev/system-config master: Run backup in zuul  https://review.opendev.org/71705122:08
openstackgerritMerged opendev/system-config master: Run registry in zuul  https://review.opendev.org/71705222:08
clarkbfungi: nb01 seems to have new elements22:11
clarkbshould we start the services there?22:11
clarkbnb02 looks good as well22:12
corvusthat's the current batch of approved changes merged22:12
fungiclarkb: yeah, i suppose 03 is less critical but i'll start them all back up now22:12
clarkbnb03 also. So ya I think we can start builers again?22:12
clarkbfungi: they should all be good now22:13
fungiall started again22:13
corvusclarkb, mordred: are the prod-service playbooks not running when we add jobs?22:27
clarkbcorvus: I don't think so because they are in deploy?22:28
clarkb(I want to say the auto run thing is only for pre merge things? but maybe I'm wrong)22:28
corvusyeah, it may be, but that may be an omission we should see if we can lift22:28
clarkbwe've definitely been merging changes to trigger them22:28
clarkbrather than relying on the self run behavior (because I don't think that happens)22:28
corvusi wonder why it doesn't trigger22:31
*** dpawlik has quit IRC22:32
clarkbhttps://nb02.openstack.org/ubuntu-bionic-0000104265.log is the log of hte image that should fix bionic for us22:32
clarkbbuild is in progress22:32
clarkb2020-04-03 22:35:45.010 | > Get:155 http://mirror.dfw.rax.openstack.org/ubuntu bionic-updates/main amd64 gpg-agent amd64 2.2.4-1ubuntu1.2 [227 kB]22:40
fungilookin good, yep22:40
mordredcorvus: yeah - I've been thinking that's weird23:00
corvusclarkb, fungi, mordred: looking into why new jobs don't run in deploy: i think the change to add the job is merged; it's enqueued into deploy, it sees that it's a config update, so in enqueues a config update event, it processes the pipelines (it's in deploy) and sends out a merger job for it, then it starts the main loop over, reconfigure events are handled first, so it stops the loop and performs a23:02
corvusreconfigure event, it resumes, some time passes, the merge job comes back, it generates a diff between the requested and current config, and there is none, so it reads it as not updating the job.23:02
corvusin short: by the time zuul evaluates whether it's a change to the config, the running config has changed, so it is no longer a change to the config23:03
fungithat mind-bending23:03
fungibut yes23:03
fungiit does make sense23:04
corvusand i think that's behavior is pretty solid (ie, not subject to races).23:04
fungiso we need it comparing against the pre-reconfigure config23:04
clarkbmy mental image now is of zuul-scheduler and zuul-merger giving each other a high five23:04
fungibwahahaha23:04
* fungi did actually lol23:05
clarkb"we did it!"23:05
corvusclarkb: i imagine them attempting a high five and just missing23:05
fungi*woosh*23:05
mordredcorvus: that's amazing23:05
corvus[i briefly investigated what would happen if there's a race -- if that were possible, i think it could start a deploy job, and then abort it when the config updated.  but i don't think that's actually possible (and i poked around with a unit test to try to make it happen and could not).]23:07
corvusso yeah, if we wanted that to happen, we'd have to compare to an earlier config... and, er, saving old versions of the layout may not be the best idea.23:08
corvusthat's pretty much our most effective memory leak creation tool :)23:08
fungias evidenced by prior cases where we failed to reap extra copies of the layout in a timely fashion23:09
corvusthis may be a topic to file away and revisit if we make any changes that might make that easier23:10
clarkbthe bionic image has built23:15
clarkbusually we get new images within about 10 minutes in some clouds23:15
*** tosky has quit IRC23:16

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!