Thursday, 2021-02-25

stewie925clarkb:  I hope you understand my issue now hehe.  hope Ubuntu One can help fix that issue00:00
*** tosky has quit IRC00:00
clarkbstewie925: no, that hasn't been something they would typically fix00:00
clarkbI think if you set the ubuntu one email back to the original email address it may start using the old openid again though (its a consistent hash)00:00
clarkband then you can log in to the old account but not the new (hwoever I'm not certain of that)00:00
clarkbprior to the big gerrit upgrade we would address these issues with db updates00:01
stewie925clarkb: you are right - I cannot log in with the new email address -  I can only log in with old email address00:01
clarkbbut now the accounts are not in the sql db and are instead in the notedb. And notedb won't allow us to push fixes until we fix all the other issues which is quite annoying00:01
clarkbits possible we can remove conflicting ids from one account or another to alleviate the issue though if you just wanted to move forward with a new account and the new emails (I suspect you cannot log in with the new email because it conflicts with the old account)00:02
stewie925clarkb:  no rush, I can go along with this for now... :)00:02
clarkbstewie925: I'm not sure what timezone you are in. Its getting close to when I need to start sorting out dinner and family time. However, if you are around tomorrow or friday with some overlap on pacific time I'm happy to dig into this with you more directly00:03
clarkbstewie925: and you can share some of the more private info like account details and I can see what the gerrit backend looks like00:03
clarkband from that hopefully we can figure out a plan00:03
stewie925clarkb: thank you so much.... I m in Pacific time.00:03
stewie925clarkb:  what is the best time to discuss with you?  maybe 3 PM PST Friday ?00:04
clarkbthat works for me, I can do just about anytime between 8am and 3pm though I think00:04
stewie925you're Pacific too, I guess?00:05
clarkbif you find time tomorrow ping me and I'll see what I'm at with zuul executor replacments (I should be able to page that out easily enough)00:05
stewie925oh thank you clarkb -   I'll catch up with you tomorrow (around 3 pm) then:) .   you have a great night.00:06
clarkbyou too00:06
ianwclarkb / corvus : i'm starting to think opendev is the right place to do it ...
clarkbianw: but if the job is running in the openstack tenant wouldn't ^ be ignored?00:09
ianwclarkb: i'm really not sure with this one :/00:11
corvusianw: i'm going off of
clarkbcorvus: ianw oh ya in that case I agree opendev is correct here, but I'm not sure why we need it in that tenant now00:14
clarkbI did leave a note on the above change though00:16
ianwclarkb: i don't think i've used "depends-on" with that project before.  so it's only been cloning as part of "required-projects" for the system-config jobs, where it has the override to "main"00:16
ianw tried it, and all the other jobs now fail with the wrong branch; so that's the gist of why it needs to be somewhere00:17
clarkbianw: I woudl expect we need to add that project to openstack and set its default branch for that to work though since system-config depends on shouldn't evaluate the opendev config but maybe things aren't as separated as I expect00:18
ianwi think i should remove the override-checkout in too00:19
ianwclarkb: maybe both?00:19
clarkbya we may want it in both places00:21
*** stewie925 has quit IRC00:24
ianwclarkb: not sure i folow your comment, there's no projects in opendev tenant that don't start with opendev/* ?00:26
clarkbianw: those ones for example00:27
clarkbbut that list of jobs is also empty so it doesn't really matter, in that case I think we should simply consider dropping the regex match and template entirely though (to avoid confusion)00:27
ianwhrm, ok.  alright, let's drop it00:27
openstackgerritIan Wienand proposed opendev/project-config master: zuul-results-summary: set default branch
clarkbI've +2'd it but may want fungi and corvus to weigh in on the removal of the system default template before approving just to make sure I'm not missing something imortant00:34
corvuswhy drop it?00:39
fungii think the idea was we'd add required jobs into that template, right?00:40
corvus-1 on change; clarkb, ianw, fungi: pls see comment in project-template00:40
ianwcorvus: ok, but then we want to *exclude* the zuul-plugin right, because it's not an opendev project?00:41
clarkbfungi: I think it was that we had them at one point00:44
* clarkb finds the comment00:45
clarkbah its a zuulism thing00:45
clarkbya I think we want a way to exclude the project from pipeline config but also set its branch config00:45
openstackgerritIan Wienand proposed opendev/project-config master: zuul-results-summary: set default branch
corvusianw: that would be most correct, yes; otherwise it may report merge failures.00:46
ianwcorvus : ^^ i think *that* might do it ...00:46
corvusto be fair, the config is already "wrong", you just happened to have discovered it while changing this setting.  the 2 things aren't really related.00:47
openstackgerritMerged opendev/project-config master: zuul-results-summary: set default branch
*** hamalq has quit IRC00:56
ianw... system-config-run-base-ansible-devel error01:01
ianwit seems ^^ maybe isn't sufficient :/01:01
ianwcorvus: so the results of seem to show the same, thing, do we need as well?01:07
corvusianw: yes; looks like it's in both tenants via a yaml anchor01:09
openstackgerritMerged opendev/system-config master: install-docker: move rsyslog handler earlier
openstackgerritMerged openstack/project-config master: zuul-results-summary: set default branch
openstackgerritIan Wienand proposed opendev/system-config master: zuul-summary-status : handle SKIPPED jobs
ianwyay, jobs seems to be running, thanks for the help :)01:36
*** mlavalle has quit IRC02:13
*** LowKey has quit IRC02:20
*** auristor has quit IRC02:29
*** auristor has joined #opendev02:43
*** ysandeep|away is now known as ysandeep03:06
*** ysandeep is now known as ysandeep|bbl03:17
*** sgw has quit IRC04:02
*** sgw has joined #opendev04:02
*** whoami-rajat has joined #opendev04:18
*** dviroel has quit IRC04:45
*** ykarel has joined #opendev04:55
*** bhagyashri|ruck is now known as bhagyashri|rover05:15
*** marios has joined #opendev05:53
*** ralonsoh has joined #opendev06:01
openstackgerritMerged opendev/system-config master: install-docker: remove fix from prior change
*** yoctozepto0 has joined #opendev06:04
*** yoctozepto has quit IRC06:05
*** yoctozepto0 is now known as yoctozepto06:05
*** mkowalski_ has joined #opendev06:35
*** ysandeep|bbl is now known as ysandeep06:41
*** zbr has quit IRC06:43
*** icey has quit IRC06:58
*** icey has joined #opendev06:58
*** icey has quit IRC07:01
*** icey has joined #opendev07:05
*** icey has quit IRC07:08
*** slaweq has joined #opendev07:12
*** lpetrut has joined #opendev07:19
*** slaweq has quit IRC07:19
*** slaweq has joined #opendev07:23
*** eolivare has joined #opendev07:40
*** jpena|off is now known as jpena07:45
*** rpittau|afk is now known as rpittau07:57
*** fressi has joined #opendev08:05
*** brinzhang has quit IRC08:05
*** brinzhang has joined #opendev08:07
*** brinzhang has quit IRC08:07
*** brinzhang has joined #opendev08:07
*** brinzhang has quit IRC08:08
*** brinzhang has joined #opendev08:09
*** andrewbonney has joined #opendev08:13
*** ykarel_ has joined #opendev08:31
*** ykarel has quit IRC08:33
*** ykarel_ is now known as ykarel08:34
*** toomer has joined #opendev08:37
*** tosky has joined #opendev08:38
*** jaicaa has quit IRC08:44
*** jaicaa has joined #opendev08:47
openstackgerritLianhao Lu proposed openstack/project-config master: intel-nfv-ci-tests: move zuul definition into repo
*** icey has joined #opendev08:57
*** lpetrut_ has joined #opendev09:54
*** yoctozepto9 has joined #opendev09:55
*** jhesketh_ has joined #opendev09:57
*** avass_ has joined #opendev09:58
*** dhellmann_ has joined #opendev09:58
*** SWAT_ has joined #opendev09:59
*** Dmitrii-Sh7 has joined #opendev09:59
*** arxcruz has quit IRC10:00
*** arxcruz has joined #opendev10:01
*** lpetrut has quit IRC10:03
*** yoctozepto has quit IRC10:03
*** dhellmann has quit IRC10:03
*** jhesketh has quit IRC10:03
*** Dmitrii-Sh has quit IRC10:03
*** SWAT has quit IRC10:03
*** avass has quit IRC10:03
*** Dmitrii-Sh7 is now known as Dmitrii-Sh10:03
*** yoctozepto9 is now known as yoctozepto10:03
*** dhellmann_ is now known as dhellmann10:03
fricklerinfra-root: looking at disk usage on bridge, it looks to me like a lot is actually being used by /root/{.ansible,.cache}, which sounds like it could be easily cleaned up? (14G according to du)10:06
fricklerin particular there are almost 500k files sized 14k in /root/.cache/python-entrypoints taking up 7.4G in total10:10
ianw++ sounds like a leak10:29
ianwprobably a cron job to mtime delete is a good idea?10:29
*** toomer has quit IRC10:54
*** dtantsur|afk is now known as dtantsur10:56
*** dviroel has joined #opendev11:03
*** sshnaidm|afk is now known as sshnaidm|pto11:22
mordredwow, what even are /root/.cache/python-entrypoints ?11:58
*** jpena is now known as jpena|lunch12:27
*** ysandeep is now known as ysandeep|afk12:29
*** zoharm has joined #opendev12:31
*** ysandeep|afk is now known as ysandeep13:00
*** roman_g has joined #opendev13:01
*** zbr has joined #opendev13:13
*** lpetrut__ has joined #opendev13:14
*** lpetrut_ has quit IRC13:16
ttxit does sound scary13:20
*** jpena|lunch is now known as jpena13:26
*** ykarel has quit IRC13:32
*** tkajinam has quit IRC13:36
*** ykarel has joined #opendev13:37
*** iurygregory has quit IRC13:43
*** iurygregory has joined #opendev13:44
*** zbr7 has joined #opendev13:47
*** zbr has quit IRC13:49
*** zbr has joined #opendev13:51
*** zbr7 has quit IRC13:53
*** yoctozepto has quit IRC14:01
*** yoctozepto has joined #opendev14:01
*** zbr0 has joined #opendev14:11
*** zbr has quit IRC14:13
*** zbr0 is now known as zbr14:13
*** whoami-rajat has quit IRC14:38
openstackgerritJeremy Stanley proposed opendev/system-config master: Correct OpenStack Security URL in sites
*** zbr1 has joined #opendev14:54
*** avass_ is now known as avass14:56
*** zbr has quit IRC14:56
*** zbr1 is now known as zbr14:56
*** fressi has quit IRC15:03
*** sgw has left #opendev15:12
*** marios has quit IRC15:33
openstackgerritMateusz Kowalski proposed openstack/diskimage-builder master: Change paths for bootloader files in iso element
*** ykarel has quit IRC15:38
*** ysandeep is now known as ysandeep|dinner15:41
*** zbr3 has joined #opendev15:50
*** lpetrut__ has quit IRC15:51
clarkbfrickler: oh nice catch, I had missed that dir (I think because I was using du -hs * to find things and * won't match .)15:51
*** zbr has quit IRC15:52
*** zbr3 is now known as zbr15:52
*** marios has joined #opendev15:53
clarkbI suspect the entrypoints cache is there to speed up how slow loading entrypoints on process startup is with python?15:53
clarkbwe can probably trim it back though and then let it repopulate what it actually needs currently15:53
fungishould be able to entirely delete it, and yeah then work out why it keeps growing15:58
clarkbI'm going to find breakfast but then will start on booting a focal server15:59
clarkbif you know of anything different about those other than /var/lib/zuul being the ephemeral drive mount let me know15:59
fungiyeah, and i guess we did that just for the added space?16:00
clarkbyup for the job working dirs16:01
clarkbI looked over an existing server a bit yesterday to see if there was anything else to consider but I think that is it. On the testing side ensuring afs is happy will be important but that will all be after we finish deploying it16:01
*** ysandeep|dinner is now known as ysandeep16:05
*** zbr9 has joined #opendev16:13
*** zbr has quit IRC16:15
*** zbr9 is now known as zbr16:15
*** ysandeep is now known as ysandeep|PTO16:16
clarkbinfra-root system-config/launch/ seems to hardcode /opt as the xvde mount path. Do you know how we were convincing it to mount at /var/lib/zuul instead?16:23
clarkbI can manually switch it over to /var/lib/zuul then resync the contents back over to /opt on the root device16:23
fungii don't, though i wouldn't be surprised if that was just done manually after the systems were booted16:24
clarkbya, I'm beginning to suspect that may have been what was done16:24
fungii.e. edit fstab, reboot16:24
*** mlavalle has joined #opendev16:25 copies /opt contents onto the new device and then I thnk config management may write some things there too. So probably need to resync it back again16:25
clarkbbut ya that should work16:25
fungii think we only put stateless stuff in /opt which will get recreated automatically by configuration management when it runs, right?16:25
fungithough i suppose for some older services (not zuul) we do software deployment inside /opt instead of /srv16:26
fungiso as a general rule, deleting the contents of /opt on some servers could leave services offline/broken until configuration management reran16:27
clarkbanother option may be to run launch node out of my homedir with a modified make_swap that replaces /opt/ with /var/lib/zuul16:27
clarkbI might start there since automating it that way will simplify the other 11 boots we need to do16:28
clarkbthough it isn't always clear to me when it is safe to run ansible out of your homedir vs the zuul copy now16:30
clarkbI think for launch node it should be fine since it is fairly detached from everything else16:30
clarkbthat is running now, we'll see how it goes16:38
*** zbr3 has joined #opendev16:39
*** zbr has quit IRC16:41
*** zbr3 is now known as zbr16:41
clarkbI think that worked actually /me makes dns and inventory updates16:49
*** jpena is now known as jpena|off16:54
openstackgerritClark Boylan proposed opendev/ master: Add to DNS
openstackgerritClark Boylan proposed opendev/system-config master: Add
mnasiadkaGood afternoon16:58
mnasiadkaIs there any way to debug DISK_FULL jobs?16:58
clarkbinfra-root ^ I suspect both of those changes are ready to go if they look correct to you16:59
clarkbmnasiadka: can you link to an example?16:59
mordredclarkb: the biggest issue with running in your home dir is the host_vars in system-config16:59
mnasiadkanever hit DISK_FULL on that job, so that's why I'm curious :)16:59
clarkbmnasiadka: generally though the disk available to our test jobs isn't infinite and you need to operate within the bounds of the system16:59
mordredclarkb: (I don't think you'll have an issue with this)16:59
clarkbmordred: ya I made sure the latest commit was checked out and it seemed to do the right things (but also launch is suepr basic ansible)17:00
fungimnasiadka: clarkb: the DISK_FULL result generally means the allowed workspace on the executor though, so has more to do with trying to collect too much log or artifact data from a build17:00
mnasiadkafungi: Ah, now that makes more sense17:00
clarkbah yup I see it is zuul reporting that message not smething from the remote ansible17:01
mnasiadkafungi: Any idea what's the limit?17:01
openstackgerritMerged opendev/irc-meetings master: Remove usused I18n meeting slot
*** roman_g has quit IRC17:01
*** marios is now known as marios|out17:02
fungimnasiadka: i think we're relying on a zuul default for that, looking now to confirm17:02
*** klonn has joined #opendev17:04
*** klonn has quit IRC17:04
mnasiadkafungi: well, I'm fetching logs from a similar job, just based on centos, and I see we have 6402 files in the attached logs, so probably ara-html report is the one that's doubling the space used...17:06
clarkbmnasiadka: fungi
clarkbthat is MB17:12
mnasiadka4.8 gig, we shouldn't produce that amount of logs :)17:12
fungii'm finding which executor that build ran from to get some details17:13
mnasiadkaI'll download those 6402 files and check what takes that amount of storage.17:13
fungilooks like ze0717:13
mnasiadka(but it takes some time)17:13
fungi2021-02-25 15:02:23,234 WARNING zuul.ExecutorDiskAccountant: /var/lib/zuul/builds/59eae5cf0c1740faba7e59a123fc67ab is using 6319MB (limit=5000)17:14
fungiheh, you're not my vi window17:16
*** zbr7 has joined #opendev17:17
mnasiadka6319, nice17:17
mnasiadkafungi: thanks for clarification17:17
fungiat one point i thought we included a du -s of the top-level subdirectories under those circumstances, but i'm not finding it17:18
clarkbfungi: do a remote df when jobs fail (as a way to detect running out of disk on the remote side17:19
clarkbbut not sure about any printout of the build working dir on the executor17:19
*** zbr has quit IRC17:19
*** zbr7 is now known as zbr17:19
clarkb*we do17:22
clarkbfungi: I +2'd probably safe to single core approve that one?17:28
fungiit was the upshot of a post to the openstack-discuss ml where a user was looking for the security site17:28
fungii'm also working with the foundation webdevs to get some links to that restored17:29
clarkbok I approved it too now17:30
openstackgerritMartin Kopec proposed opendev/system-config master: refstack: Edit URL of public RefStackAPI
*** rpittau is now known as rpittau|afk17:34
*** zoharm has quit IRC17:35
*** zbr3 has joined #opendev17:38
*** dtantsur is now known as dtantsur|afk17:39
openstackgerritMateusz Kowalski proposed openstack/diskimage-builder master: Change paths for bootloader files in iso element
*** zbr has quit IRC17:40
*** zbr3 is now known as zbr17:40
*** zbr5 has joined #opendev17:42
*** zbr has quit IRC17:44
*** zbr5 is now known as zbr17:44
openstackgerritMerged opendev/system-config master: Correct OpenStack Security URL in sites
*** eolivare has quit IRC18:05
mnasiadkafungi: downloaded logs, there's like 700MB of them - sure that's a lot - but what's taking the rest on the executor?18:06
clarkbansible itself runs out of there18:07
clarkbcould it be ansible stashing things in the build workdir that aren't logs (caches or something?)18:07
clarkbiurygregory: looking at is this something that hashtags might solve better?18:15
clarkbiurygregory: you could add a hashtag for BackportCandidate and one for DoNotBackport ?18:16
fungimnasiadka: it's also possible the failed build failed in such a way that it generated a runaway bunch of log content18:16
clarkbI know other projects aren't doing it this way but ya'll are testing the hashtags and I wonder if this would be a good use casefor them18:16
mnasiadkafungi: makes sense18:17
fungiwe've seen this before, for example with mysql slow query logs18:18
fungibut basically something breaks in such a way that one or more services go into a rapid loop logging the same set of error messages over and over, possibly until the job times out18:18
fungizuul's disk accountant is really just a backstop to prevent a build from using too much disk on the executor, and stops the build cold if it exceeds that threshold18:20
mnasiadkawell, let me try to trim the log size a bit and let's see if that helps18:20
fungiso unfortunately we don't get any more detail than the amount of disk it was using at the time it was aborted18:20
clarkbfungi: zuul is happy with and if you have time to look those over and add ze01.opendev.org18:29
openstackgerritMerged openstack/project-config master: Create inspur/venus-dashboard repository
*** marios|out has quit IRC18:36
fungiclarkb: approved both, am around if help troubleshooting is needed too18:39
clarkbthanks! I'll keep an eye out as well18:39
clarkbI don't think the executor will auto start on the new host anyway so we can check it first, then start the service, then stop the service on ze01.openstack.org18:40
fungithat's what i figured, yeah18:41
*** andrewbonney has quit IRC18:49
openstackgerritMerged opendev/ master: Add to DNS
openstackgerritMerged opendev/system-config master: Add
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add install_args role var
*** ralonsoh has quit IRC19:24
iurygregoryclarkb, hey! sorry for the delay, so the idea is: with the label for Backport-Candidate we will try to have something using gerrit API to check if the label is present and will add the hashtag "ironic-backport" so we can check the patches that will need more easily =)19:53
clarkbiurygregory: that feels like redundant effort19:54
clarkbwhy not just add the tag instead of voting +/-1 ?19:54
fungido the same users who have access to vote on backport-candidate also have access to set hashtags?19:58
iurygregorydue to the fact that people may forget =)19:58
iurygregoryfungi, yep only "cores"19:58
fungigot it, so it's not a workaround for one being more available than the other19:58
iurygregoryalso the owner of the patch can set hashtags on it19:59
clarkbif it were me I'd chose one or the other but not use both20:00
clarkbif labels work better because they show up when hitting the review button then just use the labels and don't bother with hashtags20:00
clarkbbut trying to use both just seems like it would be a mess of missing things and incomplete lists on both sides20:00
fungimaybe that calls for a not-a-backport-candidate hashtag then ;)20:01
*** stevebaker has quit IRC20:02
*** hamalq has joined #opendev20:02
iurygregorymaybe we can do a test having the Backport-Candidate label in ironic acl and tell you how it goes? =)20:02
*** stevebaker has joined #opendev20:02
*** logan- has quit IRC20:35
*** dtantsur has joined #opendev20:35
*** lbragstad_ has joined #opendev20:35
*** icey_ has joined #opendev20:36
*** ysandeep has joined #opendev20:36
*** mugsie_ has joined #opendev20:36
*** persia_ has joined #opendev20:36
*** zbr has quit IRC20:36
*** fdegir has quit IRC20:36
*** icey has quit IRC20:36
*** zigo has quit IRC20:36
*** stephenfin has quit IRC20:36
*** slaweq has quit IRC20:36
*** persia has quit IRC20:36
*** melwitt has quit IRC20:36
*** mugsie has quit IRC20:36
*** osmanlicilegi has quit IRC20:36
*** swest_ has joined #opendev20:36
*** Tengu has quit IRC20:36
*** melwitt has joined #opendev20:36
*** corvus has quit IRC20:36
*** tosky_ has joined #opendev20:37
*** dhellmann_ has joined #opendev20:37
*** dtantsur has quit IRC20:37
*** dtantsur has joined #opendev20:37
*** ysandeep has quit IRC20:37
*** ysandeep has joined #opendev20:37
*** logan- has joined #opendev20:38
*** JohnnyRa1 has quit IRC20:38
*** mtreinish has quit IRC20:38
*** ysandeep|PTO has quit IRC20:38
openstackgerritDavid Moreau Simard proposed openstack/project-config master: Retire ara projects
*** gibi has quit IRC20:38
*** parallax has quit IRC20:38
*** mattmceuen has quit IRC20:38
*** Tengu has joined #opendev20:38
*** mordred has quit IRC20:39
*** mtreinish has joined #opendev20:39
*** parallax has joined #opendev20:39
*** hillpd_ has joined #opendev20:39
*** jrosser has quit IRC20:40
*** Eighth_Doctor has quit IRC20:40
*** hillpd has quit IRC20:40
*** hillpd_ is now known as hillpd20:40
*** tosky has quit IRC20:40
*** noonedeadpunk has quit IRC20:40
*** kevinz has quit IRC20:40
*** guillaumec has quit IRC20:40
*** dhellmann has quit IRC20:40
*** dhellmann_ is now known as dhellmann20:40
*** jpenag has joined #opendev20:40
*** jrosser has joined #opendev20:40
*** JohnnyRa1 has joined #opendev20:40
*** mnasiadka has quit IRC20:41
*** tosky_ is now known as tosky20:42
*** jpena|off has quit IRC20:42
*** dtantsur|afk has quit IRC20:42
*** mnasiadka has joined #opendev20:42
*** swest has quit IRC20:43
clarkbiurygregory: one small thing with the syntax of one of the group files I noted in review20:43
*** lbragstad has quit IRC20:43
*** corvus has joined #opendev20:44
iurygregoryclarkb, tks!20:49
*** mattmceuen has joined #opendev20:54
*** mordred has joined #opendev20:58
*** guillaumec has joined #opendev21:10
*** Eighth_Doctor has joined #opendev21:20
openstackgerritJeremy Stanley proposed opendev/git-review master: Don't test rebasing with unstaged changes
openstackgerritJeremy Stanley proposed opendev/git-review master: Add a test helper for creating unstaged changes
clarkbopenafs is installing on now21:41
clarkbshould be able to turn it on soon21:41
clarkbinfra-root zuul is done zuuling I'm going to start it now if there are no objections21:46
clarkband if that looks reasonably happy I'll try to trigger a graceful stop on through the container21:47
clarkbfungi: left some notes on those git review changes21:55
clarkbI'm not hearing objections I'll start zuul-executor on now21:57
clarkbit is updating lots of repos21:58
clarkbnow to gracefully stop the executor on ze01.openstack.org21:58
clarkbthe docs say zuul-executor pause will allow the jobs to drain out22:00
clarkbI didn't realize that existed, but I think that is what I want22:00
clarkbcorvus: do you have to zuul-executor stop after doing zuul-executor pause and letting the jobs drain out?22:00
clarkboh reading the code says zuul-executor graceful is pause + stopping22:01
clarkbthat doesn't appaer to be documented22:01
clarkbzuul-executor graceful has been issues on ze01.openstack.org22:05
*** mkowalski_ has quit IRC22:06
*** mkowalski has joined #opendev22:07
corvusclarkb: seems like you got your answer.  docker-compose down after graceful sgtm22:11
clarkbcorvus: yup though I realized our docker-compose file says restart always so I think it will restart the container22:11
clarkbI expect I need to pause, then wait for it to idle then down it22:12
clarkbI'll let it run through its course of the current graceful stop to confirm though22:12
corvusclarkb: right 'docker-compose down' will take it down22:12
clarkbright, but there is a race between zuul-executor idling then stopping and me running that command22:12
clarkbI don't think I can beat a cpu for that22:12
clarkbits not a huge deal, will just be a little longer to get the old executor out of the rotation22:13
corvusclarkb: i agree.  we sholud change that22:13
mkowalskiHello everyone! Haven't been contributing for quite some years, so I have a question about changes between gerrit and launchpad...22:17
mkowalskiI open a bug in launchpad, then I open a review in gerrit. Review is linked via "Closes-Bug: #id" and indeed is clickable22:17
mkowalskiI remember in the past launchpad bug would be automatically assigned to me22:17
mkowalskiHas that changed or is it happening only once in a while or did I mess up with some email linking between accounts?22:18
clarkbmkowalski: the hooks that made that happen relied on the old gerrit sql database. We've since upgraded to a much newer version of gerrit that uses notedb instead and we haven't had time to update the hooks or replace them with something else that works22:18
mkowalskiclarkb: I see; so the expected workflow would be that I assign myself in launchpad and write a comment with gerrit review id or something, right?22:20
clarkbyes, that should work just fine22:20
mkowalskiGreat, thanks for the info22:22
clarkbcorvus: I think we can use restart: on-failure in the docker-compose file22:25
clarkbas for this case I'll just let it restart, then use pause and monitor for it to idle. Then I can down it22:26
*** lbragstad_ is now known as lbragstad22:26
corvusheh, i'm having trouble keeping up with channel changes; responded in zuul :)22:27
corvus(i'm pretty deep into non-opendev stuff today; only occasionally coming up for air)22:28
clarkbthats fine, I don't think its urgent22:28
*** tkajinam has joined #opendev22:58
*** stewie925 has joined #opendev23:20
openstackgerritIan Wienand proposed opendev/system-config master: zuul-summary-status : handle SKIPPED and ERROR jobs
clarkbinfra-root I've dug into stewie925's account situation a bit and have infos23:45
clarkbthere is an old account and a new account. The email address associated with the openid for the accounts was removed from the old account. This meant on next login gerrit decided that a new account should be created. When it did this it simply updated the accountId record in the externalid for the openid. This means the old account has no openid login ability.23:46
clarkbhowever, stewie925 is still pushing code via ssh to the old account.23:46
clarkbUnfortunately do to the account inconsistency issues we can't just push a quick fix for that.23:46
clarkbinfra-root I guess I'm looking for input on how we might proceed from here.23:49
ianwclarkb:  i felt there was a theory that this couldn't happen any more since the upgrade?23:50
clarkbianw: you can't have a conflicting email after the upgrade23:50
clarkbianw: but new gerrit has the bug where it will create a new account if the email address on an existing openid isn't found on the old account and it will just move the openid for you23:50
clarkbthey are different bugs, but similarly challenging23:50
clarkbnote there is no email conflcit here, both accounts have completely distinct emails23:51
ianwhrm, ok.  i guess how could gerrit have known the login was for the same account?23:51
clarkbianw: because the openid itself didn't change23:51
clarkbthere is only one openid involved and it associated with the new account because gerrit simply changed the accoundId value in that externalid record. This is a bug imo23:52
clarkbone options is to kindly ask stewie925 to migrate to the new account and start using it for ssh. We would then retire the old account to avoid confusion. If we do that the changes associated to the old account will be somewhat orphaned if you try to associate them to stewie92523:54
clarkbanother option is to plan a downtime and an external id update that reassociates the openid to the old account and adds the openid email back to that account23:55
clarkba third option is to more quickly fix the account inconsistencies due to external id email conflicts then revisit this with stewie925 in a week or two23:56
clarkbmy struggle with all of these is that none are great options. the first one is less ideal for stewie925. The second two rely on us to do gerrit surgery which I'd much prefer to take slowly as we're prepared for it23:56
clarkbif we go with the third option I can reprioritize trying to resolve those conflicts23:57
fungiyeah, this sounds like all options are non-ideal. i lean toward considering it a workflow-type bug with the openid plugin23:57
clarkband yes, maybe we should file a bug for the openid accounts get reassociated issue23:58
stewie925thank you clarkb -  third option is good for me23:58
clarkbI don't think that has happened yet, but we do seem to understand it fairly well now23:58
clarkbstewie925: if we want to go with the third option then any reviews you do via http will end up being orphaned as belonging to the new account23:58
clarkbstewie925: I take it that is preferable to you?23:59
clarkbalso if we go with the third option, maybe we check back in at the end of next week and see if we've made progress with fixing the ~650 email conflict accounts23:59
clarkbI can reprioritize that next week and see if we can make better progress. If we do make progress then we can try and stick it out. If we don't then maybe we reconsider one of the other two options?23:59

Generated by 2.17.2 by Marius Gedminas - find it at!