Monday, 2019-09-23

fungiunderstandable00:08
fungiyou'll be missed!00:08
*** jamesmcarthur has joined #zuul01:13
*** sgw1 has quit IRC01:35
*** sgw has joined #zuul01:39
*** rfolco has joined #zuul01:40
*** rfolco has quit IRC02:08
*** wxy-xiyuan has joined #zuul02:09
*** jamesmcarthur has quit IRC02:58
*** jamesmcarthur has joined #zuul03:01
*** jamesmcarthur has joined #zuul03:01
*** bolg has joined #zuul04:00
*** jamesmcarthur has quit IRC04:52
*** jamesmcarthur has joined #zuul04:53
*** jamesmcarthur has quit IRC05:09
*** jamesmcarthur has joined #zuul05:09
*** jamesmcarthur has quit IRC05:13
lennybSpamapS, thanks. yes05:36
lennybfungi, thanks. it upgrading git review with pip and removing rpm worked.05:37
lennybfungi, also looks that I need to run #git remote rename origin gerrit   after cloning project. FYI05:38
*** yolanda has quit IRC05:42
*** jamesmcarthur has joined #zuul05:44
*** yolanda has joined #zuul05:49
*** jamesmcarthur has quit IRC05:51
*** AJaeger has quit IRC05:52
*** AJaeger has joined #zuul05:56
*** raukadah is now known as chandankumar06:01
*** jamesmcarthur has joined #zuul06:20
*** yolanda has quit IRC06:22
*** jamesmcarthur has quit IRC06:25
*** saneax has joined #zuul06:30
*** themroc has joined #zuul06:40
*** bolg has quit IRC06:55
*** avass has joined #zuul07:07
*** jamesmcarthur has joined #zuul07:13
*** jamesmcarthur has quit IRC07:19
*** tosky has joined #zuul07:20
*** hashar has joined #zuul07:32
*** pcaruana has joined #zuul07:41
*** jpena|off is now known as jpena07:42
NBorgI have some problems running jobs that requires a UI on remote windows machines with ansible. I use winrm to start cmd -> psexec (-i to the session id for my logged in session) -> script that usually works (when run in jenkins). I can see the cmd-process starting, but none of the UI:s are responsive which causes them to crash. Has anyone solved any similar problems?07:53
*** themroc has quit IRC07:54
*** toabctl has joined #zuul08:02
*** yolanda has joined #zuul08:09
*** themroc has joined #zuul08:15
*** jamesmcarthur has joined #zuul08:15
*** jamesmcarthur has quit IRC08:20
*** jangutter has joined #zuul08:31
*** bolg has joined #zuul08:42
*** yolanda has quit IRC08:50
*** bolg has quit IRC09:15
*** jamesmcarthur has joined #zuul09:16
openstackgerritJan Kubovy proposed zuul/zuul master: Unify Github/Github Enterpise read timeout  https://review.opendev.org/68336609:18
*** bolg has joined #zuul09:20
*** jamesmcarthur has quit IRC09:21
*** bhavikdbavishi has joined #zuul09:23
*** themroc has quit IRC09:57
*** gtema_ has joined #zuul10:08
*** jamesmcarthur has joined #zuul10:17
*** jamesmcarthur has quit IRC10:23
*** gtema_ has quit IRC10:29
*** bhavikdbavishi has quit IRC10:34
*** bhavikdbavishi has joined #zuul10:35
*** hashar has quit IRC10:35
*** snapiri has quit IRC10:43
*** bhavikdbavishi1 has joined #zuul10:58
*** bhavikdbavishi has quit IRC10:59
*** bhavikdbavishi1 is now known as bhavikdbavishi10:59
clarkbNBorg: tobiash also usesthe winrm connection and may have thoughts11:02
clarkbwith linux you'd be sure the display vars were set and X wasrunning.Maybe yoi can do similr woth winrm?11:08
*** snapiri has joined #zuul11:10
NBorgI can see a window popping up but it is not responsive. But avass might have a solution (win7 bug workaround). Testing it now.11:11
*** shachar has joined #zuul11:15
*** snapiri has quit IRC11:16
*** jamesmcarthur has joined #zuul11:19
*** jamesmcarthur has quit IRC11:24
clarkbNBorg: reading abit apparently you must explicitly request a non interactive powershell if that is what you want and the python winrm does not do this. Maybe try with powershell instead of cmd as a result?11:24
*** jpena is now known as jpena|lunch11:28
NBorgclarkb: Something there does not seem correct. You mean explicitly request interactive powershell, which pywinrm does not do, right?11:28
clarkbNBorg: no the opposite according to the bugtracker. https://github.com/diyan/pywinrm/issues/17711:31
clarkbof course I may misunderstand what -NonInteractive means in this context11:31
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Add more languages for build-releasenotes  https://review.opendev.org/68392311:35
NBorgclarkb: No, you are right. I thought I had read that winrm could only use psexec(and/or powershell) non-interactively, but I was either reading old docs or misinterpreting something. I can't find it now anyway.11:44
*** hashar has joined #zuul11:50
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Add more languages for build-releasenotes  https://review.opendev.org/68392312:01
*** rfolco has joined #zuul12:12
*** jamesmcarthur has joined #zuul12:20
*** jamesmcarthur has quit IRC12:25
*** rlandy has joined #zuul12:29
openstackgerritJan Kubovy proposed zuul/zuul master: Unify Github/Github Enterpise read timeout  https://review.opendev.org/68336612:45
*** jamesmcarthur has joined #zuul12:45
*** jpena|lunch is now known as jpena12:47
*** bhavikdbavishi has quit IRC12:48
openstackgerritJan Kubovy proposed zuul/zuul master: Unify Github/Github Enterpise read timeout  https://review.opendev.org/68336612:59
*** rlandy is now known as rlandy|mtg13:03
mnaserclarkb: 013:19
clarkbmnaser: 1 ?13:19
mnaserclarkb: ^ oops that was an accident13:19
corvuszuul-maint: can you add zuul-registry to your repo watch lists?  https://review.opendev.org/683426 could use a review13:28
pabelanger+213:29
openstackgerritMerged zuul/zuul-registry master: Add README, license, and Zuul config  https://review.opendev.org/68342613:33
openstackgerritJan Kubovy proposed zuul/zuul master: Report retried builds in a build set via mqtt.  https://review.opendev.org/63272713:34
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Initial implementation  https://review.opendev.org/68395813:36
tobiashzuul-maint: I'm back from vacation. Did I miss anything interesting?13:38
clarkbtobiash: corvus started adding gerrit checker api support so that upstream gerrit can try using zuul13:39
tobiashcool13:39
clarkbtobiash: 3.10.2 fixes a securitry bug (update if you hvaen't yet)13:39
tobiashwe'll update tomorrow :)13:40
tobiashcorvus: I guess I can abandon 609773?13:40
corvustobiash: welcome back!  and yeah i think so13:41
Shrewstobiash: i added a test for your https://review.opendev.org/671704 change13:41
tobiashShrews: awesome :)13:42
openstackgerritJan Kubovy proposed zuul/zuul master: Add support for smart reconfigurations  https://review.opendev.org/65211413:43
corvussome of us are at ansiblefest this week13:43
tobiashcorvus: regarding your mail about gerrit on master, did the fix work?13:44
*** yolanda has joined #zuul13:44
*** jamesmcarthur has quit IRC13:44
*** jangutter has quit IRC13:48
*** jangutter_ has joined #zuul13:48
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: Add unit tests for tox_instalL_sibling_packages  https://review.opendev.org/68396313:48
mnaserAJaeger: ^ i added that as a patch on top of yours which passes locally13:49
mnaserso i feel more comfortable merging it13:49
mnaserit has a weird quirk where it depends on sys.executable13:49
mnaseri'm not sure how we feel about that?13:49
mnaserif that test passes ill +w the one below it13:49
*** yolanda has quit IRC13:52
*** jangutter_ has quit IRC13:54
*** yolanda has joined #zuul13:56
tristanCcorvus: is the initial implementation of the zuul-registry ready?13:57
mnasero13:58
clarkbtristanC: looks like bits are unimplemented like the swift code13:58
clarkband I think it needs a requirments list for at least cherrypy (but I hvaen't done proper review yet)13:58
mnaserpy35 tests are in gate for zuul-jobs is actually py36 ..13:58
mnaseri just saw this in the logs: /usr/local/lib/python2.7/dist-packages/tox/config/__init__.py:581: UserWarning: conflicting basepython version (set 3.6, should be 3.5) for env 'py35';resolve conflict or set ignore_basepython_conflict13:58
clarkbmnaser: that implies the job ran on bionic instead of xenial13:59
mnaserok thats likely, also i think the functional tests run on xenial for zuul too, i noticed this the other day13:59
corvusclarkb, tristanC: yeah, i'd say it's not ready yet -- it still needs more framework and has unimplemented parts.  but, it runs and implements push and pull (the test script works)13:59
corvusclarkb, tristanC: that's how much i could do on my recent plane flights :)14:00
*** rlandy|mtg is now known as rlandy14:00
clarkbcorvus: does it handle authentication to dockerhub for anonymous access? (if it does tripleo might want ot consier using it/helping too)14:00
corvusclarkb: no, it doesn't implement any proxy stuff yet14:00
corvusthat was hard to do on the plane14:01
clarkbha14:01
tristanCcorvus: i'm looking forward reviewing the registry, please let me know when it's ready14:02
*** yolanda has quit IRC14:03
*** saneax has quit IRC14:06
*** jpena is now known as jpena|brb14:11
AJaegermnaser: thanks for test!14:11
*** hashar has quit IRC14:14
openstackgerritMerged zuul/zuul-jobs master: Fix tox_install_sibling_packages.py for py3  https://review.opendev.org/68380514:18
*** yolanda has joined #zuul14:19
ShrewsAnyone have the streaming link for fest contributor summit?14:24
mnaserShrews: https://bluejeans.com/748090439114:32
Shrewsthx14:32
corvusShrews: https://etherpad.openstack.org/p/ansible-summit-atlanta-2019 may be useful14:33
Shrewscorvus: already on that, thx.14:33
mnaserShrews: and #ansible-meeting :p14:33
pabelangermnaser: it looks like fedora-30 in nodepool is using python3 by default, so we should be able to try via depends-on for some project using fedora-3014:35
mnaserpabelanger: i think the unit tests help with that i think14:35
pabelangerhttps://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L25014:36
pabelangeryup, just saying we now a node only too that is python314:36
dmelladopabelanger: that's really cool14:38
dmelladodo we also have that in our zuul?14:38
pabelangerYup, since last week. We still need to migrate some jobs to it14:39
openstackgerritJan Kubovy proposed zuul/zuul master: Add --check-config option to zuul scheduler  https://review.opendev.org/54216014:40
dmelladopabelanger: ack, great!14:40
*** tosky has quit IRC14:40
*** jamesmcarthur has joined #zuul14:54
*** yolanda has quit IRC14:54
*** jpena|brb is now known as jpena15:05
*** pcaruana has quit IRC15:05
*** timburke_ is now known as timburke15:15
openstackgerritTobias Henkel proposed zuul/zuul master: Support emitting warnings via zuul_return  https://review.opendev.org/65152615:20
tobiashcorvus: do we have test races with the new gerrit code?15:22
tobiashhttps://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_09d/632727/10/check/tox-py36/09d13ab/testr_results.html.gz15:22
tobiashthis looks unrelated15:22
*** mattw4 has joined #zuul15:22
*** mattw4 has quit IRC15:26
corvustobiash: yes, i have seen that once, i haven't figured it out yet, but added some debugging to try to help15:26
corvustobiash: i think it's related to my attempt to synchronize events around the polling that happens (the gerrit checks-api is polling driven)15:27
tobiashah ok15:27
*** jamesmcarthur has quit IRC15:30
*** mattw4 has joined #zuul15:37
*** sgw has quit IRC15:37
corvustobiash: aha! i see the problem :)15:37
corvusthat test polls for both check and gate in sequence, but the sync code waits for any poll.15:39
corvuswe really want to wait for gate15:39
*** yolanda has joined #zuul15:40
openstackgerritJames E. Blair proposed zuul/zuul master: Wait for poll twice in tests  https://review.opendev.org/68398915:45
corvustobiash: ^ a crude but should be effective fix15:46
Shrewstobiash: when you get time, would love your review on https://review.opendev.org/68185715:46
Shrewsor anyone else, for that matter15:46
AJaegercorvus: we had a change that added a jobs to check, gate, promote pipeline (via template). The check and gate jobs were run, the promote one not (they all had the same files list which was not triggered, only the template was added). Is that a bug in promote where the run job if its changed is not triggered?15:46
NBorgIs there an equivalent to build with jenkins-cli in zuul? If for instance a pipeline is triggered by a tag, and I want to run specific jobs depending on the tag name. Or is the only alternative to configure one pipeline for each combination in both the config project and the tenant project?15:54
SpamapSNBorg: Can you give a more concrete example? I think the answer is that you need a unique pipeline, but maybe what you're trying to do has a better answer.15:55
SpamapSThis kind of goes to the question a few days ago about data-based jobs. It would be kind of cool if one could run a snippet of code on a trigger and produce zuul job config just for that trigger.15:56
NBorgI have a job in jenkins. When it is triggered by creating a git tag, it starts, parses the components of the tags and enqueues jobs depending on the composition of the tag. Say git tag "Software1-HardwareA". Then I want to run the Softare1-compile job with vars: "hardware: HardwareA", then run HardwareA-HIL-tests.15:58
SpamapSNBorg: yeah, that's neat. I wonder if there's a matcher that would work for you though. Like, file path, or branch based.16:00
*** mattw4 has quit IRC16:03
*** mattw4 has joined #zuul16:05
*** pcaruana has joined #zuul16:06
NBorgSpamapS: Checking for files would be impractical to trigger releases from the master branch. Perhaps it's possible to request enough nodes for any type of job-combination from nodepool and act as a zuul-executor? (And return any unused/finished nodes)16:08
NBorgMy plan was to do some regex_replace on the zuul.project['base'].checkout to figure out what jobs to run. But I wonder if that is possible.16:12
SpamapSNBorg: I think I'd rather see this as a first-class feature than try to hack around it. I'm still not sure I understand what the concrete example is though.16:12
SpamapSLike, why are you doing tag-per-arch? Why not just tag with a unique build identifier (0.1.5 or build-4593482) and run all the builds?16:13
SpamapSOne thing that may not be stated in Zuul, but is a principle I see followed and that I support, is that dynamism complicates things unnecessarily.16:15
*** mhu has joined #zuul16:18
mhucorvus, hello, is it me you're looking for?16:19
NBorgSpamapS: Expensive software and/or hardware. But, yeah... Maybe I can force people to work a bit more rationally.16:20
SpamapSNBorg: what's more expensive: developer debugging time, or the hardware?16:20
SpamapS(and no, not just developer salary, but lost productivity)16:21
SpamapSNBorg: anyway, you should be able to work that way, and I don't think it would be all that terribly hard to make a tag/regex matcher.16:22
SpamapSNBorg: Basically you should be able to say something like this in a job stanza: 'tags: - ".*Hardware1.*"' and then that job only ever runs if a trigger has a tag that matches that regex.16:23
*** mattw4 has quit IRC16:28
NBorgSpamapS: Thanks. I'll look into it.16:31
SpamapSNBorg: I'm almost done writing it ;)16:31
NBorg:D16:31
tobiashShrews: I'll have a look later this evening16:33
SpamapSMostly just working out a good set of tests16:33
*** pcaruana has quit IRC16:46
*** mattw4 has joined #zuul16:48
*** Goneri has joined #zuul16:54
*** saneax has joined #zuul16:57
*** bolg has quit IRC17:10
*** mattw4 has quit IRC17:11
*** jpena is now known as jpena|off17:16
*** jamesmcarthur has joined #zuul17:25
*** saneax has quit IRC17:27
*** jamesmcarthur_ has joined #zuul17:28
corvusNBorg, SpamapS: https://zuul-ci.org/docs/zuul/user/jobs.html#var-zuul.child_jobs may be useful17:31
corvusNBorg, SpamapS: that lets you write a 0-node job and filter/reduce the set of jobs that run17:32
*** jamesmcarthur has quit IRC17:32
*** jamesmcarthur_ has quit IRC17:34
*** jamesmcarthur has joined #zuul17:34
corvusmhu: if you have a minute, we've seen tests.unit.test_web.TestTenantScopedWebApi.test_dequeue fail a few times -- i think it's a racy test... do you think it could be reworked to avoid the sleep?17:36
clarkbcorvus: SpamapS tobiash Shrews https://review.opendev.org/#/c/680742/1 might be a good one to get in for tomorrow (zuul booth at ansiblefest)17:37
clarkbmaybe I can get reviews on that today?17:37
Shrews+217:38
*** jamesmcarthur has quit IRC17:39
clarkbtyty17:39
*** jamesmcarthur has joined #zuul17:39
SpamapScorvus: wait, I can set values in child_jobs?17:40
corvus+317:40
tobiashSpamapS: yes, you can filter them17:40
corvusSpamapS: yes, sorry i should have linked to https://zuul-ci.org/docs/zuul/user/jobs.html#skipping-child-jobs  (cc NBorg)17:40
SpamapSTIL!17:41
pabelanger++17:42
SpamapSNBorg: there you go. Problem solved.17:42
NBorgcorvus: Thanks17:43
openstackgerritMerged zuul/zuul-website master: Add faq entry about kubernetes and docker images  https://review.opendev.org/68074217:45
*** mattw4 has joined #zuul17:46
tobiashShrews: now I see why we leak many images and nodepool forgets them17:46
tobiashthanks for that fix17:46
Shrewstobiash: yeah, that was rather nasty17:47
tobiashlgtm17:47
clarkbis it passing tests now?17:49
Shrewstobiash: that guarantees we'll retry the delete, but unless the issue causing the delete to fail is resolved, the image will still remain17:49
clarkbI was meaning to revie wit once it did but then travel happened17:49
Shrewstobiash: that issue for us is volumes remaining in-use, even though they really aren't. that's a cinder issue17:49
tobiashShrews: I remember that once upon a time (maybe back in v2 times) it has been like this17:50
tobiashShrews: we have the same issue17:50
tobiashsome volumes stay in ceph even if the instance is gone. Those then block deleting the image17:50
Shrewsyup17:50
tobiashis there a known fix for this issue btw?17:51
Shrewstobiash: for cinder? not that i'm aware of17:51
tobiashk, I was hoping there would be one17:51
clarkbwe also have cinder double accounting our volume usage :/17:52
Shrewsso many bugs17:53
tobiashShrews, clarkb: maybe one of you wants to +3 https://review.opendev.org/683989 ? It fixes a nasty test race.17:54
fungiyeah, a couple of suspected problems there... one is that if nova asks cinder to delete a backing volume and that request fails, nova ignores/reports the error and proceeds to delete the server instance, leaving the volume behind with cinder indefinitely convinced it's still used by a server attachment long after that instance is gone17:54
clarkbtobiash: looking17:54
fungibut also according to mnaser cinder never double-checks its quota accounting, it just modifies the existing value, so if it ever gets out of sync it stays that way until manual intervention is taken to recalculate it and reset it to a correct value17:55
tobiashwe have two issues I observed, one is volumes left in state attached to <non existing instance id> and the other is when boot from image in ceph the backing root disk (not a real cinder volume) sometimes doesn't get deleted with the instance17:56
Shrewsfungi: one would think those issues would be significant enough to address at some point, but i don't follow that project, so maybe they're unaddressable  *shrug*17:56
clarkbShrews: I brought the quot athing up with cinder but because I don't hvae version info or any cloud side data really they can't relaly take action on it17:57
clarkbmnaser: ^ maybe in all of your free time (I'm joking if that wasn't clear) we can collect some bsaic info and file a bug with cinder?17:58
fungismcginnis suggested the leaked attachments problem probably needs to be fixed on the caller (nova) side17:58
*** jamesmcarthur has quit IRC17:59
mnaseryeah leaked attachs are def a nova issue, but double accounted quota is a cinder one17:59
fungie.g., by refusing to delete a server instance if the call to delete the backing volume for it fails17:59
mnaseri might have sometime later this week17:59
*** igordc has joined #zuul18:06
openstackgerritMerged zuul/zuul master: Fix weak dependencies to work with child_jobs  https://review.opendev.org/67793618:09
Shrewstobiash: i just realized that call to storeImageUpload() needs a lock around it. patch incoming18:13
tobiashoops18:14
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete  https://review.opendev.org/68185718:16
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Do not overwrite image upload ZK data on delete  https://review.opendev.org/68185718:18
*** Goneri has quit IRC18:18
Shrewstobiash: that should cover it ^^18:19
*** sgw has joined #zuul18:19
Shrewstobiash: oh, hrm, i think that's still wrong. that doesn't lock the actual record18:21
Shrewstobiash: corvus: sorry, i think i just uncovered a potential issue18:23
Shrewsor i've misremembered something18:24
Shrewswe have locks for uploading images, but i don't see any for dealing with locking a specific upload record18:26
Shrewssimilar to build locks, which has one for each18:26
Shrewswow18:27
Shrewsok, yep. that needs to be fixed, and i'm very surprised i missed that for so long18:35
Shrewsi don't think it would actually cause any issues, but maybe the occasional weird error in a competing builder18:35
Shrewstobiash: can you unapproved that? it's still in gate18:36
tobiashdone18:37
*** hashar has joined #zuul18:37
Shrewstobiash: thx18:38
*** igordc has quit IRC18:51
Shrewsi've got a PT appt now so i'll have to deal with that mistake later  :(18:52
*** mgoddard has quit IRC19:17
*** mgoddard has joined #zuul19:20
*** michael-beaver has joined #zuul19:20
mhucorvus, about the dequeue test; sure, I'll have a look, but I think it was heavily inspired by the rpc variant of that test. I'll check if there's a sleep() in there19:31
*** pcaruana has joined #zuul19:43
*** sgw has quit IRC19:46
*** sgw has joined #zuul19:46
*** sgw has quit IRC19:51
*** sgw has joined #zuul20:05
*** pcaruana has quit IRC20:12
tobiashcorvus, clarkb: did you also observe that sometimes an executor stops taking jobs even if it didn't unregister?20:29
tobiashduring the last three weeks this happened to 4 of our 20 executors20:30
tobiashthis looks rare and random20:31
tobiashthe next time I observe this I'll take a thread dump for further analysis20:32
clarkbno havent noticed that20:33
*** saneax has joined #zuul20:33
clarkbbut we restart the executors often recently to fox bugs20:33
tobiashk, thx20:34
openstackgerritMerged zuul/zuul master: Wait for poll twice in tests  https://review.opendev.org/68398921:01
*** mattw4 has quit IRC21:05
*** mattw4 has joined #zuul21:05
*** saneax has quit IRC21:09
Shrewsmnaser: cool to see that you’re offering a managed zuul solution.21:09
* Shrews wishes his shoulder wasn’t holding him back from attending the fest festivities21:10
mnaserShrews: thanks :) I hope you feel better!21:10
mnaserI'm excited too!21:11
SpamapSmnaser: CONGRATULATIONS on your announcement!21:17
*** hashar has quit IRC21:25
*** igordc has joined #zuul21:27
mnaserSpamapS: thanks, I'm pretty excited and we'll see how it all pans out tomorrow some more21:27
mnaserI think it's a win for everyone if this happens21:27
*** mattw4 has quit IRC21:29
*** mattw4 has joined #zuul21:30
*** avass has quit IRC21:46
*** mattw4 has quit IRC22:22
openstackgerritKerby proposed zuul/nodepool master: AWS driver: add ability to determine AMI id using filters  https://review.opendev.org/68320522:26
*** rfolco has quit IRC22:29
*** rfolco has joined #zuul22:37
*** NBorg has quit IRC22:47
*** sgw has quit IRC23:29
*** mattw4 has joined #zuul23:29
*** mattw4 has quit IRC23:43
*** sgw has joined #zuul23:45
*** michael-beaver has quit IRC23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!