| *** greghaynes has quit IRC | 00:33 | |
| *** greghaynes has joined #zuul | 00:42 | |
| *** harlowja has quit IRC | 02:57 | |
| *** pleia2 has quit IRC | 02:58 | |
| *** dmsimard has quit IRC | 02:58 | |
| *** pleia2 has joined #zuul | 03:00 | |
| *** dmsimard has joined #zuul | 03:00 | |
| *** Zara has quit IRC | 03:13 | |
| *** Zara has joined #zuul | 03:13 | |
| *** rattboi has quit IRC | 03:14 | |
| *** rattboi has joined #zuul | 03:17 | |
| *** tflink has quit IRC | 03:26 | |
| *** elli has joined #zuul | 03:59 | |
| *** elli has quit IRC | 04:19 | |
| *** isla has joined #zuul | 04:27 | |
| *** isla has quit IRC | 04:30 | |
| *** tflink has joined #zuul | 04:35 | |
| *** tflink has quit IRC | 04:37 | |
| *** tflink has joined #zuul | 04:39 | |
| *** abregman has joined #zuul | 06:29 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/zuul: Merge branch 'master' into workingv3 https://review.openstack.org/389470 | 06:37 |
|---|---|---|
| openstackgerrit | Merged openstack-infra/zuul: Refactor skip-if tests to use irrelevant-files https://review.openstack.org/399415 | 07:24 |
| *** saneax-_-|AFK is now known as saneax | 08:07 | |
| *** abregman has quit IRC | 08:29 | |
| *** abregman has joined #zuul | 08:35 | |
| *** abregman has quit IRC | 08:39 | |
| *** abregman has joined #zuul | 08:43 | |
| *** hashar has joined #zuul | 08:48 | |
| *** olaph has quit IRC | 08:50 | |
| *** olaph has joined #zuul | 08:50 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack-infra/zuul: Replace assertEquals with assertEqual https://review.openstack.org/400119 | 08:53 |
| *** hashar has quit IRC | 08:55 | |
| *** hashar has joined #zuul | 08:56 | |
| *** hashar has quit IRC | 09:20 | |
| *** willthames has quit IRC | 09:29 | |
| *** yolanda has joined #zuul | 10:32 | |
| *** pabelanger has quit IRC | 12:15 | |
| *** pabelanger has joined #zuul | 12:15 | |
| *** hashar has joined #zuul | 12:17 | |
| Shrews | pabelanger: hrm, odd | 13:40 |
| Shrews | pabelanger: i'm still at my first cup of coffee, but not sure that rotation test is valid. You're setting state time but not state so I'm not real sure of your intent there | 13:45 |
| Shrews | pabelanger: also, those tests are making assumptions about build IDs that I don't believe is safe | 13:47 |
| Shrews | pabelanger: ZK decides the sequence numbers, and I've seen them start at 0 and I've seen them start at 1. I don't know how it chooses, but we can't know for sure which one it is. | 13:48 |
| Shrews | pabelanger: is the addition of waitForBuild() already merged? | 13:48 |
| Shrews | ah, no. in the previous review | 13:48 |
| Shrews | will comment there | 13:48 |
| Shrews | oh, i see what you're trying to do in that test now. caffeine is kicking in | 13:55 |
| Shrews | pabelanger: so, I've tested the rotation myself so logic should be sound (unless there is a race condition I didn't encounter). let's eliminate the build ID issue I mentioned in the review first. | 13:57 |
| Shrews | to make sure that's not tripping the test up | 13:58 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Print all fields of ZK objects https://review.openstack.org/400270 | 14:23 |
| Shrews | pabelanger: oh, the cleanup code has changed somewhere along the lines. I don't think it's sorting properly | 14:39 |
| Shrews | i think this broke it: https://review.openstack.org/399307 | 14:42 |
| Shrews | jeblair: Why did you remove the call to getMostRecentBuilds() on line 330 here? https://review.openstack.org/#/c/399307/2/nodepool/builder.py | 14:44 |
| Shrews | jeblair: i think we need to add that back. the builds need to be sorted by state_time. that code isn't doing that, but the get* call will | 14:44 |
| Shrews | but i want to make sure i'm not missing something before i add it back in | 14:45 |
| Shrews | ah, the "consistent view" thing | 14:48 |
| *** abregman is now known as abregman|mtg | 14:49 | |
| pabelanger | Shrews: the goal for the rotation test is to confirm we have 2 builds for an image at any given time. Unless that hasn't changes, that is how many we had for nodepoolv2. using the state_time was my way of expiring an build to force a rebuilt. | 14:51 |
| pabelanger | agreed about assumptions about build name / id, will have to change that | 14:51 |
| pabelanger | Shrews: Ya, I think sort is the issue too. Was just poking at it a little while waiting for the train | 14:52 |
| Shrews | pabelanger: also, storeBuild() should really not be used w/o imageBuildLock() ... though it's somewhat safe to set 'delete' state | 14:53 |
| pabelanger | k | 14:54 |
| pabelanger | I can clean that up too | 14:54 |
| openstackgerrit | Tuan Luong-Anh proposed openstack-infra/zuul: Replace assertEquals with assertEqual https://review.openstack.org/400119 | 14:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add test to validate rebuilds are scheduled https://review.openstack.org/399974 | 15:01 |
| pabelanger | Shrews: okay, update to use imageBuildLock^ Still working on build-id issue | 15:01 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Sort existing builds properly in cleanup thread https://review.openstack.org/400292 | 15:10 |
| Shrews | pabelanger: jeblair: ^^^ I think that corrects the sorting issue | 15:10 |
| Shrews | but your test still doesn't work for some reason | 15:11 |
| *** saneax is now known as saneax-_-|AFK | 15:12 | |
| *** dmsimard is now known as dmsimard|away | 15:24 | |
| Shrews | pabelanger: oh! i know why it still doesn't work | 15:25 |
| Shrews | pabelanger: so, in addition to keeping the 2 most recent builds, we ALSO have a check to make sure that we keep the two most recent UPLOADS. So both 1 and 2 have uploads, so 1 will never get deleted | 15:25 |
| * Shrews forgot about that | 15:26 | |
| Shrews | pabelanger: that's what that upload recency table is all about | 15:26 |
| *** markmcd has joined #zuul | 15:28 | |
| Shrews | oh, but 3 should get uploaded. hrm | 15:30 |
| Shrews | grr | 15:30 |
| pabelanger | Shrews: when I last looked 3 does get uploaded, but then deleted right away | 15:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Enable DIB_CHECKSUM in devstack tests https://review.openstack.org/400309 | 15:49 |
| Shrews | for some reason the cleanup code is only ever seeing the upload for 1 | 15:50 |
| phschwartz | jeblair: when you have some time today I want to talk more about the pipeline dependency tree. I have tried a few different ways of doing it and none feel 100% right | 15:56 |
| Shrews | phschwartz: lol. indentation bug... patch incoming | 16:06 |
| Shrews | err, pabelanger ^^^ | 16:06 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Fix upload recency table bug https://review.openstack.org/400320 | 16:08 |
| Shrews | pabelanger: ^^^ | 16:08 |
| Shrews | pabelanger: rebase your review on top of that stack, it should pass now | 16:10 |
| pabelanger | ack | 16:11 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add test_image_delete test https://review.openstack.org/399757 | 16:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Remove image-update based tests https://review.openstack.org/399764 | 16:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_image_delete_invalid https://review.openstack.org/399739 | 16:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Remove image-upload command and tests https://review.openstack.org/399766 | 16:17 |
| markmcd | i noticed in http://inaugust.com/talks/zuulv3.html#/4 a mention of a design for container support, has someone got a link to this design ? thks | 16:26 |
| pabelanger | I don't think that spec is written yet | 16:28 |
| markmcd | pabelanger: thanks, i was thinking that was the case, just wanted to confirm it | 16:30 |
| pabelanger | markmcd: keep an eye out for a nodepool spec, that's where we'd be adding support for it | 16:30 |
| pabelanger | We talked about it recently at ansiblefest last month, but future work right now | 16:31 |
| markmcd | pabelanger: perfect, its being developed as part of the zuulv3 effort though ? | 16:32 |
| pabelanger | markmcd: I believe it is on the table, but mordred knows more about the details. It is something we want for sure | 16:33 |
| clarkb | fwiw using containers today should just work if using nova + container flavors, or plugging something like jenkins mesos into jenkins. The gotcha there is that doesnt get you containers running speculative git merge based images whoch I think may be the thing mordred is interested in | 16:37 |
| markmcd | pabelanger: ack | 16:39 |
| pabelanger | clarkb: speaking of containers, I'm going to try running disk-image-create within lxc later today. See how that works, that might be a dirty way to support more then 1 builder on a host | 16:40 |
| markmcd | clarkb: im interested in using speculative merge and all the good stuff zuul offers with containers :) | 16:40 |
| pabelanger | otherwise, I found an old blueprint for DIB to support parallel builds | 16:41 |
| clarkb | pabelanger: you have to open a bunch of perms like mounting iirc | 16:41 |
| jeblair | Shrews: yeah, i sorted, but forgot to sort by time, sorry | 16:42 |
| pabelanger | clarkb: Ya, that's what I've been reading | 16:42 |
| jeblair | phschwartz: yes, let's try to catch up on the dependency thing later today | 16:42 |
| jeblair | pabelanger: why do we want more than one builder on a host? | 16:42 |
| pabelanger | jeblair: clarkb was noting we might get better testing times for devstack nodepool job, if we did parallel image builds. Was looking at adding more labels to the devstack job to help catch DIB regressions | 16:44 |
| jeblair | pabelanger: i thought we only made one image in there? | 16:45 |
| clarkb | specufically I didnt want adding redundant builds to the integration job because they serialize | 16:45 |
| clarkb | jeblair: pabelanger is trying to add another | 16:45 |
| pabelanger | jeblair: yes, trusty, but wanted to add xenial too | 16:45 |
| jeblair | ah, that makes more sense if i read pabelanger's sentences in reverse order. :) | 16:45 |
| pabelanger | or all flavors for diskimage-builder devstack job. | 16:46 |
| pabelanger | https://review.openstack.org/#/c/399865/ | 16:46 |
| jeblair | SpamapS: over the weekend, i was thinking there's a subtlety we may have overlooked which might make for a simple solution: we really only intend the job stanzas in the project-pipelines to *modify* existing job definitions. therefore, if we say that a change has to match one of the standalone job definitions as well as the one in the project-pipeline, then we may have the behavior we want. | 16:47 |
| jeblair | pabelanger, clarkb: re https://review.openstack.org/399642 did we also decide to remove 'min-ready: -1' support? | 16:59 |
| *** abregman|mtg has quit IRC | 16:59 | |
| clarkb | jeblair: yes I am fine with that | 16:59 |
| pabelanger | same | 16:59 |
| pabelanger | I can update the testing today | 16:59 |
| jeblair | okay will -1 that for now then | 16:59 |
| SpamapS | jeblair: That sounds like a good constraint. | 17:13 |
| mordred | clarkb, pabelanger: yes - the 'hard' problem I'm interested in solving is supporting folks using containers like app containers in a kubernetes environment where the containers are single process. using containers like VMs where you can ssh in to them is, I agree, easy | 17:25 |
| mordred | and I think that plugging a mesos provider, for instance, into nodepool in such a way that it returned a container that ansible could ssh in to should be 'easy' and should 'just work' | 17:25 |
| mordred | but to be able to support testing folks' multi-container orchestrated applications in a way that feels robust is, I think, a thing that we're probably going to have to play with for a while | 17:27 |
| *** hashar has quit IRC | 17:43 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add test for provider uploads enable / disable https://review.openstack.org/399681 | 17:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Re-enable TestWebApp tests https://review.openstack.org/399716 | 17:59 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Remove waitForBuiltImages() / JobTracker() from nodepool.py https://review.openstack.org/399727 | 17:59 |
| pabelanger | just a rebase | 18:00 |
| pabelanger | while I work on 399642 | 18:00 |
| *** harlowja has joined #zuul | 18:14 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add tests for enable / disabled labels https://review.openstack.org/399642 | 18:19 |
| pabelanger | jeblair: clarkb: let me know what you think^, setting providers: [] for a label, means it will be disabled | 18:20 |
| jeblair | pabelanger: how's that different than 0? | 18:25 |
| jeblair | min-ready=0 | 18:25 |
| pabelanger | we upload images still with min-ready 0 | 18:26 |
| pabelanger | IIRC: we set min-ready: 0, to prime the cloud with images | 18:27 |
| pabelanger | then toggle it when we want to enable the cloud | 18:27 |
| jeblair | right | 18:27 |
| jeblair | pabelanger: if you don't want an image in a provider, don't put an image stanza for that image on that provider | 18:28 |
| jeblair | if you don't want to create nodes in a provider with a label that uses that image, don't add that provider to the list of providers in the labels | 18:28 |
| pabelanger | sure, we can do that | 18:28 |
| jeblair | pabelanger: i think we already do that, that's what my changes last week do | 18:29 |
| pabelanger | jeblair: okay, do you mind commenting on the review with your objections. No issue for me to rework it | 18:30 |
| jeblair | pabelanger: let me write up an etherpad and make sure we're on the same page | 18:31 |
| pabelanger | WFM | 18:31 |
| jeblair | pabelanger: https://etherpad.openstack.org/p/um87VzudKq | 18:37 |
| jeblair | pabelanger: i think that lays out a coherent approach to this | 18:38 |
| jeblair | pabelanger: however, that first section, about diskimages, isn't actually how it behaves now -- i think we need a change to implement that. i think the second and third sections are the current behavior. | 18:38 |
| jeblair | pabelanger: (but as i was writing the other sections, i thought that changing that first section to match makes sense) | 18:39 |
| jeblair | clarkb: ^ ? | 18:39 |
| pabelanger | readying now, trying to process | 18:41 |
| jeblair | i will continue to examplify | 18:42 |
| pabelanger | jeblair: okay, I see the pattern now. Ya, that appears to be much simpler then today | 18:45 |
| jeblair | i think what it doesn't do is allow us to stop updating an image, but still keep it around. | 18:45 |
| pabelanger | ya | 18:46 |
| jeblair | i think maybe whatever need we have for that could be solved by the 'pause' think we mentioned last week | 18:47 |
| pabelanger | indeed, we could have used that over the weekend, to stop uploading broken ubuntu-xenial images | 18:47 |
| pabelanger | but not make a massive nodepool.yaml update | 18:48 |
| jeblair | for that, maybe we can add an attribute to both the diskimage, as well as the provider-image, that says "don't update this diskimage" and "don't upload new versions of this image to this provider" respectively | 18:49 |
| jeblair | i don't think the second would be used much. i think our main need is for the first. | 18:49 |
| jeblair | pabelanger: there, updated etherpad to mention pause | 18:51 |
| jeblair | clarkb, Shrews: does https://etherpad.openstack.org/p/um87VzudKq look good? | 18:52 |
| *** jamielennox is now known as jamielennox|away | 18:52 | |
| pabelanger | jeblair: yes, I think that works. Thanks | 18:53 |
| Shrews | looking | 18:56 |
| Shrews | 'pause' attribute? | 18:57 |
| jeblair | Shrews: it's a new thing -- we realized friday as we were cranking out broken xenial builds, we need a way to tell nodepool to chill out for a while | 18:57 |
| jeblair | Shrews: right now, we can just delete an image and we have a day to fix things. but new nodepool builder would immediately replace that. | 18:58 |
| Shrews | ooh, then let's rename the attribute 'chill' | 18:58 |
| pabelanger | haha | 18:58 |
| Shrews | i guess the etherpad makes sense then | 18:59 |
| *** jamielennox|away is now known as jamielennox | 19:00 | |
| pabelanger | jeblair: that means, min-ready for a label would be 1 or greater, right? | 19:04 |
| jeblair | pabelanger: i think min-ready=0 still means "don't use this label" | 19:05 |
| jeblair | grr | 19:06 |
| jeblair | pabelanger: yes, that's right -- i just realized i wrote max-servers in the etherpad though | 19:06 |
| jeblair | (etherpad corrected) | 19:06 |
| pabelanger | k, we can update voluptuous for min-ready to ensure it is 1 based | 19:08 |
| jeblair | pabelanger: no min-ready=0 is still a thing | 19:08 |
| pabelanger | ah | 19:09 |
| jeblair | just not min-ready=-1 | 19:09 |
| pabelanger | Doh, right | 19:09 |
| Shrews | geez, my commit msg on 400292 is just all sorts of messed up. i wonder if my brain is broken | 19:19 |
| mordred | Shrews: I've wondered that for a long time | 19:19 |
| Shrews | mordred: now we have documented proof | 19:20 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: Add test to validate when a node build is disabled https://review.openstack.org/399642 | 19:25 |
| pabelanger | jeblair: okay, I believe that is the test to validate min-ready: 0, make sure a node is not launched | 19:25 |
| *** rcarrillocruz has joined #zuul | 20:21 | |
| pabelanger | jeblair: clarkb: do you mind helping land https://review.openstack.org/#/c/399770/ to fix a race | 21:13 |
| morgan_ | o/ | 21:17 |
| morgan_ | sorry. will be missing g the IRC meeting today | 21:17 |
| morgan_ | going to be on a plane. | 21:17 |
| morgan_ | I'll look for the notes. | 21:17 |
| morgan_ | cheers and have a safe holidays (if you're in the us ) | 21:18 |
| clarkb | me too, taking advantage of post dentist babysitting for afternonn without kids | 21:18 |
| rcarrillocruz | hah | 21:18 |
| pabelanger | about to find the hotel bar, and prep for the meeting :) | 21:20 |
| jeblair | pabelanger: +3 | 21:24 |
| jhesketh | Morning | 21:24 |
| jeblair | pabelanger: 642 looks good in spirit -- some minor nits inline, one of which should make it pass. | 21:28 |
| jeblair | phschwartz: maybe we can talk about the job dependency stuff after the meeting? | 21:28 |
| rcarrillocruz | pabelanger: mind looking at my devstack-gate ansible roles changes when you get a chance? topic zuulv3 as usual | 21:29 |
| phschwartz | Jeblair: I will message you if I am able to after the meeting. If not I will ping you tomorrow morning. | 21:30 |
| jeblair | phschwartz: sounds good. | 21:30 |
| phschwartz | Yeah. The 4 month old is a little cranky and the wife is very sick so it is interesting. | 21:31 |
| jeblair | phschwartz: understood. take care. | 21:31 |
| openstackgerrit | Merged openstack-infra/nodepool: Better protect for race condition in waitForBuildDeletion() https://review.openstack.org/399770 | 21:34 |
| openstackgerrit | Merged openstack-infra/nodepool: Re-enable test: test_image_delete_invalid https://review.openstack.org/399739 | 21:38 |
| jeblair | pabelanger, Shrews: i'll start working on the change needed to make the diskimage delete happen as described in https://etherpad.openstack.org/p/um87VzudKq | 21:38 |
| openstackgerrit | Merged openstack-infra/nodepool: Add test_image_delete test https://review.openstack.org/399757 | 21:39 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove image-update based tests https://review.openstack.org/399764 | 21:39 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove image-upload command and tests https://review.openstack.org/399766 | 21:39 |
| Shrews | jeblair: ack | 21:41 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Delete builds when diskimage removed from config https://review.openstack.org/400421 | 21:58 |
| jeblair | it's about time for our meeting in #openstack-meeting-alt | 21:59 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable tests: test_alien_* https://review.openstack.org/400422 | 22:01 |
| *** willthames has joined #zuul | 22:13 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_alien_list_fail https://review.openstack.org/400422 | 22:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Re-enable test: test_alien_list_fail https://review.openstack.org/400422 | 22:32 |
| Shrews | jeblair: hrm, think maybe we should leave the alien* tests disabled until the next nodepool phase? | 22:56 |
| Shrews | that one above will pass, but will break again once we start storing node info in ZK | 22:57 |
| jeblair | Shrews: let's re-enable it, then re-disable it -- mostly because we will probably want to maintain the master branch, and having all tests running will help. | 22:57 |
| Shrews | ok. need to fix up the alien-image-list command. it's still pulling from the db, so it will pass, but it's a false positive | 22:59 |
| jeblair | ++ | 22:59 |
| openstackgerrit | Merged openstack-infra/nodepool: Print all fields of ZK objects https://review.openstack.org/400270 | 23:13 |
| openstackgerrit | Merged openstack-infra/nodepool: Sort existing builds properly in cleanup thread https://review.openstack.org/400292 | 23:14 |
| openstackgerrit | Merged openstack-infra/nodepool: Fix upload recency table bug https://review.openstack.org/400320 | 23:14 |
| *** openstack has joined #zuul | 23:46 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!