| *** dims_ has quit IRC | 00:07 | |
| *** doug-fish has quit IRC | 00:26 | |
| *** LouisF has quit IRC | 00:49 | |
| openstackgerrit | Matt Riedemann proposed openstack/releases: Release python-novaclient 3.2.0 https://review.openstack.org/264495 | 00:52 |
|---|---|---|
| openstackgerrit | Matt Riedemann proposed openstack/releases: Release python-novaclient 3.2.0 https://review.openstack.org/264495 | 00:53 |
| mriedem | jroll: ^ | 00:53 |
| jroll | thanks :) | 00:54 |
| *** dims has joined #openstack-release | 00:56 | |
| *** mriedem has quit IRC | 02:13 | |
| *** dims has quit IRC | 02:58 | |
| *** bdemers_ has joined #openstack-release | 05:29 | |
| *** bdemers has quit IRC | 05:29 | |
| *** bdemers has joined #openstack-release | 05:39 | |
| *** bdemers_ has quit IRC | 05:40 | |
| *** bdemers has quit IRC | 05:44 | |
| *** bdemers has joined #openstack-release | 05:45 | |
| *** bdemers has quit IRC | 05:48 | |
| *** bdemers has joined #openstack-release | 05:49 | |
| *** bdemers has quit IRC | 05:54 | |
| *** bdemers has joined #openstack-release | 05:55 | |
| *** bdemers has quit IRC | 06:26 | |
| *** bdemers has joined #openstack-release | 06:27 | |
| *** bdemers has quit IRC | 06:30 | |
| *** bdemers has joined #openstack-release | 06:30 | |
| *** bdemers has quit IRC | 06:35 | |
| *** bdemers has joined #openstack-release | 06:36 | |
| *** bdemers has quit IRC | 07:02 | |
| *** bdemers has joined #openstack-release | 07:03 | |
| *** ifat_afek has joined #openstack-release | 08:00 | |
| *** bdemers has quit IRC | 08:39 | |
| *** bdemers has joined #openstack-release | 08:39 | |
| *** dtantsur|afk is now known as dtantsur | 10:03 | |
| *** bdemers has quit IRC | 10:53 | |
| *** bdemers has joined #openstack-release | 10:55 | |
| *** dims has joined #openstack-release | 11:10 | |
| *** ifat_afek has quit IRC | 11:24 | |
| dims | ttx : i fixed up https://review.openstack.org/#/c/220268/ (mimic), let's get it in please | 11:30 |
| *** ifat_afek has joined #openstack-release | 11:37 | |
| *** bdemers has quit IRC | 11:38 | |
| *** bdemers has joined #openstack-release | 11:38 | |
| *** bdemers has quit IRC | 11:49 | |
| *** bdemers has joined #openstack-release | 11:49 | |
| *** devananda has quit IRC | 11:54 | |
| *** devananda has joined #openstack-release | 11:55 | |
| *** bdemers has quit IRC | 12:00 | |
| *** bdemers has joined #openstack-release | 12:00 | |
| *** gordc has joined #openstack-release | 12:31 | |
| *** bdemers_ has joined #openstack-release | 12:31 | |
| *** bdemers has quit IRC | 12:32 | |
| ttx | dims: done | 12:33 |
| dims | thanks ttx | 12:38 |
| *** doug-fish has joined #openstack-release | 12:38 | |
| *** bdemers has joined #openstack-release | 12:47 | |
| *** bdemers_ has quit IRC | 12:48 | |
| ttx | dims: while you're at it, you may want to have a look at https://review.openstack.org/264754 | 12:54 |
| dims | ttx : whoa! how did you generate it? :) | 12:55 |
| *** doug-fish has quit IRC | 13:05 | |
| *** bdemers has quit IRC | 13:09 | |
| *** ifat_afek has quit IRC | 13:10 | |
| ttx | dims: PyPI data extraction using yolk, then a few manual checks where non-obvious | 13:22 |
| ttx | I'd advise that we require the data to be present from now on :) | 13:23 |
| dims | ttx : yep +1 on that | 13:23 |
| *** doug-fish has joined #openstack-release | 13:24 | |
| *** ifat_afek has joined #openstack-release | 13:44 | |
| *** doug-fis_ has joined #openstack-release | 13:53 | |
| *** doug-fish has quit IRC | 13:55 | |
| *** mriedem has joined #openstack-release | 14:17 | |
| *** bdemers has joined #openstack-release | 14:35 | |
| *** bdemers_ has joined #openstack-release | 14:36 | |
| jroll | ttx: dims: you removed more than necessary here https://review.openstack.org/#/c/220268/12 | 14:40 |
| jroll | all of the u-c additions should have been included | 14:40 |
| dims | jroll : next rev of the bot will add those | 14:40 |
| dims | jroll : see dhellmann 's comments - "@Jay - You should not regenerate the upper-constraints file just to add a new dependency. Simply add the line for your new thing. The bot will regenerate the file to pull in other changes as appropriate, under a separate patch, where it will be clear that the changes are not related to adding mimic." | 14:41 |
| jroll | dims: oh neat, good point | 14:42 |
| dims | jroll : the README.rst says otherwise and we should fix it | 14:42 |
| dims | Jay followed the README.rst properly :) | 14:42 |
| jroll | right | 14:43 |
| *** ifat_afek_ has joined #openstack-release | 14:43 | |
| *** ifat_afek has quit IRC | 14:44 | |
| *** tristanC has joined #openstack-release | 14:45 | |
| tristanC | mriedem: Greeting sir, we are about to push security fixes to kilo and liberty. Just to make sure, the Nova stable/liberty point release number would be 12.0.1 right ? | 15:00 |
| mriedem | tristanC: we have a regression patch proposed to stable/liberty in nova that we have to get in before 12.0.1 can happen | 15:02 |
| mriedem | https://review.openstack.org/#/c/264793/ | 15:03 |
| mriedem | i should have that done today | 15:03 |
| dhellmann | jroll , dims: I'll workon an update to the readme this week | 15:03 |
| mriedem | lifeless: dhellmann: sdague: dims: dansmith: btw, i just laid out the details on that regression and why https://review.openstack.org/#/c/226157/ doesn't prevent it like caps would | 15:03 |
| mriedem | details are in that spec | 15:03 |
| dhellmann | mriedem : it sounds like that would have been fixed if the unit test jobs used the constraints? | 15:04 |
| mriedem | dhellmann: nope | 15:04 |
| tristanC | mriedem: alright, so can 12.0.1 also include https://review.openstack.org/264815 (and the 2 depends-on patch) ? | 15:04 |
| mriedem | dhellmann: details are in my comment, i cover that | 15:04 |
| dhellmann | ok, still reading | 15:04 |
| mriedem | tristanC: probably - i'm in a meeting for the next hour | 15:05 |
| mriedem | and dealing with this regression first | 15:05 |
| dhellmann | mriedem : ok, *proper* constraints? | 15:05 |
| dhellmann | mriedem: we all agree we have bad constraints settings in liberty right now and we need to figure out how to fix that. lifeless' plans assume we've fixed it. | 15:05 |
| tristanC | mriedem: ok thanks! | 15:05 |
| mriedem | dhellmann: i also commented on that | 15:06 |
| mriedem | dhellmann: just by pure luck oslo.serialization in upper-constraints at liberty GA was 1.9.0 | 15:06 |
| mriedem | and the new method was in 1.10.0 | 15:06 |
| mriedem | so yes a release-constraints file being used on that nova backport would have failed | 15:06 |
| mriedem | however, let's assume the method being used was in 1.8.0 | 15:06 |
| mriedem | but you were using oslo.serialization 1.4.0 b/c that's the min in g-r | 15:07 |
| mriedem | you'd fail | 15:07 |
| mriedem | plus we've said that release-constraints will be updated by humans as needed | 15:07 |
| dhellmann | right, so we need a test job to prove that the minimum version range actually works right. I can't remember if that's in the spec or not, but it's definitely something we've talked about adding | 15:07 |
| dhellmann | that's right, new releases of stable libs will require a manual update of release-constraints | 15:08 |
| mriedem | but if we have no lower bounds testing, things are still going to break | 15:08 |
| mriedem | if we would have simply capped oslo.serialization<1.5 at liberty GA in g-r, this wouldn't have happened | 15:09 |
| mriedem | anyway, meeting for the next hour, i should pay attention | 15:10 |
| dhellmann | mriedem : if we have lower bounds tests, upper bounds tests, and release-constraints tests, we'll be *better* but not perfect, and I think that's an improvement over what we have now | 15:10 |
| *** mriedem is now known as mriedem_meeting | 15:10 | |
| *** doug-fis_ is now known as doug-fish | 15:11 | |
| mriedem_meeting | dhellmann: i think that's a ton of complexity for something that (1) sane caps and (2) following semver when releasing clients and libs would resolve in most cases | 15:11 |
| sdague | mriedem_meeting: I thought we were talking about release-constraints on unit tests though, in which case this wouldn't have been an issue, right? | 15:11 |
| * dims nods to sdague's observation | 15:12 | |
| mriedem_meeting | sdague: like i said above, in this specific case, assuming we had unit tests using -constraints and assuming we didn't screw up and update u-c on stable/liberty, then in this specific case yes it would have failed the backport proposed to nova | 15:12 |
| mriedem_meeting | HOWEVER | 15:12 |
| mriedem_meeting | if that new method in oslo.serialization being used here were in 1.9.0 rather than 1.10.0, we would have passed tests but been broken for anyone using oslo.serialization 1.4.0 | 15:13 |
| mriedem_meeting | which is the min bound in g-r for stable/liberty | 15:13 |
| mriedem_meeting | so it's a matter of missing 1 week of oslo lib releases | 15:13 |
| *** gmurphy has joined #openstack-release | 15:13 | |
| mriedem_meeting | even if the new method were added right after 1.4.0 were released, it's a new API so it'd be a 1.5.0 release at least | 15:14 |
| sdague | mriedem_meeting: right, so minimum testing is still a thing that's all fubared | 15:14 |
| mriedem_meeting | and stable/liberyt would be capped at 1.5 in my mind | 15:14 |
| sdague | mriedem_meeting: what is the stable/liberty branch for oslo.serialization? | 15:14 |
| mriedem_meeting | what is the version? | 15:15 |
| sdague | yeh | 15:15 |
| mriedem_meeting | idk? we post-version | 15:15 |
| mriedem_meeting | https://github.com/openstack/oslo.serialization/commit/03f34312d437a1419ebd1f90da6b9a96aa7bcb36 | 15:15 |
| mriedem_meeting | 2.2.0? | 15:16 |
| mriedem_meeting | that doens't make sense, it must be 1.9.0 | 15:16 |
| mriedem_meeting | b/c dump_as_bytes isn't in here https://github.com/openstack/oslo.serialization/blob/stable/liberty/oslo_serialization/jsonutils.py | 15:16 |
| mriedem_meeting | dhellmann: aren't we supposed to revert these? https://review.openstack.org/#/c/246211/ and https://review.openstack.org/#/c/232918/ | 15:17 |
| sdague | so, honestly, lets ignore min bounds, because that never breaks *us* | 15:18 |
| sdague | if we tested with a constraints file which was the libraries at release time for stable/liberty + stable/liberty oslo updates, on both tempest and unit tests, would we have broken ourselves? | 15:19 |
| mriedem_meeting | in this specific case i think so | 15:21 |
| mriedem_meeting | b/c liberty GA oslo.serialization was 1.9.0 | 15:21 |
| mriedem_meeting | and the new method was in 1.10.0 | 15:21 |
| mriedem_meeting | b/c we bumped u-c on stable (probably erroneously), it's using new enough oslo.serialization for dsvm jobs to pass | 15:21 |
| dhellmann | mriedem_meeting : yes, probably we need to revert those, but maybe not entirely? I haven't studied the details yet | 15:24 |
| mriedem_meeting | i have a feeling that reverting those would fail tempest badly | 15:24 |
| mriedem_meeting | but idk | 15:25 |
| dhellmann | mriedem_meeting : let's see: https://review.openstack.org/#/c/264831/ | 15:25 |
| mriedem_meeting | you should probably revert in order? | 15:25 |
| dhellmann | yeah, here's the other: https://review.openstack.org/#/c/264832/1 | 15:26 |
| *** dims_ has joined #openstack-release | 15:35 | |
| *** bdemers_ has quit IRC | 15:35 | |
| *** dims has quit IRC | 15:36 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:42 | |
| *** mriedem_meeting is now known as mriedem | 15:42 | |
| *** bdemers has quit IRC | 15:45 | |
| dhellmann | mriedem : after giving it more thought, I'm going to prepare one patch to restore us to where we ought to be -- all of the constraints reset for liberty except our own releases | 15:45 |
| dansmith | dhellmann: does that mean rolling back o.vo or not? | 15:55 |
| *** bdemers has joined #openstack-release | 15:57 | |
| mriedem | this is probably something we can all agree on https://review.openstack.org/#/c/264495/ | 16:15 |
| mriedem | there was a regression for py27 in novaclient 3.0.0 | 16:16 |
| mriedem | breaking extensions that apparently ironic uses | 16:16 |
| jroll | mriedem: nah, rackspace uses them | 16:18 |
| jroll | surprise~~ | 16:18 |
| dhellmann | dansmith : I haven't gotten so far as to look at specific things. Did you want it rolled back, or is that going to break things? | 16:19 |
| mriedem | jroll: ah ok | 16:20 |
| dansmith | dhellmann: I think rolling back is going to require more change at this point, and our people have already started on updating our oslo liberty packages, since all of them have drifted away from those branches | 16:21 |
| dhellmann | dansmith : ok, noted. | 16:21 |
| dansmith | dhellmann: I mostly just want to know what you're thinking so we can stop doing that if you think that that's the plan | 16:21 |
| dansmith | there's no real great path out of the current box, that I know of | 16:21 |
| dansmith | so less change starting now is probably the least impactful | 16:22 |
| dhellmann | dansmith : I was going to experiment with rolling everything back to the stable branch versions, but if that's an issue for any libs I'm ok with living with that if we somehow note that we're not doing backports to those libs any more or something | 16:22 |
| dhellmann | at least until we get better testing in place, I suppose | 16:22 |
| dansmith | I think we also need to decide what to do with the existing wrong stable branches | 16:23 |
| dhellmann | mriedem : do you have a constraints update for that novaclient release? | 16:23 |
| dansmith | i.e. do we do a semver-breaking sync so that they match what we have in constraints now? or delete them to avoid confusion? or sync every change? | 16:23 |
| mriedem | dhellmann: forgot about that, sec | 16:24 |
| dhellmann | dansmith : if we get the release-constraints stuff implemented, then we can live with the difference in the constraints, right? | 16:24 |
| dansmith | dhellmann: I don't understand what you mean. IMHO, the stable/foo branches *have* to match what is tested, and what is in the lower constraints file | 16:24 |
| dansmith | dhellmann: have to match, or have to not exist (for confusion sake) | 16:25 |
| dhellmann | dansmith : I'm thinking about the end state, where we test both release-constraints and upper-constraints. If we have both sets of tests in place, then the constraints could differ as long as both tests pass. | 16:25 |
| dansmith | dhellmann: could differ from what is actually in the stable branch? | 16:26 |
| dhellmann | dansmith : release-constraints would be required to match the stable branch, upper-constraints would not. | 16:26 |
| dansmith | well, right, they can't *both* match :) | 16:26 |
| mriedem | dhellmann: https://review.openstack.org/#/c/264855/ | 16:27 |
| dhellmann | mriedem : ok, I'll work on that release now | 16:27 |
| mriedem | danka | 16:28 |
| openstackgerrit | Merged openstack/releases: Release python-novaclient 3.2.0 https://review.openstack.org/264495 | 16:29 |
| dhellmann | mriedem : done | 16:30 |
| mriedem | nice, thanks again | 16:31 |
| dhellmann | dansmith : is the minimum version for o.vo in liberty correct? | 16:33 |
| dansmith | dhellmann: it's not what o.vo was at when liberty released, no | 16:34 |
| dansmith | dhellmann: and what is in the upper-constraints is not what is on stable/liberty for o.vo | 16:34 |
| dhellmann | dansmith : does the minimum accurately reflect the version needed to work with liberty? | 16:34 |
| dansmith | and since there is no release-constraints yet, that one is wrong by default :) | 16:34 |
| dansmith | dhellmann: we have no minimum right? | 16:35 |
| dhellmann | global-requirements.txt:oslo.versionedobjects>=0.9.0 | 16:35 |
| dhellmann | upper-constraints.txt:oslo.versionedobjects===0.10.0 | 16:35 |
| dhellmann | oh, wait, that's probably my reverted version, hang on | 16:35 |
| dansmith | yeah, that's not current | 16:35 |
| dhellmann | upper-constraints.txt:oslo.versionedobjects===1.1.0 | 16:35 |
| dansmith | I guess I don't know what was in g-r, if that's a lower bound | 16:36 |
| dhellmann | ok, so does version 0.9.0 work? | 16:36 |
| dhellmann | yeah, that's the lower bound from g-r | 16:36 |
| dansmith | dhellmann: I don't know, I'd have to go look, but 0.10 is what is in the stable/liberty branch (I think), so I expect 0.10 is what g-r should say | 16:36 |
| dansmith | actually, | 16:36 |
| dhellmann | well, no | 16:37 |
| dansmith | we had to patch stable/liberty nova to make it work with 1.1.0 I think | 16:37 |
| dansmith | so I don't know that 0.10 will even still work with stable/liberty to be honest | 16:37 |
| dansmith | we probably need a test run to see | 16:37 |
| dhellmann | the minimum value doesn't need to reflect the branch at all, as long as it works | 16:37 |
| dhellmann | yeah | 16:37 |
| dansmith | dhellmann: well, it depends on what lower bound we're testing with | 16:37 |
| dhellmann | I'm worried that we need to set the minimum to 1.1.0 | 16:37 |
| dansmith | dhellmann: if we're testing 0.10 because that's in the branch, but requirements say 0.9 then I think that's a problem, but... | 16:38 |
| dansmith | dhellmann: right, I think that might be the case now.. we're definitely in a weird spot at the moment | 16:38 |
| dhellmann | we have to keep these separate types of tests distinct | 16:38 |
| dhellmann | we need to establish the earliest version that works, the current version from the branch that works, and then the newest overall version that works | 16:38 |
| dhellmann | those are 3 separate questions | 16:39 |
| dansmith | right, which is my primary complaint about the complexity here. I'd like to just offer one version that works :) | 16:39 |
| dansmith | I'm not sure how we get testing to show that the version in g-r works, unless we're really going to test that, the release-constraints version, and the upper-constraints version in each stack | 16:39 |
| dhellmann | the goal of the constraints file is to do that. but we can't pin versions in the dependency list of projects because it makes it impossible to upgrade anything | 16:39 |
| dhellmann | dansmith : submitting a job to change the constraint to match the minimum would tell us if it works | 16:40 |
| dansmith | dhellmann: I mean how we get automatic testing of those three points in peacetime. I know we can submit a bunch of canary patches to get runs for this now, but longer term we'll still have three points we need to care about at all times, it sounds like | 16:41 |
| dhellmann | dansmith : right, we need to add more test jobs so we're testing all 3 cases all the time. but for now, I'm just trying to work out how to get the data back to a good state, and I don't want to wait for those jobs to do it | 16:42 |
| dansmith | do we get multinode/partial nova results on changes to the requirements repo/ | 16:43 |
| dhellmann | I don't know off the top of my head. What job is that? | 16:44 |
| dansmith | that was rhetorical, I'm checking, just a sec :) | 16:44 |
| dansmith | I think we do | 16:46 |
| dansmith | dhellmann: are you going to submit those canaries | 16:46 |
| dansmith | ? | 16:46 |
| dhellmann | dansmith : I'm working on the hard reset patch, do you want to do that one for o.vo? | 16:46 |
| dhellmann | I'm assuming that's the only one we have a major issue with, but maybe I'm wrong | 16:47 |
| dansmith | my packaging guys say that all the oslo libs in upper-constraints don't match the corresponding stable branches | 16:47 |
| dansmith | so I'm not sure why o.vo would be unique here, but.. I don't follow those | 16:47 |
| dhellmann | yeah, we just updated them all, I expect the others to be easier to roll babck | 16:49 |
| mriedem | dansmith: dhellmann: requirements changes test with grenade but not partial n-cpu | 16:49 |
| mriedem | https://review.openstack.org/#/c/246564/ | 16:49 |
| mriedem | unless that's the multinode job? | 16:49 |
| dansmith | mriedem: yeah, multinode job does it I think | 16:50 |
| mriedem | ok, new name for the old partial ncpu job then | 16:50 |
| dansmith | the example I looked at was failing, so I didn't check that it's actually doing the right thing, but I will on this one | 16:50 |
| dansmith | https://review.openstack.org/264860 | 16:50 |
| dhellmann | dansmith, mriedem : we should start an etherpad to track this work, can one of you do that? | 16:54 |
| mriedem | i'm unclear on what we're doing, except trying to get stable/liberty u-c back to near liberty GA form | 16:55 |
| dhellmann | mriedem : that's right, but we're likely to have several experiments and so I thought keeping track of the patches involved and the attempts we made to get there would help | 16:56 |
| dansmith | so people that have already moved up to what u-c says will _now_ be running untested stuff, right? | 16:56 |
| mriedem | dhellmann: https://etherpad.openstack.org/p/stable-liberty-constraints-sanity | 16:56 |
| dansmith | meaning if they pick up a new nova backport, that backport won't be getting tested against what they've rolled to right? | 16:56 |
| openstackgerrit | Doug Hellmann proposed openstack/releases: add list-constraints command https://review.openstack.org/264866 | 16:57 |
| dhellmann | dansmith : I don't necessarily expect us to be able to roll everything back. And when we get the separate jobs set up, we'll have tests for both versions. Right now I'm trying to figure out what the plan to get to that state *is* I'm not proposing that we actually merge these changes without agreeing on that. | 16:59 |
| dansmith | sure, I'm just saying.. rolling back at this point might be somewhat uncool for CDers | 16:59 |
| dhellmann | right | 16:59 |
| *** dims has joined #openstack-release | 16:59 | |
| dansmith | feels like it'd be better to freeze where we are, mea culpa, and stop the bleeding | 17:00 |
| dhellmann | so we'll make a list of patches to roll back, and maybe use those same values to create a release-constraints.txt file to start with | 17:00 |
| dansmith | but, anyway, damned either way I guess | 17:00 |
| dhellmann | well, yeah, I don't think we want to approve any more constraints changes | 17:00 |
| mriedem | heh, i'm pink, doug is light pink | 17:01 |
| mriedem | thanks etherpad | 17:01 |
| mriedem | -2 is in place on bot updates to u-c on stable https://review.openstack.org/#/c/257693/ | 17:02 |
| *** dims_ has quit IRC | 17:02 | |
| dhellmann | https://review.openstack.org/264878 reset upper-constraints for stable/liberty | 17:14 |
| dhellmann | https://review.openstack.org/264879 update constraints for our managed libraries | 17:14 |
| dhellmann | mriedem, dansmith : more experiments ^^ | 17:15 |
| *** dtantsur is now known as dtantsur|afk | 17:15 | |
| mriedem | dhellmann: was https://review.openstack.org/264878 based on a specific commit hash? | 17:17 |
| dhellmann | yeah, I should have documented that let me find it again | 17:18 |
| dhellmann | mriedem : commit message updated | 17:19 |
| dhellmann | mriedem, dansmith, lifeless, dims, sdague: here's the project-config change to disable constraints updates for liberty: https://review.openstack.org/264883 | 17:26 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:27 | |
| *** bdemers has quit IRC | 17:36 | |
| *** bdemers has joined #openstack-release | 17:40 | |
| *** yarkot has joined #openstack-release | 17:53 | |
| *** bdemers has quit IRC | 17:58 | |
| *** bdemers has joined #openstack-release | 18:00 | |
| *** bdemers has quit IRC | 18:01 | |
| dims | dhellmann : ack | 18:04 |
| *** bdemers has joined #openstack-release | 18:04 | |
| lifeless | mriedem: minimums being wrong is also a known problem, but it affects master equally | 18:05 |
| lifeless | mriedem: caps won't address that | 18:05 |
| lifeless | mriedem: (unless I'm confused?) | 18:05 |
| dansmith | depends on whether you mean caps are <=some_version or ==some_version | 18:11 |
| dhellmann | lifeless : can you put this instruction change on your review queue, please? https://review.openstack.org/264907 | 18:20 |
| dhellmann | jroll : I think you might be interested, too https://review.openstack.org/264907 add simpler instructions for adding new dependencies | 18:20 |
| *** ifat_afek_ has quit IRC | 18:23 | |
| lifeless | dhellmann: I'm not sure i agree with the premise | 18:26 |
| dhellmann | lifeless : I want to understand why each line of that file is changed when a human makes the change. | 18:27 |
| lifeless | dhellmann: sure. The existing instructions give us only the new dependencies | 18:28 |
| dhellmann | ok, that was not the results we were seeing on a recent patch, let me find it | 18:28 |
| lifeless | dhellmann: I'll be spotty for the next 90m - family morning time | 18:29 |
| dhellmann | lifeless : ok | 18:29 |
| dhellmann | lifeless : early versions of https://review.openstack.org/#/c/220268/ had all sorts of updates included | 18:29 |
| lifeless | so https://review.openstack.org/#/c/220268/2/upper-constraints.txt looks like it is entirely new things, not changes to existing things | 18:31 |
| dhellmann | https://review.openstack.org/#/c/220268/11/upper-constraints.txt | 18:31 |
| lifeless | 3 looks... ok | 18:32 |
| dhellmann | I have no really way of evaluating that set of changes without reproducing it myself. | 18:32 |
| *** mriedem has quit IRC | 18:32 | |
| lifeless | so 11 does look problematic. We'd have to ask, but I suspect they didn't use the diff technique requested in README (right under your patched lines) | 18:32 |
| dhellmann | that could be | 18:33 |
| dhellmann | even if they had, though, as a reviewer I don't know what to do with that number of changes | 18:33 |
| dhellmann | is Pillow related to this new thing? should it be updated? | 18:33 |
| lifeless | we could automake it further, though its not hard at the moment - run, patch, run, diff, patch | 18:33 |
| dhellmann | maybe we can enhance the check job to help with that? | 18:33 |
| dhellmann | yeah | 18:33 |
| dhellmann | yeah, well, as a reviewer I don't want to have to reproduce the patch to check its validity, so I was offering some new instructions with a compromise | 18:34 |
| lifeless | sure | 18:34 |
| lifeless | so we can either make the check job do more | 18:34 |
| lifeless | or make it easier for submitters | 18:34 |
| lifeless | or both | 18:34 |
| lifeless | my concern with your patch is that its harder for submitters | 18:34 |
| dhellmann | harder? | 18:35 |
| lifeless | because figuring out what dependencies something has is nontrivial | 18:35 |
| lifeless | they are often not clearly documented, or the clear documentation is wrong | 18:35 |
| dhellmann | should we leave those out, then? just add the new thing | 18:35 |
| lifeless | and we need the transitive deps | 18:35 |
| lifeless | like | 18:35 |
| lifeless | seeing that we're bringing in twisted should probably be setting of alarm bells | 18:35 |
| dhellmann | mimic is a new testing tool, so I wasn't too worried about it | 18:36 |
| lifeless | so its going to mean that folk debugging testing failures need to read twisted code | 18:36 |
| lifeless | as a project we moved away from twisted | 18:36 |
| dhellmann | I don't think so, it's a functional test tool | 18:36 |
| dhellmann | it was described as mock for services | 18:36 |
| lifeless | dhellmann: its an httpd that pretends to be nova | 18:37 |
| dhellmann | the client lib tests would not be running twisted code, iiuc | 18:37 |
| lifeless | and keystone etc | 18:37 |
| dhellmann | right | 18:37 |
| lifeless | if it behaves wrongly | 18:37 |
| lifeless | - if it has a bug | 18:37 |
| *** mriedem has joined #openstack-release | 18:38 | |
| lifeless | won't that mean that whoever is trying to figure out whats up is now reading twisted code ? | 18:38 |
| dhellmann | I don't know | 18:39 |
| dhellmann | I see what you're saying. I have no idea how likely that is. | 18:39 |
| lifeless | Personally I like twisted, for a bunch of reasons. But I'm worried that this is essentialy stealth-reintroducing it | 18:39 |
| dhellmann | do you want to revert that and talk with jroll and the ironic team? | 18:40 |
| lifeless | lets talk first | 18:40 |
| lifeless | jroll: ^ | 18:40 |
| lifeless | anyhow - thats my point about seeing the pins change | 18:41 |
| lifeless | we also have this thing about licensing etc, and really it should be a transitive concern | 18:41 |
| lifeless | that we're not doing anything about right now | 18:41 |
| lifeless | but the same data could be useful | 18:41 |
| dhellmann | I still have no way of knowing if the proposer followed the instructions and got that right or not :-/ | 18:41 |
| lifeless | yah | 18:41 |
| lifeless | you wouldn't with your instructions either - there's a huge good faith assumption present in both cases | 18:42 |
| jroll | lifeless: dhellmann hi | 18:42 |
| dhellmann | true | 18:42 |
| jroll | what's up? | 18:42 |
| jroll | oh, twisted | 18:42 |
| lifeless | jroll: so, mimic is written in twisted, and its something folk will potentially have to debug when tests fail | 18:42 |
| lifeless | jroll: and while I like twisted and support bringing it or asyncio or something back, the current thing is 'no thanks' | 18:43 |
| lifeless | jroll: and - I don't recall any discussion on -dev about this | 18:43 |
| lifeless | jroll: perhaps I'm being overly sensitive to how folk may react | 18:43 |
| jroll | lifeless: honestly I was wondering how folks would react, too, I'm surprised this is the first time someone has brought it up | 18:44 |
| lifeless | dhellmann: so, it would take a while, but not terribly long, to do two runs and recreate the scenario in check | 18:44 |
| lifeless | jroll: I suspect only a few folk have seen | 18:44 |
| dhellmann | lifeless : 2 runs? | 18:44 |
| jroll | indeed | 18:44 |
| lifeless | dhellmann: git checkout HEAD^; generate-constraints -o baseline ; git checkout ORIG_HEAD; generate-constraints -o new; git checkout HEAD^:upper-constraints.txt; patch < (diff baseline new) | 18:46 |
| lifeless | dhellmann: or approximately that | 18:46 |
| lifeless | dhellmann: better I think to do the two runs and present the reviewer with the diff rather than auto-failing | 18:46 |
| lifeless | dhellmann: but thats still two runs | 18:46 |
| lifeless | jroll: so, I suggest a thread on -dev, rather than waiting | 18:47 |
| dhellmann | ok, I see what you mean now. Yes, let's add that. Requirements changes run more complex and longer running jobs than that already, I imagine. | 18:47 |
| jroll | lifeless: so I get your point about twisted, I'm okay with it personally, and I'm happy to talk about it further if you like | 18:47 |
| lifeless | dhellmann: right, devstack :) | 18:48 |
| jroll | lifeless: I'm not sure what to ask on -dev, "we brought in twisted, any objection?" | 18:48 |
| lifeless | jroll: fundamentally, yes | 18:48 |
| dhellmann | jroll : yeah, I think lifeless' point is this is a bigger dependency than we've added in a while, and involves some tools we have previously said we didn't, as a community, want to use, so combining those two things means we should talk about it some before going ahead too fast | 18:49 |
| lifeless | afk | 18:49 |
| jroll | yeah | 18:51 |
| jroll | dhellmann: lifeless: is someone reverting then I guess? | 18:51 |
| dhellmann | jroll : we haven't, yet. I'm trying to decide if that's necessary. | 18:52 |
| jroll | ok | 18:52 |
| dhellmann | jroll : why don't you start the discussion and we'll see; and hold off on approving changes in ironic related to this for a day or two? | 18:52 |
| dhellmann | I'd hate to churn a bunch of patches if it turns out to not be a big deal, so let's hit the slow-mo button instead of rewind :-) | 18:53 |
| jroll | dhellmann: okay | 18:53 |
| dhellmann | great, thank you | 18:53 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:54 | |
| jroll | dhellmann: is there a requirements tag or should I just [all] this? | 18:55 |
| dhellmann | jroll : use all, I think | 18:55 |
| jroll | k | 18:55 |
| jroll | thanks | 18:55 |
| dhellmann | jroll : thanks for your patience on this | 18:55 |
| jroll | np | 18:56 |
| openstackgerrit | Doug Hellmann proposed openstack/releases: add send-announcements-to field https://review.openstack.org/262279 | 19:01 |
| jroll | dhellmann: lifeless: fired it off | 19:09 |
| dansmith | mriedem: I didn't get a grenade run on this: https://review.openstack.org/#/c/264860/ | 19:17 |
| dansmith | mriedem: nor nova unit tests, but maybe that's expected? | 19:17 |
| mriedem | dansmith: nova unit tests don't run on requirements repo chagnes | 19:18 |
| mriedem | *changes | 19:18 |
| mriedem | grenade ran o nit | 19:18 |
| mriedem | just not multinode | 19:18 |
| dansmith | sorry, not grenade multihost tempest | 19:18 |
| dansmith | regular grenade did, but that's not helpful | 19:18 |
| mriedem | hmm, do we run multinode on liberty changes? | 19:18 |
| dansmith | so I guess I have to run the nova unit test manually here? | 19:18 |
| mriedem | checking | 19:18 |
| mriedem | probably | 19:18 |
| dansmith | ah, | 19:18 |
| dansmith | I was looking at a master change that's why | 19:19 |
| dansmith | but I guess we didn't run partial on these? | 19:19 |
| mriedem | i guess not | 19:19 |
| mriedem | probably should | 19:19 |
| mriedem | i see we run partial grenade on nova stable/liberty changes https://review.openstack.org/#/c/258695/ | 19:19 |
| mriedem | there is probably a branch filter on the requirements repo and that job in project-config | 19:19 |
| dansmith | nova ones yeah | 19:19 |
| dansmith | well, since I think the guarantees this is going to provide are pretty weak, I guess I'll make sure nova unit tests pass | 19:20 |
| dansmith | and if so, then we can just assume it's close enough I suppose | 19:20 |
| *** bdemers has quit IRC | 19:30 | |
| *** nikhil_k has joined #openstack-release | 19:31 | |
| *** yarkot has quit IRC | 19:31 | |
| *** yarkot has joined #openstack-release | 19:31 | |
| *** mriedem1 has joined #openstack-release | 19:33 | |
| *** dims_ has joined #openstack-release | 19:33 | |
| *** dims has quit IRC | 19:34 | |
| *** stevemar_znc has joined #openstack-release | 19:37 | |
| *** mriedem has quit IRC | 19:38 | |
| *** nikhil has quit IRC | 19:38 | |
| *** sileht has quit IRC | 19:38 | |
| *** stevemar has quit IRC | 19:38 | |
| *** mriedem1 is now known as mriedem | 19:38 | |
| *** stevemar_znc is now known as stevemar | 19:40 | |
| *** sileht has joined #openstack-release | 19:42 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:45 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:48 | |
| *** dhellmann_ has joined #openstack-release | 19:55 | |
| *** bdemers has joined #openstack-release | 19:59 | |
| *** dims_ has quit IRC | 20:00 | |
| *** dhellmann has quit IRC | 20:11 | |
| *** dhellmann_ is now known as dhellmann | 20:11 | |
| *** dhellmann has quit IRC | 20:12 | |
| *** dims has joined #openstack-release | 20:12 | |
| *** dhellmann has joined #openstack-release | 20:14 | |
| *** dims has quit IRC | 20:19 | |
| *** dims has joined #openstack-release | 20:22 | |
| *** openstackgerrit has quit IRC | 20:23 | |
| *** dims has quit IRC | 20:23 | |
| *** openstackgerrit has joined #openstack-release | 20:24 | |
| dansmith | dhellmann: o.vo 0.9 and 0.10 work with liberty nova because we removed that offending test in nova | 20:27 |
| openstackgerrit | Adam Gandelman proposed openstack/releases: Release python-keystoneclient 1.3.4 https://review.openstack.org/264952 | 20:28 |
| dansmith | dhellmann: I'm still not really willing to claim that it's okay to roll back to those for real until/unless I get partial or multinode-grenade jobs that prove it | 20:28 |
| dansmith | dhellmann: and we really need a partial from k->l and a multinode-grenade run from l->m(aster) | 20:28 |
| dhellmann | dansmith : we should start making a list of these test jobs that we need to add in the etherpad mriedem set up | 20:29 |
| dhellmann | https://etherpad.openstack.org/p/stable-liberty-constraints-sanity | 20:29 |
| dhellmann | I agree, we need to set up the tests before we roll back | 20:29 |
| dhellmann | let's get the list together so we make sure we hit them all | 20:29 |
| dansmith | done | 20:30 |
| dhellmann | dansmith : great, thanks | 20:31 |
| mriedem | dansmith: dhellmann: looks like the multinode job runs in master branch requirements changes https://review.openstack.org/#/c/264855/ | 20:34 |
| mriedem | so we have liberty->mitaka multinode testing | 20:34 |
| mriedem | here is our problem on stable | 20:37 |
| mriedem | - name: ^gate-grenade-dsvm-multinode | 20:37 |
| mriedem | branch: ^(?!stable/(kilo|liberty)).*$ | 20:37 |
| dhellmann | that looks like an easy fix | 20:38 |
| mriedem | well | 20:38 |
| mriedem | https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L1080-L1089 | 20:38 |
| dhellmann | wait, no, what's wrong with that? it's running for liberty, right? | 20:38 |
| mriedem | i'm not sure the multinode job works from kilo->liberty | 20:38 |
| dhellmann | ah | 20:38 |
| mriedem | it's not running for liberty changes in the requirements repo | 20:38 |
| mriedem | the old partial-ncpu job is marked as deprecated | 20:38 |
| mriedem | but that's the one we'd run on stable/liberty requirements changes (i think) | 20:39 |
| *** dims has joined #openstack-release | 20:39 | |
| mriedem | i think we can just add gate-grenade-dsvm-partial-ncpu to the requirements repo | 20:40 |
| mriedem | given the filter, it should only run on stable/liberty changes | 20:40 |
| mriedem | let me try | 20:40 |
| lifeless | back | 20:46 |
| sdague | right, the multihost grenade job doesn't work until M1 | 20:52 |
| sdague | it's pretty new | 20:52 |
| sdague | mriedem: though, it might be easy enough to backport those changes | 20:53 |
| dansmith | sdague: why does <M1 matter | 20:53 |
| dansmith | / | 20:53 |
| dansmith | dammit -> ? | 20:53 |
| dansmith | we just care about liberty (with o.vo 0.10) -> master | 20:54 |
| sdague | if you only care about liberty -> master then why https://review.openstack.org/264957 | 20:54 |
| sdague | that's about kilo -> liberty | 20:54 |
| dansmith | sdague: for k->l I said we should have the partial job, not the multinode one | 20:56 |
| sdague | dansmith: right | 20:56 |
| sdague | however, it occurs to me that the multinode job might be pretty close to working | 20:56 |
| dansmith | for k->l ? | 20:57 |
| sdague | for k -> l, given that a lot of the changes were in things which don't have branches | 20:57 |
| dansmith | huh, okay | 20:57 |
| sdague | it might be worth doing some runs and figuring out what would be needed to completely fix it | 20:57 |
| dansmith | well, we really just need to see one or two good runs.. if the partial job works, seems like that's easier | 20:58 |
| mriedem | good runs | 20:58 |
| sdague | ok, so if that's the case, are we going to leave this on all the time? | 20:59 |
| *** nikhil_k is now known as nikhil | 21:29 | |
| *** dtantsur|afk has quit IRC | 21:50 | |
| *** dtantsur|afk has joined #openstack-release | 21:51 | |
| *** bdemers has quit IRC | 21:51 | |
| *** bdemers has joined #openstack-release | 22:20 | |
| *** mriedem has quit IRC | 22:54 | |
| *** dims_ has joined #openstack-release | 23:00 | |
| *** dims has quit IRC | 23:01 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:12 | |
| *** gordc has quit IRC | 23:28 | |
| *** dims_ has quit IRC | 23:39 | |
| *** bdemers has quit IRC | 23:41 | |
| *** doug-fish has quit IRC | 23:41 | |
| *** mriedem has joined #openstack-release | 23:44 | |
| *** dims_ has joined #openstack-release | 23:45 | |
| *** dims_ has quit IRC | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!