| fungi | clarkb: i think we never did anything for 1253774, no. probably first you want to hop onto bare-precise, py3k-precise and devsatck-precise node examples and see what's lingering in their sources | 00:00 |
|---|---|---|
| sdague | yeh, salv-orlando if there were different configs that you liked, we could do them. But I definitely think running more than one tempest job helped the project | 00:00 |
| clarkb | fungi: ya just held a devstack-precise node to do that | 00:00 |
| clarkb | fungi: there is the puppetlabs sources list in sources.list.d | 00:00 |
| *** aysyd has quit IRC | 00:01 | |
| fungi | clarkb: noting that each of them may have different stray things (for example the py3k nodes have ppa links for python3.3 and pypy) | 00:01 |
| clarkb | fungi: quick question though. will apt-get freak out if it has installed a thing from a source (puppet) which results in a newer version of the normal repo when it goes to install other things? | 00:01 |
| salv-orlando | sdague: I think it's ok this way. I'm not sure we need to enable different configs at this stage. I just saw the doubles and wondered if it was by design or not. | 00:01 |
| fungi | clarkb: it will not care in the least | 00:01 |
| clarkb | fungi: I think my plan is to rm /etc/sources.list.d/* | 00:01 |
| clarkb | fungi: awesome. I will just propose a change that does ^ | 00:01 |
| fungi | clarkb: it leaves packages it doesn't know about alone unless it sees newer versions available somewhere it knows about | 00:02 |
| clarkb | fungi: rm'ing sources.list.d/* is sufficient right? | 00:02 |
| fungi | clarkb: sounds perfectly safe | 00:02 |
| fungi | clarkb: note that if that were not safe, it would also muck with packages you install via dpkg -i somefile.deb | 00:02 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291108 https://review.openstack.org/79812 | 00:02 |
| fungi | clarkb: you *do* want to do another apt-get update after that though, just to be on the safe side (in case something later tries to do an apt-get install without first doing another update) | 00:03 |
| salv-orlando | sdague: as a side question, do you remember if before the integrated gate template we were running exercises for neutron jobs? | 00:03 |
| krotscheck | jeblair, clarkb: We test our migrations against mysql and postgres - see TestMigrations, it overrides the necessary properties to make the cifail switch not flip. You're right though, we don't set up a DB per test. | 00:03 |
| sdague | salv-orlando: a long time ago | 00:03 |
| sdague | we phased out exercises basically entirely last summer, except for grenade | 00:04 |
| *** gokrokve has quit IRC | 00:04 | |
| sdague | and we got rid of them in grenade a month ago | 00:04 |
| krotscheck | jeblair, clarkb: hence the yelling. | 00:04 |
| clarkb | fungi: oh right thanks | 00:05 |
| *** yamahata__ has joined #openstack-infra | 00:06 | |
| *** thuc_ has joined #openstack-infra | 00:07 | |
| *** thuc_ has quit IRC | 00:07 | |
| *** thuc_ has joined #openstack-infra | 00:08 | |
| salv-orlando | sdague: ok.. so exercise should not be running for check-tempest-dvsm-neutron? | 00:08 |
| jogo | mriedem: unit test are called gate in check queue as well | 00:09 |
| *** thuc_ has quit IRC | 00:09 | |
| mriedem | jogo: well kiss my grits | 00:09 |
| jogo | re: https://review.openstack.org/#/c/79812/ | 00:09 |
| clarkb | we lost gerrit bot | 00:10 |
| clarkb | fungi: https://review.openstack.org/79814 | 00:10 |
| * jogo likes his grits cheesy | 00:10 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Remove unneeded apt sources from nodepool nodes. https://review.openstack.org/79814 | 00:10 |
| clarkb | oh nvermind, just slow | 00:10 |
| mriedem | jogo: anyway, if we drop build_name from the query that's one less wildcard search | 00:10 |
| mriedem | and sdague has me all paranoid about wildcards in the queries now | 00:10 |
| mriedem | jogo: or is that somehow offset by filtering by build_name, even with the wildcard? | 00:10 |
| mriedem | clarkb: ^? | 00:10 |
| mriedem | i had grits in tennessee last year for the first time, grossed me out | 00:11 |
| jogo | mriedem: not sure which way is better, I agree lets ask clarkb | 00:11 |
| clarkb | mriedem: instead of using the wildcard be specific | 00:11 |
| clarkb | and OR them all together | 00:11 |
| mriedem | yeah, that's the alternative | 00:11 |
| fungi | mriedem: what's gross about grits? you just soak dry corn in wet lye for a while and then... | 00:11 |
| mriedem | pew | 00:12 |
| clarkb | gate-cinder-python27 OR gate-cinder-python26 OR gate-cinder-pypy andso on (I don't think we are testing pypy or py3k with cinder yet, 26 and 27 are the values you need) | 00:12 |
| mriedem | clarkb: we don't even really need build_name in the query though, | 00:12 |
| sdague | salv-orlando: correct | 00:12 |
| mriedem | this only shows up in python UT jobs anyway | 00:12 |
| fungi | mriedem: but really, if you grow up with them, it's no stranger than, say, lutefisk | 00:12 |
| mriedem | fungi: also gross, and my mom's side is swedish | 00:13 |
| mriedem | so every xmas was lutefish or oyster stew, both nasty | 00:13 |
| salv-orlando | sdague: I'm off to bed now, but I'll leave you with this -> http://logs.openstack.org/43/79443/3/check/check-devstack-dsvm-neutron/6118ec2/console.html | 00:13 |
| mriedem | until uncle don put his foot down and brought a ham one year, xmas was saved! | 00:13 |
| fungi | mriedem: you just have an aversion to caustic alkaline-preserved foods, obviously ;) | 00:13 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291108 https://review.openstack.org/79812 | 00:13 |
| sdague | salv-orlando: so.... that job exists for grenade | 00:13 |
| *** mgagne has joined #openstack-infra | 00:13 | |
| sdague | except we don't need it any more | 00:13 |
| clarkb | mriedem: right you aren't doing that to make sure you only see those jobs you are doing it so that the poor ES cluster doesn't have to search everything | 00:14 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try patched libvirt from @hallyn - Not for review https://review.openstack.org/79816 | 00:14 |
| clarkb | mriedem: filters restrict the search space | 00:14 |
| mriedem | clarkb: ok, that's the answer i was looking for | 00:14 |
| *** dstufft has quit IRC | 00:14 | |
| salv-orlando | sdague: so I can avoid worrying about those failures, and move on to the next one? | 00:14 |
| sdague | clarkb: so in reality, do you think wildcards in filters like that is actually helpful or not | 00:15 |
| *** dstufft has joined #openstack-infra | 00:15 | |
| sdague | salv-orlando: yeh, I think we can actually probably delete that job honestly | 00:15 |
| salv-orlando | ok thanks then | 00:15 |
| salv-orlando | and goodnight! | 00:15 |
| clarkb | sdague: I think it is less helpful. wildcards are a search. non wildcards are a filter | 00:15 |
| sdague | goodnight | 00:15 |
| clarkb | sdague: so by having that as a wildcard we have to serach through that space. having it as a filter reduces any remaining search space | 00:16 |
| sdague | clarkb: so if that was SQL, it would actually be slower with the wildcards | 00:16 |
| sdague | because the message field is full text indexed, so it's pretty fast | 00:16 |
| sdague | but that ends up being string compares with regex, which is slow | 00:17 |
| clarkb | sdague: thats what I am saying | 00:17 |
| clarkb | wildcards are slower | 00:17 |
| sdague | clarkb: I mean slower with wildcards, then making no reference to that field at all | 00:17 |
| mriedem | ^ that's what i was wondering | 00:17 |
| sdague | as in deleting the filter would make the query faster | 00:17 |
| *** yamahata has joined #openstack-infra | 00:18 | |
| clarkb | sdague: making no reference in this case is probbaly slower because then the console.html for every single job in the last 2 weeks will be searched | 00:18 |
| clarkb | sdague: rather than searching through only the those for gate-python-cinder2X | 00:18 |
| sdague | right, but the way it will be searched is actually fast | 00:18 |
| sdague | relatively | 00:18 |
| *** atiwari has quit IRC | 00:18 | |
| clarkb | sdague: right but we are talking orders of magnitude different data that must be combed through (granted the index helps so it isn't that terrible) | 00:18 |
| sdague | it would be interesting to ask an ES person about things like these actually | 00:18 |
| clarkb | sdague: I did | 00:19 |
| clarkb | at a local meetup | 00:19 |
| clarkb | sdague: filtering is good | 00:19 |
| clarkb | filtering != wildcards | 00:19 |
| sdague | ok, awesome +1 | 00:19 |
| sdague | right, filtering definitely good | 00:19 |
| sdague | I guess the wildcard thing would be interesting to understand | 00:19 |
| clarkb | so explicitly spell it out. build_name:"gate-cinder-python26" OR build_name:"gate-cinder-python27" | 00:19 |
| sdague | right that would be faster | 00:19 |
| clarkb | sdague: I think if you flip the order of the searches it may be faster | 00:20 |
| clarkb | console.html is about half our log data | 00:20 |
| clarkb | so on a busy day a quarter of a billion documents | 00:20 |
| sdague | it doesn't do query reordering automatically? | 00:20 |
| sdague | most of those do | 00:20 |
| *** thuc has joined #openstack-infra | 00:20 | |
| clarkb | if you can restrict that first with a relatively cheap wildcard serach to go from .25 billion to .01 billion I think you win | 00:21 |
| clarkb | sdague: I am not sure. the behavior I have seen is that order matters but I haven't heard that from someone that actually know | 00:21 |
| sdague | so it might be really nice to add another tag for "unittests" | 00:21 |
| *** thuc has quit IRC | 00:21 | |
| clarkb | sdague: we don't have a reliable way to do that | 00:21 |
| clarkb | same problem as slave az and image | 00:21 |
| sdague | because then we could tag:unittests AND project:openstack/cinder | 00:21 |
| *** thuc has joined #openstack-infra | 00:22 | |
| sdague | clarkb: *python* ? | 00:22 |
| *** thuc has quit IRC | 00:22 | |
| sdague | when we create that field | 00:22 |
| clarkb | sdague: but that misses pypy | 00:22 |
| clarkb | and py3k | 00:23 |
| clarkb | and all the funny django tests | 00:23 |
| clarkb | and so on | 00:23 |
| clarkb | :/ | 00:23 |
| *** thuc has joined #openstack-infra | 00:23 | |
| sdague | ok, well we can add them over time | 00:23 |
| sdague | it would still be much better to filter on | 00:23 |
| sdague | just a thought | 00:23 |
| mriedem | sdague: clarkb: so the take away for me is be really strict now on wildcard usage when i see it in reviews | 00:23 |
| *** thuc has quit IRC | 00:23 | |
| mriedem | i'd like to get something in the readme at least on wildcards | 00:23 |
| clarkb | sdague: I agree, I just think we need to be better about the information upfront rather than approximating | 00:23 |
| mriedem | we used to say it wasn't supported/didn't work, but it does... | 00:23 |
| clarkb | sdague: because with approximations we will fail and confusion will happen | 00:24 |
| *** thuc has joined #openstack-infra | 00:24 | |
| sdague | bah, approximations are the spice of life :) | 00:26 |
| *** matsuhashi has joined #openstack-infra | 00:26 | |
| *** mgagne has quit IRC | 00:26 | |
| fungi | the approximations must flow | 00:26 |
| mriedem | and he who controls the spice controls the galaxy | 00:26 |
| fungi | d'oh! | 00:26 |
| mriedem | ha | 00:27 |
| mriedem | where is my wife, she loves dune references... | 00:27 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Load project settings from ini file, if present. https://review.openstack.org/71323 | 00:27 |
| * fungi whistles and walks away from this mess | 00:27 | |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: allow sqla 0.9.x https://review.openstack.org/79817 | 00:27 |
| *** bhuvan_ has quit IRC | 00:27 | |
| *** bhuvan has quit IRC | 00:27 | |
| mriedem | sdague: ^ typo in there i think | 00:28 |
| *** bhuvan has joined #openstack-infra | 00:28 | |
| *** bhuvan_ has joined #openstack-infra | 00:28 | |
| sdague | mriedem: probably, we need test results anyway | 00:29 |
| mriedem | fungi: clarkb: do you know where the code is that sends out the automatic global-reqs updates to the other projects? | 00:30 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1291108 https://review.openstack.org/79812 | 00:30 |
| clarkb | mriedem: ya I will get a link | 00:31 |
| clarkb | mriedem: it needs to not do that when there are changes in the gate otherwise they get kicked out | 00:31 |
| fungi | mriedem: just a sec, let me let clarkb get that for you ;) | 00:31 |
| mriedem | clarkb: what needs to not do what now? | 00:31 |
| *** mgagne has joined #openstack-infra | 00:32 | |
| fungi | mriedem: when the gate is long, a project might approve their requirements updates change while there is a requirements change ahead of them in te gate | 00:32 |
| clarkb | mriedem: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/propose_requirements_update.sh those jobs push updates immediately after a requirements change merges | 00:32 |
| clarkb | then what fungi says happens and we kick stuff out of the gate due to new patchsets | 00:32 |
| *** markmcclain has joined #openstack-infra | 00:32 | |
| fungi | mriedem: then the merger of the global requirements change triggers an update job, and snipes any currently approved per-project requirements changes pending merger | 00:32 |
| *** HenryG_ has quit IRC | 00:32 | |
| clarkb | also maybe it hsan't been advertized enough but requirements should be feature frozen too for the most part | 00:32 |
| clarkb | so now is a good time to get everything synced | 00:33 |
| mriedem | clarkb: reqs freeze is kind of what i was thinking too | 00:33 |
| fungi | clarkb: yes, i think feature freeze and requirements freeze should go hand-in-hand personally | 00:33 |
| *** gokrokve has joined #openstack-infra | 00:33 | |
| *** mgagne has quit IRC | 00:33 | |
| clarkb | I wonder if we can make that a ttx release check, for project in $PROJECTS; do check_reqs_syncage ; done | 00:34 |
| clarkb | iirc nova's havana release was immediately broken because we didn't do ^ | 00:34 |
| fungi | though fwiw, in havana there was a lot less time to get all that matched up, but we definitely hoped that for icehouse everyone would be in sync | 00:35 |
| clarkb | this felt like a productive bug day | 00:35 |
| clarkb | pleia2: thank you for putting it together (and reminding us that we should do one) | 00:35 |
| fungi | it was bug-productive, but i didn't get to most of the other stuff i need to do | 00:35 |
| pleia2 | clarkb: sure thing :) | 00:35 |
| fungi | pleia2: awesome job! | 00:35 |
| *** Sukhdev has quit IRC | 00:36 | |
| pleia2 | now I get to go to a hockey game | 00:36 |
| mriedem | jogo: looks like the propose_requirements_update.sh script actually does try to match the change-id to the requirements project: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/propose_requirements_update.sh#n51 | 00:36 |
| StevenK | pleia2: Ice or grass? | 00:36 |
| pleia2 | StevenK: ice, with canadians! | 00:36 |
| *** mgagne has joined #openstack-infra | 00:36 | |
| clarkb | mriedem: thats matching the change id to any existing unmerged requirement updates for that change | 00:37 |
| clarkb | mriedem: this way we push new patchsets not new changes | 00:37 |
| mriedem | clarkb: so what i wanted was to link something like this https://review.openstack.org/#/c/79338/ back to the original global-requirements change | 00:38 |
| mriedem | so we can see the original commit message to know why it's being updated, b/c that's what i do when i review these | 00:38 |
| mriedem | i.e. if it's a bug fix or something | 00:38 |
| jogo | 2014-03-12 00:34:09.832 TRACE nova Stderr: 'Traceback (most recent call last):\n File "/usr/local/bin/nova-rootwrap", line 6, in <module>\n from oslo.rootwrap.cmd import main\nImportError: No module named rootwrap.cmd\n' | 00:38 |
| jogo | is that one of those weird oslo namespace issues? | 00:39 |
| *** thuc has quit IRC | 00:39 | |
| jogo | that was when running grenade locally | 00:39 |
| clarkb | mriedem: you can do that if you can build the state in that script | 00:39 |
| clarkb | mriedem: I would need to think about that more to know how difficult it would be | 00:39 |
| *** thuc has joined #openstack-infra | 00:39 | |
| clarkb | mriedem: so honestly, I think people should be reviewing openstack/requirements if they care at that level | 00:39 |
| clarkb | mriedem: you are already being tested with the requirements in the global requirements. The only reason we havethose files in nova is so that you can pip install nova without an additional step | 00:40 |
| mriedem | clarkb: well and packagers rely on them right? | 00:40 |
| mriedem | i know we do internally, we use that to update rpm specs | 00:40 |
| clarkb | mriedem: maybe, openstack/requirements is largely for the packagers | 00:40 |
| clarkb | they package the stuff in the list then don't have to worry about stuff on a case by case basis | 00:40 |
| mriedem | right | 00:41 |
| *** ociuhandu has quit IRC | 00:41 | |
| clarkb | my point is adding that info to the nova requirements update isn't a bad thing but it is the wrong place to be reviewing that data | 00:41 |
| fungi | mriedem: packagers probably rely on the requirements names listed in each project, but the versions in global-reqs are what's going to be needed if they want to package more than one project for teh same distro | 00:41 |
| clarkb | it needs to be reviewed when proposed to openstack/requirements | 00:41 |
| *** ryanpetrello has joined #openstack-infra | 00:41 | |
| mriedem | fungi: yeah, and that's what we do | 00:41 |
| mriedem | fungi: and so like when you have some projects capped on different versions of sqla/migrate, it gets weird | 00:42 |
| clarkb | pleia2: good point about if debian. do you know how yum works when we add the puppet repos? we should remvoe them from yum too if possible | 00:42 |
| clarkb | mriedem: not really, they are all tested with the same version | 00:42 |
| clarkb | mriedem: which is why openstack/requirements exists | 00:42 |
| fungi | clarkb: well, i suspect he meant it gets weird trying to package them | 00:43 |
| *** mrodden1 is now known as mrodden | 00:43 | |
| fungi | for the same distro | 00:43 |
| mriedem | clarkb: so as of yesterday i know that all projects run against global-requirements, | 00:43 |
| mriedem | fungi: yes | 00:43 |
| *** thuc has quit IRC | 00:43 | |
| mriedem | so is it pip/pbr that allows version caps to be ignored? | 00:43 |
| locke105 | wut? | 00:44 |
| clarkb | mriedem: fungi but it isn't weird | 00:44 |
| clarkb | you just package what oepnstack/requirements says and ignore what nova says because nova is wrong | 00:44 |
| fungi | mriedem: well, not entirely true. what clarkb neglected to mention is that it's just the integration tests which get run reset to the global requirements lists. not individual project unit tests | 00:44 |
| fungi | clarkb: you have to pay attention to the list of dependencies nova declares, just not necessarily the versions of them | 00:44 |
| mriedem | ok, "you just package what oepnstack/requirements says and ignore what nova says because nova is wrong" is news to me | 00:45 |
| fungi | clarkb: you wouldn't package nova and claim it depends on every single package listed in the global requirements | 00:45 |
| clarkb | fungi: right but we aren't talking about that we are talking about 50 different versions of sqlalchemy | 00:45 |
| mriedem | right, like pecan and the other 20 things that ceilometer uses but no one else does, for exapmle | 00:45 |
| mriedem | but i'm getting the point | 00:45 |
| mriedem | and yeah it's what happens anyway, we just end up updating the nova rpm spec using nova's requirements.txt, but the values in there match global requirements more or less anyway | 00:46 |
| *** rms_13 has left #openstack-infra | 00:46 | |
| fungi | clarkb: true... though if nova doesn't keep up to date on global requirements sync, it may run with the versions in global-reqs per our integration tests, but may not be able to pass its own unit tests | 00:46 |
| mriedem | we're just usually a day or so behind b/c someone in nova has to merge those changes | 00:46 |
| fungi | corner case, but i expect it to happen at some point | 00:47 |
| mriedem | so is there value in updating the sync commit message to explain some of this for reviewers? | 00:47 |
| clarkb | fungi: it is certianly possibly | 00:47 |
| clarkb | *possible | 00:47 |
| mriedem | like 'if you're worried about testing, you're too late because it's already tested, this is more or less for downstream CD' | 00:47 |
| clarkb | mriedem: IMO it doesn't hurt, but I also think that if you care about that at that level you need to review openstack/requirements changes instead | 00:47 |
| fungi | openstack proves the infinite monkey theorem on a regular basis | 00:47 |
| mriedem | clarkb: my caring is really only in doing some kind of review on these changes when they show up in nova | 00:48 |
| clarkb | mriedem: I don't want nova to get the impression that they can refuse an update because they don't like the commit from requirements | 00:48 |
| mriedem | *on doing | 00:48 |
| clarkb | it should've been refused in requirements | 00:48 |
| *** thuc has joined #openstack-infra | 00:48 | |
| mriedem | clarkb: no i get that | 00:48 |
| mriedem | not my point | 00:48 |
| * locke105 google inifinite monkey theorem | 00:48 | |
| locke105 | googles* bah | 00:48 |
| mriedem | so i'm sure this has been asked, but then why don't we just auto-merge those changes in the projects if we shouldn't care about them there? | 00:48 |
| *** thuc has quit IRC | 00:48 | |
| *** bhuvan_ has quit IRC | 00:49 | |
| *** bhuvan has quit IRC | 00:49 | |
| fungi | locke105: basically saying that if there is a corner case we can define, we will almost certainly encounter it at some point due to the sheer size of the project | 00:49 |
| mriedem | although if i didn't poke on the sqla nova sync change yesterday, we wouldn't have known this was broken in how things were tested... | 00:49 |
| *** thuc has joined #openstack-infra | 00:49 | |
| clarkb | mriedem: we really don't want jenkins to have merge rights :) | 00:49 |
| clarkb | for code that it proposes | 00:49 |
| mriedem | for when it becomes self-aware | 00:50 |
| fungi | mriedem: it provides some oversight, but it's on the same level as translation proposals | 00:50 |
| clarkb | right now we mostly have that separated out with zuul being the organizer | 00:50 |
| mriedem | yeah, just playing devil's advocate | 00:50 |
| *** ryanpetrello has quit IRC | 00:50 | |
| *** ryanpetrello has joined #openstack-infra | 00:50 | |
| jeblair | it's dangerously close to self-aware; don't give it any more weapons | 00:50 |
| *** Alexandra has quit IRC | 00:50 | |
| *** isviridov has quit IRC | 00:50 | |
| *** derekh has quit IRC | 00:50 | |
| mriedem | alright, well not sure if i have something useful to propose for the sync commit message or not | 00:50 |
| *** ryanpetrello has quit IRC | 00:51 | |
| * locke105 proposes a change: New Jenkins job: gate-become-self-aware | 00:51 | |
| *** CaptTofu has joined #openstack-infra | 00:51 | |
| fungi | mriedem: also keep in mind that the next patchset on a change is going to be updated to include an adjustment from an entirely different global requirements change, so you may not be providing much context unless you never let it get past one patchset on the change | 00:51 |
| fungi | mriedem: or unless you keep rereading and appending to the commit message | 00:52 |
| mriedem | fungi: yeah, i've abandoned that idea | 00:52 |
| fungi | ahh, okay | 00:52 |
| mriedem | i'm thinking more general guidance for reviewers | 00:52 |
| locke105 | what about running nova unit tests against global req changes? | 00:52 |
| fungi | it's probably doable, but i would expect it to get real ugly real quick | 00:52 |
| fungi | locke105: we've got some things going on with oslo.test which may make that viable | 00:53 |
| mriedem | locke105: nova UT/grenade/tempest is run against the nova sync, but not global-requirements | 00:53 |
| fungi | locke105: basically a pattern for changes to one project getting gated on the unit tests of a list of other projects | 00:54 |
| *** gondoi has quit IRC | 00:56 | |
| *** hallyn has quit IRC | 00:56 | |
| *** vishy has quit IRC | 00:56 | |
| *** mugsie has quit IRC | 00:57 | |
| *** zhiyan_ has quit IRC | 00:57 | |
| *** kiall has quit IRC | 00:57 | |
| *** pabelanger has quit IRC | 00:57 | |
| *** StevenK has quit IRC | 00:57 | |
| *** sdake has quit IRC | 00:57 | |
| *** bogdando has quit IRC | 00:57 | |
| *** jraim has quit IRC | 00:57 | |
| *** branen__ has quit IRC | 00:57 | |
| *** mancdaz has quit IRC | 00:57 | |
| *** nijaba_ has quit IRC | 00:57 | |
| *** odyi has quit IRC | 00:58 | |
| *** exz has quit IRC | 00:58 | |
| *** zigo has quit IRC | 00:58 | |
| *** sc68cal has quit IRC | 00:58 | |
| *** fungi has quit IRC | 00:58 | |
| mriedem | if a sync breaks a project, the project fixes themselves i think - that's what happened when sqla-migrate 0.8.2 was released i believe and synced to nova | 00:58 |
| locke105 | sure but we have a downstream job that is similar | 00:58 |
| *** isviridov has joined #openstack-infra | 00:58 | |
| *** ryanpetrello has joined #openstack-infra | 00:58 | |
| mriedem | nova had some sqla-migrate version capped code in it that needed to be removed | 00:58 |
| locke105 | some extension gets updated, jenkins pulls in nova, pulls in latest extension, runs tests, reports back on extension change | 00:58 |
| openstackgerrit | Sean Dague proposed a change to openstack/requirements: allow sqla 0.9.x https://review.openstack.org/79817 | 00:58 |
| sdague | mriedem: oh.... that kind of typo.... | 00:58 |
| * sdague facepalms | 00:58 | |
| sdague | jeblair: we should only give it cupcakes | 00:58 |
| sdague | then it will spare us in the revolution | 00:58 |
| mriedem | sdague: yeah, i wasn't sure what you were saying earlier | 00:58 |
| sdague | mriedem: I assumed you were dinging me for a commit message | 00:58 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Remove unneeded package sources from nodepool node https://review.openstack.org/79814 | 00:58 |
| sdague | because I mess those up plenty | 00:58 |
| mriedem | sdague: i assumed you were way over my head on something | 00:58 |
| clarkb | pleia2: ^ I think that fixes the problem you pointed out | 00:58 |
| *** alexandra_ has joined #openstack-infra | 00:58 | |
| *** branen_ has joined #openstack-infra | 00:58 | |
| clarkb | pleia2: I looked at a centos machine but figure all redhat ish distros use yum.repos.d since they use yum | 00:58 |
| sdague | so yeh, bad rebase, which I totally didn't resolve right | 00:58 |
| sdague | anyway, it's right now | 00:58 |
| mriedem | yeah, +1 | 00:58 |
| *** mugsie_ has joined #openstack-infra | 00:58 | |
| *** CaptTofu has quit IRC | 00:58 | |
| sdague | now we just need to figure out if it works | 00:58 |
| sdague | also... if it runs the right versions | 00:58 |
| * mriedem ponders signing up for all new openstack/requirements patches | 00:58 | |
| *** zhiyan` has joined #openstack-infra | 00:58 | |
| *** jgriffith has quit IRC | 00:58 | |
| *** akscram has quit IRC | 00:58 | |
| *** GheRivero has quit IRC | 00:58 | |
| *** ProfFalken has quit IRC | 00:58 | |
| *** nijaba has joined #openstack-infra | 00:58 | |
| sdague | it's pretty light load | 00:58 |
| mriedem | you've convinced me | 00:58 |
| mriedem | i have to deal with these things when they merge anyway so i might as well know they are coming | 00:58 |
| *** VijayTripathi1 has quit IRC | 00:58 | |
| *** sc68cal_ has joined #openstack-infra | 00:58 | |
| dims_ | gah! netslpit | 00:58 |
| *** vish1 has joined #openstack-infra | 00:58 | |
| *** vish1 is now known as vishy | 00:58 | |
| *** GheRivero has joined #openstack-infra | 00:58 | |
| *** fungi has joined #openstack-infra | 00:58 | |
| *** jraim has joined #openstack-infra | 00:58 | |
| *** thuc has quit IRC | 00:59 | |
| *** sileht has quit IRC | 00:59 | |
| *** openstack has joined #openstack-infra | 01:05 | |
| *** zns has quit IRC | 01:05 | |
| *** mattymo has joined #openstack-infra | 01:05 | |
| jogo | sdague: so I am a little concerned that we have 100% nodepool utilization most ofthe time | 01:05 |
| fungi | jeblair: heh, great... i'll be done cleaning that up by lunch time if i'm lucky ;) | 01:05 |
| jogo | it slows down checks and whatnot | 01:05 |
| jogo | the concern is purely about velocity | 01:06 |
| jeblair | jogo: you could contribute to the effort to move into hpcloud 1.1 | 01:06 |
| jeblair | jogo: where apparently hpcloud has more resources | 01:07 |
| jeblair | also, there are lots of public openstack clouds we aren't using | 01:08 |
| *** Daisy_ has joined #openstack-infra | 01:08 | |
| jeblair | people have said they are interested in contributing clouds, but we've seen very little followup | 01:08 |
| jogo | jeblair: sure, although I question the value of the 24 recheck (when not before gate) ... I wonder what 36 or 48 hours would look like | 01:09 |
| jeblair | jogo: it's 72 hours when not approved (24 after approved) | 01:10 |
| jogo | jeblair: ohh in that case ... | 01:10 |
| jogo | having to wait an extra hour for check queue definitly slows things down | 01:11 |
| jogo | I do like the get more resoures solution in the long run | 01:11 |
| jeblair | jogo: i think we should do all the things. get more resources, be smarter about the ones we have (sdague is working on that as you know) | 01:12 |
| *** jnoller has joined #openstack-infra | 01:13 | |
| *** ryanpetrello has quit IRC | 01:13 | |
| clarkb | I think rechecking was crucial to gate health during feature freeze | 01:13 |
| clarkb | it was so smooth compared to havana | 01:14 |
| jogo | clarkb: while I would like to agree with you, I don't think we really have enough data to say *what* helped -- too many changes | 01:14 |
| jeblair | we can also do mean things like fast-fail (abort jobs on the first failure), or hang everything off of a group of jobs (people suggest pep8, but i also think requirements check failures should get the same treatment) | 01:15 |
| SpamapS | Yeah, could just be Neutron sucking less. ;) | 01:15 |
| jeblair | those would be zuul changes | 01:15 |
| *** weiwei has quit IRC | 01:15 | |
| *** jhesketh_ has quit IRC | 01:15 | |
| jogo | so I am not sure how much of the resource issue is gate jobs vs check jobs | 01:15 |
| jeblair | also, if developers are impatient... they _could_ run tests locally before uploading :) | 01:15 |
| jogo | my hunch is we now just have tons of check jobs (for several reasons) | 01:16 |
| jogo | and not enough nodes | 01:16 |
| clarkb | check most == gate | 01:16 |
| *** gokrokve has quit IRC | 01:17 | |
| clarkb | jeblair ++ | 01:17 |
| jogo | clarkb: ? | 01:17 |
| fungi | i still think we don't allow the gate window floor to dip low enough when there's high volume and a lot of gate resets going on, wasting resources over and over while more or less starving out the check pipeline for a while (which slows new changes entering gating too) | 01:17 |
| sdague | jeblair: we need that as the new openstack hoodie "we should do all the things" | 01:17 |
| jogo | fungi: interesting idea | 01:17 |
| jeblair | fungi: really? i think 20 is too low but a good compromise | 01:18 |
| jogo | sdague: haha | 01:18 |
| locke105 | sdague: ++ would buy | 01:18 |
| jogo | sdague: quick grenade question | 01:18 |
| jogo | how do I run locally from havna to trunk? | 01:18 |
| jeblair | fungi: considering how important the gate is, i think it's a tiny amount of resources... :) | 01:18 |
| jogo | stock doesn't seem to do that | 01:18 |
| sdague | jogo: stock should | 01:18 |
| jogo | sdague: unless I missed a step (which I bet I did) | 01:18 |
| sdague | but thingee said he was running into an issue the other day as well | 01:19 |
| *** krotscheck has quit IRC | 01:19 | |
| fungi | jeblair: well, granted the gate is important, but we've also effectively made the check pipeline part of the gate now | 01:19 |
| sdague | so.... I should verify tomorrow | 01:19 |
| *** mkerrin has joined #openstack-infra | 01:19 | |
| *** talluri has joined #openstack-infra | 01:19 | |
| *** zhiwei has joined #openstack-infra | 01:19 | |
| sdague | fungi: and I'd be -1 on a floor < 20. Because we just as often as reset, complete all the work in the 20 nodes except for on slow job on the tip | 01:20 |
| sdague | 20 jobs ... | 01:20 |
| fungi | 20 seems plenty low if we're merging 5-10 changes before a reset, but today the job failures in the gate pipeline were a good bit more plentiful than that | 01:20 |
| jeblair | fungi: i would be okay with a more _accurate_ heuristic | 01:20 |
| fungi | as would i | 01:20 |
| jeblair | fungi: but with the current model, i think going below 20 would hurt us more than help | 01:21 |
| sdague | yeh, we'd starve | 01:21 |
| fungi | mmm, perhaps so | 01:21 |
| jeblair | fungi, clarkb: i think we can make a more accurate heuristic too... | 01:21 |
| sdague | and, honestly, I think it's a low item to optimize | 01:21 |
| sdague | because the floor was about not thrashing | 01:21 |
| sdague | not really about waste | 01:21 |
| sdague | and we've managed to stop thrashing | 01:21 |
| *** tteggel has quit IRC | 01:22 | |
| sdague | which is goodness | 01:22 |
| fungi | we're not thrashing hard, but we are getting to a point where we're maxed out on quotas waiting for deletes | 01:22 |
| sdague | fungi: sure | 01:22 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/requirements: Update the keyring dependency https://review.openstack.org/79796 | 01:22 |
| sdague | the other option would be put stackforge into a lower priority queue | 01:22 |
| fungi | but we seem to be riding high enough that the deletes aren't hurting us too bad either, so maybe i'm just over-thinking the current tuning | 01:23 |
| sdague | yeh, I think we're actually ok there | 01:23 |
| *** tteggel has joined #openstack-infra | 01:23 | |
| jeblair | openstack could make deletes faster. ;) | 01:24 |
| sdague | from a development optimization perspective, I think the cross project dependencies in zuul is going to be something important to get in zuul | 01:24 |
| sdague | in juno | 01:24 |
| mordred | I still think we shouldn't do that | 01:24 |
| fungi | it looks like at any one point in time we have roughly 300 nodes in use running jobs, the rest are building or deleting | 01:24 |
| sdague | mordred: did you see how the event patch to fix neutron / nova interaction had to get tested? | 01:24 |
| *** gokrokve has joined #openstack-infra | 01:25 | |
| jeblair | sdague, mordred: i'm okay with the feature, though i do think it's worth considering how it relates to CD | 01:25 |
| mordred | it's the CD thing I'm worried about | 01:25 |
| sdague | with 10+ integrated projects, having to test changes in multiple projects is important | 01:25 |
| sdague | mordred: so I'm not saying we land the changes atomically | 01:25 |
| mordred | I don't think we should land it until we've got somethign that tests rolling upgrades | 01:25 |
| sdague | I'm saying we can define ordering | 01:26 |
| mordred | sdague: oh ... you're saing | 01:26 |
| mordred | ok | 01:26 |
| mordred | yeah | 01:26 |
| sdague | yeh, never atomic lands | 01:26 |
| jeblair | mordred, sdague: i think sdague is describing the one-way variant of cross-project dependencies | 01:26 |
| mordred | like, "I need this unlanded change in neutron to land before testing this unlanded change in nova" | 01:26 |
| mordred | yeah | 01:26 |
| sdague | yes | 01:26 |
| sdague | that | 01:26 |
| mordred | that I think is a great idea | 01:26 |
| jeblair | a much easier problem than the full-cycle variant | 01:26 |
| sdague | that I think is going to be critical with 10+ integrated projects | 01:27 |
| *** zigo has quit IRC | 01:27 | |
| locke105 | CD =? | 01:27 |
| jeblair | locke105: continuous deployment | 01:27 |
| locke105 | by that you mean devstack-gate? | 01:27 |
| jeblair | locke105: (deploy every comment, or as near as you can make it) | 01:27 |
| sdague | otherwise we end up with "oh, land this neutron change, guess it's the right one, wait until it's landed, then start working on the nova change" | 01:27 |
| clarkb | why is atomic required? | 01:27 |
| jeblair | locke105: it's specifically referring to operators, though the devstack-gate process is designed to make it easier for them | 01:27 |
| jeblair | clarkb: it's not | 01:28 |
| clarkb | ok didnt think it was | 01:28 |
| mordred | clarkb: in fact, it's not desired | 01:28 |
| *** jhesketh_ has joined #openstack-infra | 01:28 | |
| mordred | the coordinated change dependency version requires atomic landing | 01:28 |
| *** weiwei has joined #openstack-infra | 01:28 | |
| sdague | clarkb: no, I don't want it | 01:28 |
| jeblair | clarkb: that's only required for full-cycle cross-repo deps, which are considered to be hazardous to CD and possibly upgrade testing | 01:28 |
| sdague | I think atomic is bad | 01:28 |
| sdague | I agreed with mordred on that | 01:28 |
| mordred | sdague: I figured we proably agreed | 01:28 |
| sdague | he did't realize that :) | 01:28 |
| locke105 | oh so CD isn't something that infra does... its just consideration of how operators would be live consuming and deploying openstack changes | 01:29 |
| locke105 | ? | 01:29 |
| sdague | this other thing, we really do need | 01:29 |
| jeblair | locke105: correct | 01:29 |
| jeblair | locke105: (though we actually do cd some of our own tools) | 01:29 |
| jeblair | locke105: (but that's beside the point) | 01:29 |
| locke105 | like gerrit-dev | 01:29 |
| *** yaguang has joined #openstack-infra | 01:29 | |
| locke105 | err | 01:29 |
| locke105 | review-dev | 01:29 |
| jeblair | locke105: trying to get there with gerrit | 01:29 |
| sdague | jeblair: so that I might hit you up about post release to hack on in zuul | 01:30 |
| sdague | just to figure out what's needed and see if I could take a first pass | 01:30 |
| *** reed has quit IRC | 01:30 | |
| jeblair | sdague: sure thing; i think many of the bits are there. i think the main design review thing is how to specify the c-r deps; i think we should look at the defunct gerrit patch to see their syntax and possibly adopt it | 01:31 |
| sdague | jeblair: I was assuming we'd use an Idempotent id | 01:31 |
| jeblair | sdague: i think that may be what they did. | 01:32 |
| sdague | cool | 01:32 |
| clarkb | it would be goid to make it not gerrit specific | 01:32 |
| sdague | yeh, honestly, I've been doing Depends-On: Idempotent id | 01:32 |
| locke105 | if they merge in-order, and if its shortly one after another, whats stopping the CD framework from say, picking up both the nova and neutron update and not realizing they need to be done in-order? | 01:32 |
| sdague | just manually for signaling in the past | 01:32 |
| sdague | clarkb: it needs to be an idempotent id | 01:32 |
| *** salv-orlando_ has joined #openstack-infra | 01:32 | |
| jeblair | sdague: but istr something about using the change-id and s/I/D/... but i don't recall all details. main thing is -- a lot of thinking has already been done there | 01:32 |
| sdague | otherwise it breaks on rebasing | 01:32 |
| *** zigo has joined #openstack-infra | 01:33 | |
| sdague | coolio, if you have pointers, would appreciate them | 01:33 |
| jeblair | clarkb: it's necessarily not gerrit-specific because it's not in gerrit. :) | 01:33 |
| jeblair | sdague: not off hand, but i should be able to find the patch series when it's time | 01:33 |
| *** dstanek_afk is now known as dstanek | 01:34 | |
| jeblair | i need to run now, ttyl | 01:34 |
| clarkb | whats wromg with change id + project? | 01:34 |
| clarkb | oh branches | 01:34 |
| *** salv-orlando has quit IRC | 01:37 | |
| *** salv-orlando_ is now known as salv-orlando | 01:37 | |
| clarkb | change id's may not be best because gerrit pairs them with project:branch to get the unique key | 01:37 |
| *** gokrokve has quit IRC | 01:38 | |
| *** nosnos has joined #openstack-infra | 01:39 | |
| clarkb | sdague: https://gerrit-review.googlesource.com/#/q/topic:cross-repo-dependencies is the code jeblair is talking about I think | 01:39 |
| mordred | sdague: so - it doesn't affect the gate per-se ... but why is n-novnc enabled by default in devstack? | 01:40 |
| fungi | okay, the project plan for tomorrow morning's maintenance is here for anyone who wants to give it a once-over... https://etherpad.openstack.org/p/project-rename-2014-03-12 | 01:40 |
| mordred | sdague: we do not have a project option for it | 01:40 |
| clarkb | fungi: looking | 01:40 |
| sdague | mordred: for raisons? | 01:42 |
| mordred | sdague: awesome | 01:42 |
| fungi | clarkb: also note that SergeyLukjanov's more details series of savanna-specific stuff is here... https://etherpad.openstack.org/p/savanna-renaming-process | 01:42 |
| sdague | honestly, that so predates me, I have no idea | 01:42 |
| sdague | could be linear-b | 01:43 |
| clarkb | mordred: probably because people like vnc consoles by default? | 01:43 |
| mordred | clarkb: for their devstack working env? | 01:43 |
| clarkb | mordred: ya | 01:43 |
| mordred | hrm | 01:43 |
| clarkb | fungi: plan looks mostly good to me, couple of notes | 01:44 |
| sdague | I thought we had working spice now though | 01:44 |
| clarkb | mordred: remember devstack is the most common deployment method :) | 01:44 |
| mordred | well, just pointing out, spurred by looking at it again because of the mailing list thread - that it's the only thing we clone from github with no other optino | 01:44 |
| mordred | option | 01:44 |
| *** gokrokve has joined #openstack-infra | 01:44 | |
| clarkb | mordred: there is an option | 01:44 |
| clarkb | you can override the location, seem my response to the thread | 01:45 |
| sdague | anyway, time to call it a night. See you guys in the morning. | 01:45 |
| mordred | sdague: have fun! | 01:45 |
| clarkb | mordred: I mean its not necessarily a great option but you can clone from elsewhere | 01:47 |
| clarkb | for most elsewhere is probably a fork of the github project on github though :) | 01:48 |
| *** sarob has joined #openstack-infra | 01:48 | |
| fungi | clarkb: suggestions integrated--thanks! | 01:48 |
| fungi | SergeyLukjanov: when you wake up, you'll probably want to review the project plan for the bits i'm doing at 12:00 utc (except step 14, which is up to you for your projects) | 01:54 |
| *** khyati_ has quit IRC | 01:56 | |
| *** arosen has joined #openstack-infra | 01:57 | |
| *** gokrokve has quit IRC | 01:58 | |
| *** Ryan_Lane has quit IRC | 01:58 | |
| *** matsuhashi has quit IRC | 01:58 | |
| *** gokrokve has joined #openstack-infra | 02:01 | |
| *** changbl has joined #openstack-infra | 02:02 | |
| *** dkliban has joined #openstack-infra | 02:06 | |
| *** nicedice has joined #openstack-infra | 02:06 | |
| *** vkozhukalov has joined #openstack-infra | 02:07 | |
| *** talluri has quit IRC | 02:07 | |
| *** arosen is now known as arosen-home | 02:08 | |
| *** nicedice_ has quit IRC | 02:08 | |
| *** thuc has joined #openstack-infra | 02:09 | |
| *** sarob has quit IRC | 02:11 | |
| *** sarob has joined #openstack-infra | 02:12 | |
| *** HenryG has joined #openstack-infra | 02:12 | |
| *** gokrokve has quit IRC | 02:14 | |
| *** thuc has quit IRC | 02:14 | |
| *** gokrokve has joined #openstack-infra | 02:15 | |
| *** sarob has quit IRC | 02:16 | |
| fungi | SergeyLukjanov: oh, the link would probably help... https://etherpad.openstack.org/p/project-rename-2014-03-12 | 02:16 |
| fungi | anyway, that's enough fun for me for the night. time to get ready for a nap | 02:16 |
| *** matsuhashi has joined #openstack-infra | 02:17 | |
| zhiwei | thank you very much for your work. | 02:17 |
| fungi | zhiwei: you're welcome! don't thank me until it's done though ;) | 02:18 |
| zhiwei | :) | 02:18 |
| *** gokrokve has quit IRC | 02:19 | |
| *** zigo has quit IRC | 02:19 | |
| *** bhuvan has joined #openstack-infra | 02:21 | |
| *** bhuvan_ has joined #openstack-infra | 02:21 | |
| *** sweston has joined #openstack-infra | 02:23 | |
| *** zigo has joined #openstack-infra | 02:23 | |
| *** caleb_ has quit IRC | 02:24 | |
| *** khyati has joined #openstack-infra | 02:27 | |
| *** VijayTripathi has joined #openstack-infra | 02:31 | |
| *** rcarrillocruz has joined #openstack-infra | 02:32 | |
| *** vkozhukalov has quit IRC | 02:36 | |
| *** rcarrillocruz has quit IRC | 02:37 | |
| *** rpodolyaka has quit IRC | 02:38 | |
| *** harlowja is now known as harlowja_away | 02:48 | |
| *** sarob has joined #openstack-infra | 02:48 | |
| *** ryanpetrello has joined #openstack-infra | 02:49 | |
| *** CaptTofu has joined #openstack-infra | 02:52 | |
| *** jepoy has joined #openstack-infra | 02:54 | |
| *** gyee has quit IRC | 02:54 | |
| *** CaptTofu has quit IRC | 02:57 | |
| *** akscram has joined #openstack-infra | 02:57 | |
| *** amcrn has quit IRC | 03:05 | |
| *** talluri has joined #openstack-infra | 03:09 | |
| *** david-lyle has joined #openstack-infra | 03:09 | |
| *** dcramer__ has joined #openstack-infra | 03:12 | |
| *** mriedem has quit IRC | 03:15 | |
| *** rwsu has quit IRC | 03:19 | |
| *** sarob has quit IRC | 03:21 | |
| *** pcrews has quit IRC | 03:21 | |
| *** CaptTofu has joined #openstack-infra | 03:23 | |
| *** matsuhashi has quit IRC | 03:23 | |
| *** bhuvan_ has quit IRC | 03:25 | |
| *** bhuvan has quit IRC | 03:25 | |
| *** dkliban has quit IRC | 03:28 | |
| clarkb | fungi: https://developer.github.com/v3/oauth/ we need to add the read:org scope to our token. this may require a new token | 03:29 |
| clarkb | fungi: turns out I am in a channel with someone that contributes to pygithub and they were able to help. seriously though why have api versions if when you change stuff like this you don't create a new version | 03:29 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add zuul overlay to review page https://review.openstack.org/79836 | 03:31 |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add zuul overlay to review page https://review.openstack.org/79836 | 03:32 |
| *** SumitNaiksatam has joined #openstack-infra | 03:32 | |
| *** jnoller has quit IRC | 03:33 | |
| *** jnoller has joined #openstack-infra | 03:33 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add zuul overlay to review page https://review.openstack.org/79836 | 03:33 |
| jhesketh__ | jeblair, clarkb: ^ what do you think of this for an improvement to gerrit? | 03:34 |
| jhesketh__ | (more or less proof of concept at this point) | 03:34 |
| *** wchrisj has joined #openstack-infra | 03:37 | |
| clarkb | jhesketh__: I am not the greatest person to review css and javascript, but I am super willing to throw that on review-dev and give it a spin | 03:37 |
| *** sarob has joined #openstack-infra | 03:37 | |
| clarkb | I think it is a great idea | 03:37 |
| *** reed has joined #openstack-infra | 03:37 | |
| jhesketh__ | clarkb: heh, I'm not the greatest person at writing javascript/css! | 03:37 |
| jhesketh__ | particularly the css part | 03:37 |
| clarkb | its basically zuul status in the change for that change right? | 03:37 |
| jhesketh__ | yep | 03:37 |
| clarkb | then yes I think we should put this on review-dev asap :) | 03:38 |
| jhesketh__ | woo :-) | 03:38 |
| clarkb | I am currently taste testing a couple IPAs though so will have to wait for tomorrow before I can help :) | 03:38 |
| arosen-home | one change i'd like to make is to add the patch owner to that page by the id. I find myself having to look up the patch id and search the zuul page over and over when i only care about my patches :) | 03:39 |
| jhesketh__ | clarkb: mmm, nice | 03:39 |
| jhesketh__ | enjoy! :-) | 03:39 |
| jhesketh__ | arosen-home: you mean to the zuul status page? | 03:39 |
| arosen-home | jhesketh_ i mean here: http://status.openstack.org/zuul/ | 03:40 |
| *** oda-g_ has joined #openstack-infra | 03:40 | |
| jhesketh__ | arosen-home: sure. I would want to get clarkb's and jeblair's opinion on that. I'd be cautious of not adding too much clutter. The change I proposed embeds those boxes on the associated patches. So to see the status you'd just have to open your patches. | 03:41 |
| jhesketh__ | (in gerrit that is) | 03:41 |
| mattoliverau | jhesketh__: awesome idea :) | 03:42 |
| *** sarob__ has joined #openstack-infra | 03:43 | |
| *** sarob has quit IRC | 03:44 | |
| *** sarob__ has quit IRC | 03:45 | |
| *** sarob has joined #openstack-infra | 03:46 | |
| *** lcheng has joined #openstack-infra | 03:48 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add pypy jobs to pecan. https://review.openstack.org/79839 | 03:50 |
| *** rwsu has joined #openstack-infra | 03:50 | |
| openstackgerrit | Ryan Petrello proposed a change to openstack-infra/config: Add pypy jobs to pecan. https://review.openstack.org/79839 | 03:50 |
| *** fifieldt has joined #openstack-infra | 03:51 | |
| *** Ryan_Lane has joined #openstack-infra | 03:53 | |
| *** dstanek has quit IRC | 03:54 | |
| *** sileht has quit IRC | 03:56 | |
| *** jnoller_ has joined #openstack-infra | 03:56 | |
| clarkb | dhellmann: the oslo.test chagnes lgtm | 03:56 |
| *** jnoller has quit IRC | 03:56 | |
| *** Daisy_ has quit IRC | 03:57 | |
| *** sileht has joined #openstack-infra | 03:58 | |
| dstufft | mordred: I almost have pip taking control of setup_requires | 04:00 |
| *** SumitNaiksatam has quit IRC | 04:01 | |
| dstufft | scratch that | 04:01 |
| dstufft | I do have it | 04:01 |
| dstufft | at least a working PoC | 04:01 |
| *** zhiyan` is now known as zhiyan | 04:02 | |
| *** SumitNaiksatam has joined #openstack-infra | 04:02 | |
| *** dstanek has joined #openstack-infra | 04:02 | |
| *** gokrokve has joined #openstack-infra | 04:05 | |
| *** markmcclain has quit IRC | 04:05 | |
| clarkb | dstufft: that half does something pbr did do and half makes it safer to install pbr yeah? | 04:05 |
| clarkb | slowly one day pbr will die and I will be happy | 04:05 |
| *** thuc has joined #openstack-infra | 04:06 | |
| dstufft | clarkb: well even if you ``pip install foo``, if foo uses setup_requires in it's setup.py, then setuptools will be used to download and install wahtever is in the setup_requires | 04:06 |
| dstufft | my PoC uses pip to do that instead | 04:07 |
| *** thuc_ has joined #openstack-infra | 04:07 | |
| dstufft | so that if you're using pip, setuptools should never touch the network | 04:07 |
| dstufft | it's horrible though | 04:07 |
| dstufft | I'm monkeypatching distutils (which will then be monkeypatched by setuptools) inside of a python -c "" which reads the setup.py file and evals it | 04:08 |
| dstufft | https://github.com/dstufft/pip/compare/pypa:develop...dstufft:eldritch-horror | 04:08 |
| *** sarob has quit IRC | 04:08 | |
| clarkb | wow | 04:08 |
| clarkb | appropriate branch name | 04:08 |
| *** sarob has joined #openstack-infra | 04:09 | |
| dstufft | mostly this means there won't be a second place to make settings for setup_requires | 04:10 |
| *** thuc has quit IRC | 04:10 | |
| clarkb | mikal: I want you to know that if someone were to read your twitter stuff they would probably think you run an ISP | 04:12 |
| clarkb | mikal: I am sure you would run a great isp | 04:12 |
| *** sarob has quit IRC | 04:13 | |
| *** wchrisj has quit IRC | 04:15 | |
| *** jepoy_ has joined #openstack-infra | 04:16 | |
| *** jepoy has quit IRC | 04:18 | |
| mordred | dstufft: nice! | 04:22 |
| *** talluri has quit IRC | 04:23 | |
| *** CaptTofu has quit IRC | 04:24 | |
| *** Daisy_ has joined #openstack-infra | 04:25 | |
| *** matsuhashi has joined #openstack-infra | 04:25 | |
| *** psedlak has quit IRC | 04:28 | |
| *** jepoy_ has quit IRC | 04:28 | |
| *** nati_ueno has quit IRC | 04:28 | |
| *** pcrews has joined #openstack-infra | 04:29 | |
| *** psedlak has joined #openstack-infra | 04:31 | |
| *** arosen-home has quit IRC | 04:33 | |
| *** jnoller_ has quit IRC | 04:34 | |
| *** pcrews has quit IRC | 04:34 | |
| *** jnoller has joined #openstack-infra | 04:37 | |
| *** ryanpetrello has quit IRC | 04:37 | |
| lifeless | mordred: btw, if you have a few cycles, a patch to testrepository to use pbr (assume that i have a candidate git tree) would be awesome | 04:39 |
| mordred | lifeless: sure thing | 04:40 |
| mordred | lifeless: (do you have a candidate git tree somewhere I can clone it?) | 04:41 |
| lifeless | mordred: haven't pushed it anywhere yet, will do so | 04:41 |
| *** alexandra_ is now known as alex-luncheon | 04:42 | |
| lifeless | mordred: https://github.com/testing-cabal/testrepository | 04:44 |
| lifeless | mordred: basically I need to make sure sdist does the right thing in a git tree, and in a non-git tree, including sucking in README and version handling, like it did in our old bespoke code. | 04:44 |
| lifeless | mordred: then I can do a release and make folk happy | 04:44 |
| mordred | lifeless: can you tell me what you want from version handling? | 04:46 |
| mordred | lifeless: like - how much like openstack things do you want it - vs. how much like what your current code does? | 04:47 |
| lifeless | mordred: current codebase is semver with a special doohicky for unreleased. | 04:47 |
| lifeless | mordred: what openstack does for that is just fine | 04:47 |
| mordred | ok. cool | 04:47 |
| mordred | that's less work | 04:48 |
| *** VijayTripathi has quit IRC | 04:48 | |
| lifeless | yeah, that bit didn't concern me | 04:48 |
| lifeless | its the different doc meta etc handling that I was unsure where to poke at. | 04:48 |
| lifeless | I'm fine in principle with moving other files around to line up with pbr's expectations | 04:48 |
| mordred | k - you look mostsly fine | 04:48 |
| lifeless | subject to the right to go OMG no on a specific patch :> | 04:48 |
| mordred | lemme make patch | 04:48 |
| *** talluri has joined #openstack-infra | 04:49 | |
| *** talluri has quit IRC | 04:53 | |
| *** jepoy has joined #openstack-infra | 04:55 | |
| *** jepoy has quit IRC | 04:56 | |
| *** thuc_ has quit IRC | 04:57 | |
| *** thuc has joined #openstack-infra | 04:58 | |
| mikal | clarkb: no, just filled with rage at the govt | 04:59 |
| *** sarob has joined #openstack-infra | 05:00 | |
| *** jnoller has quit IRC | 05:00 | |
| *** xchu_ has joined #openstack-infra | 05:01 | |
| *** thuc has quit IRC | 05:02 | |
| mordred | mikal: what's the difference? | 05:05 |
| *** talluri has joined #openstack-infra | 05:05 | |
| *** nicedice has quit IRC | 05:05 | |
| *** nicedice has joined #openstack-infra | 05:06 | |
| *** talluri has quit IRC | 05:06 | |
| *** talluri has joined #openstack-infra | 05:07 | |
| *** Daisy__ has joined #openstack-infra | 05:09 | |
| mordred | lifeless: your tags aren't signed, just fyi | 05:09 |
| *** Daisy_ has quit IRC | 05:09 | |
| mordred | lifeless: I do not know how much you care | 05:09 |
| *** mgagne has quit IRC | 05:09 | |
| mordred | lifeless: but version creation will be weird until you have one signed tag | 05:09 |
| mordred | lifeless: https://github.com/testing-cabal/testrepository/pull/1 | 05:11 |
| *** talluri has quit IRC | 05:11 | |
| *** SumitNaiksatam has quit IRC | 05:11 | |
| *** jepoy has joined #openstack-infra | 05:12 | |
| StevenK | mordred: Which is partly bong | 05:13 |
| StevenK | +# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. | 05:14 |
| StevenK | -# Copyright (c) 2009-2013 Testrepository Contributors | 05:14 |
| mordred | StevenK: what' partly bong? what did I miss? | 05:14 |
| StevenK | In setup.py | 05:14 |
| mordred | StevenK: yeah | 05:14 |
| mordred | it's a completely different file | 05:14 |
| mordred | StevenK: I'm pretty sure that "Testrepository Contributors" don't have any copyright in the file | 05:15 |
| StevenK | mordred: Well, your PR makes you a contributor, so ... :-) | 05:15 |
| mordred | ah. well. now I get to decide if I'm going to disagree with this read on how the Berne Convention works - or just say screw it and go along | 05:16 |
| clarkb | mordred: honesly that file is so small you could probably argue it isn't copyrightable :P | 05:17 |
| mordred | clarkb: that's the better argument, tbh | 05:17 |
| StevenK | mordred: It's up to lifeless. Not sure if I even have a valid opinion, since I can't recall if I have any code in testr | 05:17 |
| mordred | to me - Copyright (c) 2009-2013 Testrepository Contributors implies that "Testrepository Contributors" have copyright over this particular work, which, if it is copyrightable in the first place, is invalid | 05:18 |
| mordred | as neither I nor HP have granted any copyright to any of the other contributors | 05:18 |
| mordred | but tbh - clarkb's opinion makes this much easier | 05:18 |
| *** markwash has joined #openstack-infra | 05:20 | |
| mordred | StevenK: fixed | 05:21 |
| mordred | StevenK, lifeless: actually - the history in that branch was silly - re-submitted https://github.com/testing-cabal/testrepository/pull/2 | 05:23 |
| *** markwash has quit IRC | 05:24 | |
| *** markwash_ has joined #openstack-infra | 05:24 | |
| *** xchu_ has quit IRC | 05:26 | |
| jhesketh__ | Hey guys, looks like zuul is having trouble communicating to gerrit: http://status.openstack.org/zuul/ | 05:26 |
| jhesketh__ | ah, ignore me, just a delay | 05:26 |
| lifeless | mordred: btw you can push --overwrite ;) | 05:27 |
| lifeless | erm | 05:27 |
| lifeless | -f | 05:27 |
| lifeless | mordred: on github | 05:27 |
| lifeless | mordred: are | 05:28 |
| lifeless | +[files] | 05:28 |
| lifeless | +scripts = | 05:28 |
| lifeless | +[files] | 05:28 |
| lifeless | bah | 05:28 |
| lifeless | is that needed? | 05:29 |
| lifeless | oh hmm, tox. I think I'll disect this patch a little, since tox isn't a thing in this context. | 05:29 |
| mordred | ok | 05:30 |
| mordred | feel free - the bits you need should be there- feel free to do whatever you want | 05:30 |
| lifeless | thanks | 05:30 |
| *** rpodolyaka has joined #openstack-infra | 05:31 | |
| mordred | lifeless: the [files] scriptts = is how you do the equiv of scripts=['testr'], from the old setup.py | 05:31 |
| lifeless | on the (C) thing, see COPYING; e.g. HP was fine (its NAME) | 05:31 |
| *** oda-g_ has quit IRC | 05:31 | |
| lifeless | and its simpler to avoid confusion in lawyers to have it, so I'll reinstate the HP (C) you had | 05:31 |
| *** sarob has quit IRC | 05:32 | |
| mordred | lifeless: ok. works for me | 05:32 |
| *** reed has quit IRC | 05:32 | |
| mordred | lifeless: also- I checked python setup.py sdist and python setup.py build_sphinx - I did not check other potential uses | 05:34 |
| lifeless | that + pip install is all there is | 05:35 |
| lifeless | since setup.py test is so awful | 05:35 |
| *** nicedice has quit IRC | 05:35 | |
| lifeless | but thats a different rant | 05:35 |
| mordred | yah | 05:35 |
| *** alex-luncheon is now known as Alexandra | 05:35 | |
| lifeless | mordred: you have no problem with BSD + Apache2 I presume | 05:36 |
| mordred | lifeless: I do not | 05:36 |
| *** SumitNaiksatam has joined #openstack-infra | 05:36 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Add zuul overlay to review page https://review.openstack.org/79836 | 05:37 |
| mordred | I also really am not sure there is much that's truly copyrightable in that patch I just sent you | 05:37 |
| lifeless | mordred: I know | 05:37 |
| lifeless | mordred: its pro forma | 05:37 |
| lifeless | mordred: since e.g. debian gets weird about shit, and I can't be bothered .... | 05:38 |
| mordred | yah | 05:38 |
| mordred | lifeless: ok. I'm off to bed - as I said - my patch is your patch - i hope it's helpful - if it's not, you know, no ego attachment to any of it here | 05:39 |
| lifeless | mordred: its super helpful | 05:39 |
| mordred | awesome | 05:39 |
| lifeless | mordred: I'm just doing qa of it | 05:39 |
| lifeless | mordred: looking for surprises, things to document, etc | 05:39 |
| * mordred does a little jig | 05:39 | |
| lifeless | mordred: also tweaking the commit a little :) | 05:39 |
| mordred | oh - I did not change version consumption in testrepository/__init__.py at all | 05:40 |
| mordred | I figure using pbr.version is simple enough if you want to do that - so I left it alone | 05:40 |
| lifeless | mordred: signed tags - any version, or the most recent? | 05:40 |
| mordred | most recent is the most important | 05:40 |
| lifeless | mordred: I presume e.g. nova has an example of pbr.version use ? | 05:40 |
| mordred | the calculated version comes from latest signed tags | 05:40 |
| mordred | yup | 05:40 |
| mordred | actually, nova is a bad example - they're weird... | 05:41 |
| lifeless | version_info = pbr.version.VersionInfo('nova') | 05:41 |
| mordred | yah | 05:41 |
| lifeless | ok, what's a good example :) | 05:41 |
| mordred | nah -that's fine | 05:41 |
| mordred | I was just scanning the file and saw all the other stuff | 05:41 |
| lifeless | mordred: so remind me how to tell pbr about the next version # to use ? | 05:43 |
| dstufft | mordred: oh | 05:44 |
| dstufft | and lifeless | 05:44 |
| dstufft | speaking of setup.py test being awful | 05:44 |
| dstufft | this stuff i'm doing might make it pretty easy to make seup.py test use pip | 05:45 |
| lifeless | dstufft: how do you get from VersionInfo to a __version__ style version tuple ? | 05:47 |
| lifeless | dstufft: e.g. the same as sys.version_info | 05:48 |
| dstufft | I have no idea what a __version__ style tuple is, there is no standard for that :) | 05:48 |
| lifeless | dstufft: I rather think there is :) - sys.version_info is it, no ? | 05:48 |
| dstufft | All packages I have have a str __version__ tbh | 05:48 |
| lifeless | sys.version_info | 05:48 |
| lifeless | sys.version_info(major=3, minor=3, micro=1, releaselevel='final', serial=0) | 05:48 |
| dstufft | yea that's something special in python, I don't think that's exposed anywhere | 05:49 |
| lifeless | dstufft: well, its just a tuple | 05:49 |
| dstufft | namedtuple! | 05:49 |
| dstufft | maybe what you want is pkg_resources.parse_version | 05:49 |
| lifeless | dstufft: so not very special, but anything that can process said tuple can process other versions in the same structure, so I have a strong interest in using it | 05:49 |
| dstufft | I don't know what CPython itself uses | 05:50 |
| lifeless | aieee no :) | 05:50 |
| lifeless | pkg_resources.parse_version(testrepository.new_version.version) | 05:50 |
| lifeless | ('00000000', '00000000', '00000018', '*final') | 05:50 |
| dstufft | lol | 05:52 |
| *** vkozhukalov has joined #openstack-infra | 05:53 | |
| lifeless | 0.0.18.5, hmm | 05:55 |
| lifeless | 5 commits after 0.0.18 | 05:55 |
| lifeless | mordred: how do I get alpha version strings out ? | 05:56 |
| lifeless | mordred: 0.0.18 is now signed | 05:57 |
| clarkb | lifeless: you have to tag alphas iirc | 05:57 |
| *** zhiwei has quit IRC | 05:57 | |
| *** CaptTofu has joined #openstack-infra | 06:00 | |
| jhesketh__ | is anybody else having a very slow response time from gerrit queries? (ie via the commandline) | 06:00 |
| jhesketh__ | or are any root/core infra's around to help me debug? | 06:00 |
| clarkb | jhesketh__: gerrit ls-projects is responsive | 06:01 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=25&rra_id=all there si a bit of iowait there | 06:02 |
| jhesketh__ | okay, so I can query successfully from my local machine, but it seems slow | 06:02 |
| clarkb | wonder if that is backups | 06:02 |
| jhesketh__ | clarkb: more to the point though our zuul is failing to get a response altogether | 06:02 |
| * clarkb is about to go to bed and doesn't trust himself to debug | 06:02 | |
| jhesketh__ | right, possibly | 06:02 |
| jhesketh__ | that's okay | 06:02 |
| *** sarob has joined #openstack-infra | 06:11 | |
| *** unicell has quit IRC | 06:11 | |
| *** unicell has joined #openstack-infra | 06:14 | |
| *** ildikov_ has quit IRC | 06:15 | |
| openstackgerrit | David Pursehouse proposed a change to openstack-infra/jenkins-job-builder: Minor fixes in the trigger documentation https://review.openstack.org/79854 | 06:16 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Fix up some docstrings. https://review.openstack.org/79855 | 06:19 |
| openstackgerrit | lifeless proposed a change to openstack-dev/pbr: Return the real class in VersionInfo __repr__. https://review.openstack.org/79856 | 06:19 |
| *** dstufft|laptop has joined #openstack-infra | 06:20 | |
| *** oubiwann has quit IRC | 06:26 | |
| *** khyati has quit IRC | 06:29 | |
| *** Alexandra is now known as alex-out | 06:29 | |
| lifeless | so, I think get_version(pkg, pre_version...) is dead code, no ? | 06:29 |
| *** lcheng has quit IRC | 06:30 | |
| *** sarob has quit IRC | 06:43 | |
| *** lukego has joined #openstack-infra | 06:44 | |
| *** gokrokve has quit IRC | 06:46 | |
| *** dangers_away has quit IRC | 06:48 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: ZMQ Event Publisher plugin support added https://review.openstack.org/77561 | 06:50 |
| *** denis_makogon has joined #openstack-infra | 06:50 | |
| openstackgerrit | Sergey Kolekonov proposed a change to openstack-infra/jenkins-job-builder: Added send-to options support to email-ext plugin https://review.openstack.org/73601 | 06:50 |
| lifeless | yeah, I'm going to need to patch pbr. Tomorrow. mordred :) - pbr isn't compatible with it's own semver doc :> | 06:51 |
| *** saju_m has joined #openstack-infra | 06:52 | |
| *** rpodolyaka has quit IRC | 07:00 | |
| SergeyLukjanov | fungi, clarkb, mordred, jeblair, are you sleeping folks? | 07:05 |
| SergeyLukjanov | we need a force push for savanna to revert 0.9 sqla | 07:05 |
| *** pblaho has joined #openstack-infra | 07:08 | |
| *** ildikov_ has joined #openstack-infra | 07:10 | |
| *** mrda is now known as mrda_away | 07:11 | |
| SergeyLukjanov | fungi, thank you for the renaming doc, looking on it | 07:15 |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Initialize transifex data for openstack-manuals https://review.openstack.org/79866 | 07:16 |
| SergeyLukjanov | fungi, everything looks ok, I'll do #14 by myself | 07:17 |
| AJaeger | SergeyLukjanov, good morning! If you find some time between renaming to sahara, could you review 79866, please? | 07:20 |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Fix of using concurrency for tempest in manila job https://review.openstack.org/79867 | 07:21 |
| SergeyLukjanov | AJaeger, morning | 07:21 |
| SergeyLukjanov | AJaeger, sure, I'll take a look on it | 07:21 |
| *** AaronGreen has quit IRC | 07:21 | |
| *** AaronGr has joined #openstack-infra | 07:23 | |
| *** vponomaryov1 has joined #openstack-infra | 07:23 | |
| *** bknudson has quit IRC | 07:24 | |
| *** flaper87|afk is now known as flaper87 | 07:24 | |
| AJaeger | SergeyLukjanov, thanks | 07:25 |
| *** yolanda has joined #openstack-infra | 07:26 | |
| *** bknudson has joined #openstack-infra | 07:26 | |
| SergeyLukjanov | AJaeger, could you please capitalize and remove ',' in comment in https://review.openstack.org/#/c/79866/1/modules/jenkins/files/slave_scripts/upstream_translation_update_manuals.sh | 07:30 |
| AJaeger | SergeyLukjanov, will do - thanks | 07:33 |
| *** skraynev_afk is now known as skraynev | 07:34 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Initialize transifex data for openstack-manuals https://review.openstack.org/79866 | 07:34 |
| *** jlibosva has joined #openstack-infra | 07:35 | |
| *** CaptTofu has quit IRC | 07:35 | |
| SergeyLukjanov | AJaeger, thank you | 07:35 |
| AJaeger | SergeyLukjanov, thanks for spotting! | 07:36 |
| *** iv_m_ has joined #openstack-infra | 07:38 | |
| *** vkozhukalov has quit IRC | 07:38 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Rename migration revisions https://review.openstack.org/79869 | 07:40 |
| *** iv_m_ is now known as iv_m | 07:40 | |
| *** vponomaryov1 has quit IRC | 07:41 | |
| *** sarob has joined #openstack-infra | 07:41 | |
| *** vponomaryov1 has joined #openstack-infra | 07:41 | |
| *** gokrokve has joined #openstack-infra | 07:42 | |
| *** gokrokve has quit IRC | 07:47 | |
| *** sarob has quit IRC | 07:48 | |
| *** rcarrillocruz has joined #openstack-infra | 07:50 | |
| *** rlandy has joined #openstack-infra | 07:50 | |
| AJaeger | SergeyLukjanov, did you forgot to review 79866 with a vote? | 07:51 |
| *** dstufft|laptop has quit IRC | 07:57 | |
| *** jcoufal has joined #openstack-infra | 07:59 | |
| SergeyLukjanov | AJaeger, /me searching for coffee :) will do | 08:02 |
| *** rossella_s has joined #openstack-infra | 08:06 | |
| *** gokrokve has joined #openstack-infra | 08:13 | |
| *** morganfainberg is now known as morganfainberg_Z | 08:17 | |
| *** locke105 has quit IRC | 08:17 | |
| *** vponomaryov1 has left #openstack-infra | 08:18 | |
| *** gokrokve has quit IRC | 08:18 | |
| *** eharney has joined #openstack-infra | 08:20 | |
| *** matsuhashi has quit IRC | 08:21 | |
| *** jgallard has joined #openstack-infra | 08:24 | |
| *** bada_ has joined #openstack-infra | 08:26 | |
| *** bada has quit IRC | 08:30 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/devstack-gate: Archive config files along with logs https://review.openstack.org/69344 | 08:32 |
| *** ociuhandu has joined #openstack-infra | 08:35 | |
| *** amcrn has joined #openstack-infra | 08:35 | |
| *** jooools has joined #openstack-infra | 08:35 | |
| *** jungleboyj has quit IRC | 08:38 | |
| *** e0ne has joined #openstack-infra | 08:40 | |
| *** saju_m has quit IRC | 08:41 | |
| *** hashar has joined #openstack-infra | 08:41 | |
| *** gokrokve has joined #openstack-infra | 08:43 | |
| *** sarob has joined #openstack-infra | 08:44 | |
| *** ekarlso has quit IRC | 08:47 | |
| *** ekarlso has joined #openstack-infra | 08:47 | |
| *** gokrokve has quit IRC | 08:47 | |
| *** e0ne has quit IRC | 08:50 | |
| *** amcrn_ has joined #openstack-infra | 08:50 | |
| *** amcrn has quit IRC | 08:53 | |
| *** iv_m has left #openstack-infra | 08:53 | |
| *** amcrn_ is now known as amcrn | 08:53 | |
| *** yassine has joined #openstack-infra | 08:54 | |
| *** e0ne has joined #openstack-infra | 08:56 | |
| *** jepoy_ has joined #openstack-infra | 08:57 | |
| *** e0ne_ has joined #openstack-infra | 08:57 | |
| *** jepoy has quit IRC | 08:59 | |
| *** e0ne has quit IRC | 09:01 | |
| *** zhiyan is now known as zhiyan_ | 09:06 | |
| *** jpich has joined #openstack-infra | 09:07 | |
| *** dizquierdo has joined #openstack-infra | 09:08 | |
| *** saju_m has joined #openstack-infra | 09:09 | |
| *** derekh has joined #openstack-infra | 09:12 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Fill the last_login field https://review.openstack.org/79887 | 09:18 |
| *** sarob has quit IRC | 09:18 | |
| *** Daisy__ has quit IRC | 09:20 | |
| *** jcoufal has quit IRC | 09:20 | |
| *** jcoufal has joined #openstack-infra | 09:22 | |
| *** jhesketh_ has quit IRC | 09:23 | |
| *** jhesketh__ has quit IRC | 09:24 | |
| *** enikanorov_ has joined #openstack-infra | 09:24 | |
| *** CaptTofu has joined #openstack-infra | 09:32 | |
| *** johnthetubaguy has joined #openstack-infra | 09:33 | |
| *** jepoy_ has quit IRC | 09:35 | |
| *** yaguang has quit IRC | 09:36 | |
| *** jhesketh__ has joined #openstack-infra | 09:36 | |
| *** afazekas has joined #openstack-infra | 09:36 | |
| *** jhesketh_ has joined #openstack-infra | 09:36 | |
| *** CaptTofu has quit IRC | 09:37 | |
| *** amcrn has quit IRC | 09:38 | |
| *** dizquierdo has quit IRC | 09:40 | |
| *** gokrokve has joined #openstack-infra | 09:43 | |
| *** gokrokve has quit IRC | 09:48 | |
| *** denis_makogon has quit IRC | 09:51 | |
| *** hashar has quit IRC | 09:58 | |
| *** pbelanyi has joined #openstack-infra | 10:01 | |
| *** zhiwei has joined #openstack-infra | 10:06 | |
| *** yamahata__ has quit IRC | 10:10 | |
| *** hashar has joined #openstack-infra | 10:10 | |
| *** jp_at_hp has joined #openstack-infra | 10:13 | |
| *** sarob has joined #openstack-infra | 10:14 | |
| *** yamahata has quit IRC | 10:17 | |
| *** rlandy has quit IRC | 10:18 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Missing security decorators added https://review.openstack.org/79895 | 10:18 |
| *** branen_ has quit IRC | 10:18 | |
| *** jerryz has quit IRC | 10:20 | |
| *** andreaf has joined #openstack-infra | 10:26 | |
| *** rlandy has joined #openstack-infra | 10:31 | |
| *** dizquierdo has joined #openstack-infra | 10:32 | |
| *** jamielennox is now known as jamielennox|away | 10:34 | |
| *** mugsie_ has quit IRC | 10:36 | |
| *** dcramer__ has quit IRC | 10:41 | |
| *** gokrokve has joined #openstack-infra | 10:43 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create a nova-docker repo https://review.openstack.org/79900 | 10:44 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create a nova-docker repo https://review.openstack.org/79900 | 10:46 |
| *** sarob has quit IRC | 10:47 | |
| *** gokrokve has quit IRC | 10:48 | |
| openstackgerrit | Victor Stinner proposed a change to openstack/requirements: Allow trollius 0.2 https://review.openstack.org/79901 | 10:48 |
| *** saju_m has quit IRC | 10:49 | |
| *** eharney has quit IRC | 10:50 | |
| *** lcostantino has joined #openstack-infra | 10:51 | |
| jlibosva | hello, is there a way how can I re-trigger check on gate even though Jenkins gave +1? | 10:53 |
| StevenK | jlibosva: 'recheck no bug' as a review comment | 10:54 |
| jlibosva | StevenK: I tired that but it didn't trigger recheck | 10:54 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Create a nova-docker repo https://review.openstack.org/79900 | 10:54 |
| * fungi missed the eldritch-horror pip branch discussion... that's what i get for taking a nap | 10:56 | |
| fungi | jlibosva: do you have a link to the review where that didn't work? | 10:57 |
| fungi | SergeyLukjanov: where's the savanna reqs change you need forced in? | 10:57 |
| SergeyLukjanov | fungi, morning | 10:58 |
| SergeyLukjanov | fungi, all our d-g jobs are failing now | 10:58 |
| SergeyLukjanov | and I can't understand why | 10:58 |
| SergeyLukjanov | fungi, potential reason is that we've merged sqla 0.9 | 10:58 |
| *** bada_ has quit IRC | 10:58 | |
| SergeyLukjanov | fungi, and IIRC due to the devstack issue with requirements revert failing | 10:59 |
| jlibosva | fungi: yes, https://review.openstack.org/#/c/58695/27 | 10:59 |
| fungi | SergeyLukjanov: there was a logic error in devstack which was causing changes not to get tested with the global requirements list, and instead tested them with the combined requirements of projects being integrated. the fix for that finally merged yesterday but it's possible there were some requirements changes in projects which ended up being unsupportable and need to get adjusted | 11:00 |
| *** Ryan_Lane has quit IRC | 11:00 | |
| SergeyLukjanov | fungi, oh, got it | 11:00 |
| SergeyLukjanov | fungi, so, could you, please, forse push https://review.openstack.org/#/c/79647/ | 11:00 |
| SergeyLukjanov | to revert | 11:00 |
| SergeyLukjanov | and probably take a look on how d-g jobs failing for savanna | 11:00 |
| fungi | looking over the failure log on that one now... | 11:01 |
| *** spm has joined #openstack-infra | 11:01 | |
| SergeyLukjanov | fungi, thank you | 11:01 |
| SergeyLukjanov | in fact I see "ImageKilledException: Image 8c68628a-3cdd-4e4a-87aa-9ed2562a6cd4 'killed' while waiting for 'active'" that's libvirt issue IIRC and horizon 500 | 11:02 |
| *** Ryan_Lane has joined #openstack-infra | 11:02 | |
| *** AaronGr has quit IRC | 11:02 | |
| fungi | yeah, on the check-tempest-dsvm-savanna-full failure, that looks like one i've seen before | 11:02 |
| cyeoh | does anyone know how I can get the ical feed for the openstack meetings updated with a new meeting? | 11:04 |
| fungi | the other two look like error responses from a service api | 11:04 |
| fungi | cyeoh: you update the wiki. ttx subscribes to updates to that article and manually updates the ical feed | 11:05 |
| cyeoh | fungi: ah ok, thanks! Have updated the wiki so I'll just wait :-) | 11:05 |
| fungi | cyeoh: http://git.openstack.org/cgit/openstack-infra/gerrit-powered-agenda/ hopes to fix that | 11:05 |
| *** AaronGr has joined #openstack-infra | 11:05 | |
| cyeoh | fungi, ah that'd be nice | 11:06 |
| SergeyLukjanov | fungi, the only change we merged in the last few days is bump sqla, so, revert should help | 11:06 |
| cyeoh | was also wondering if the various commands like #topic, #startmeeting etc are documented anywhere? | 11:07 |
| fungi | cyeoh: a little here http://ci.openstack.org/irc.html#id3 | 11:08 |
| cyeoh | fungi: thanks again | 11:08 |
| fungi | cyeoh: and more at https://git.openstack.org/cgit/openstack-infra/meetbot/tree/doc/Manual.txt | 11:08 |
| fungi | we probably ought to publish that in a slightly more discoverable place | 11:09 |
| *** mugsie has joined #openstack-infra | 11:09 | |
| SergeyLukjanov | fungi, before repo rename we need the to have landed https://review.openstack.org/#/c/79063/ and https://review.openstack.org/#/c/79347/ to add aliases | 11:09 |
| cyeoh | heh, thx - yea I did google a bit first for it, but I didn't know that it was called meetbot so didn't find anything | 11:09 |
| SergeyLukjanov | but both blocked by failing gate | 11:09 |
| *** jgallard has quit IRC | 11:10 | |
| fungi | SergeyLukjanov: well, there's actually no way of knowing what else might have changed in global-requirements.txt which is breaking savanna devstack jobs now that it's being applied again. in actuality i don't think any change to your requirements.txt or test-requirements.txt files in savanna should affect devstack jobs now that global-requirements.txt is being used to override those | 11:11 |
| SergeyLukjanov | fungi, savanna-related tests passed successfully | 11:12 |
| SergeyLukjanov | http://logs.openstack.org/47/79647/2/check/check-tempest-dsvm-savanna-full/877f1ca/console.html#_2014-03-11_19_16_43_140 | 11:12 |
| fungi | SergeyLukjanov: so i don't have a lot of faith that 79647 will make any difference there at all | 11:12 |
| fungi | but i'll go ahead and force submit it to merge | 11:12 |
| SergeyLukjanov | fungi, thx | 11:13 |
| jlibosva | thanks Jeremy :) seems like Jenkins is just ignoring my name | 11:13 |
| SergeyLukjanov | fungi, there were only oslo sync in savanna and sqla version bump for the last 5 days | 11:13 |
| SergeyLukjanov | fungi, and savanna tests pass | 11:13 |
| SergeyLukjanov | fungi, so, I just don't know how it could affect other tests... | 11:13 |
| fungi | SergeyLukjanov: right, i mean the devstack jobs for savanna should be completely ignoring versions listed in your requirements files in savanna now that devstack is fixed, so i'm pretty sure changing your requirements in savanna isn't going to change the behavior for those. it would change behavior for your unit tests though | 11:15 |
| SergeyLukjanov | fungi, yup... if so, then it looks like Savanna addition to the d-g breaks something... | 11:15 |
| fungi | SergeyLukjanov: probably | 11:16 |
| fungi | SergeyLukjanov: okay, the three you requested are merged | 11:19 |
| SergeyLukjanov | fungi, thank you | 11:19 |
| SergeyLukjanov | fungi, I've recheck https://review.openstack.org/#/c/78789/ | 11:19 |
| SergeyLukjanov | it's just a doc change, so, we should wee how it works | 11:19 |
| fungi | SergeyLukjanov: okay. i'm going to start putting jenkins masters in shutdown mode now | 11:20 |
| SergeyLukjanov | fungi, thank you! | 11:20 |
| fungi | which means currently running jobs will finish but new jobs will wait in a pending state in zuul | 11:20 |
| *** hashar has quit IRC | 11:20 | |
| SergeyLukjanov | fungi, I've reviewed renaming pad and it's lgtm (one path fixed) | 11:21 |
| SergeyLukjanov | and I'm volunteering to update .gitreviews on all updated repos | 11:21 |
| fungi | SergeyLukjanov: thanks--i guess i had an extra letter in that line or something? | 11:22 |
| SergeyLukjanov | fungi, double paste path | 11:22 |
| SergeyLukjanov | I mean doubled | 11:22 |
| fungi | ahh, overzealous middle button ;) | 11:22 |
| SergeyLukjanov | :) | 11:22 |
| *** e0ne_ has quit IRC | 11:22 | |
| fungi | thanks | 11:22 |
| SergeyLukjanov | thanks to *you* :) | 11:23 |
| *** e0ne has joined #openstack-infra | 11:23 | |
| *** CaptTofu has joined #openstack-infra | 11:24 | |
| *** masayukig has joined #openstack-infra | 11:25 | |
| *** bada has joined #openstack-infra | 11:25 | |
| fungi | okay, all jenkins masters are quiescing now | 11:25 |
| fungi | time to see if statusbot catches fire and burns to the ground | 11:25 |
| *** e0ne_ has joined #openstack-infra | 11:26 | |
| fungi | #status alert test/gate jobs are queuing now in preparation for gerrit maintenance at 12:00 utc (eta to resume is 12:30 utc) | 11:27 |
| openstackstatus | fungi: sending alert | 11:27 |
| SergeyLukjanov | aaand looks like we still alive | 11:27 |
| SergeyLukjanov | but topic is unchanged | 11:27 |
| openstackstatus | NOTICE: test/gate jobs are queuing now in preparation for gerrit maintenance at 12:00 utc (eta to resume is 12:30 utc) | 11:27 |
| *** ChanServ changes topic to "test/gate jobs are queuing now in preparation for gerrit maintenance at 12:00 utc (eta to resume is 12:30 utc)" | 11:27 | |
| fungi | patience ;) | 11:27 |
| SergeyLukjanov | oh | 11:27 |
| SergeyLukjanov | :) | 11:27 |
| openstackstatus | fungi: finished sending alert | 11:28 |
| SergeyLukjanov | awesome | 11:28 |
| SergeyLukjanov | and the same with other channels | 11:28 |
| SergeyLukjanov | except both savanna / sahara ;) | 11:28 |
| *** CaptTofu has quit IRC | 11:28 | |
| fungi | with the latest changes, it's now in >50 official irc channels, so it has a built-in throttle to prevent it from getting killed by the irc server for abuse | 11:28 |
| fungi | which is also why it now replies to you when it starts and finishes updating | 11:29 |
| *** e0ne_ has quit IRC | 11:29 | |
| SergeyLukjanov | yeah, I've reviewed it, but forget about throttle | 11:29 |
| *** e0ne has quit IRC | 11:29 | |
| *** e0ne_ has joined #openstack-infra | 11:29 | |
| *** bada has quit IRC | 11:30 | |
| sdague | fungi: man, what are you doing up so early :) | 11:31 |
| fungi | sdague: breaking things of course | 11:31 |
| *** masayukig has quit IRC | 11:31 | |
| fungi | sdague: i'm usually up this early, i just don't normally jump in irc this early | 11:31 |
| sdague | heh | 11:31 |
| fungi | i typically wake up around 10:00 utc | 11:32 |
| sdague | ah, I jump straight to irc in the morning. with my wife still sleeping might as well get some work done early. | 11:32 |
| sdague | fungi: regardless of dst? | 11:32 |
| * fungi doesn't subscribe to his government's notions of clock-related fascism | 11:33 | |
| sdague | damn fiat clocks! | 11:33 |
| fungi | i find that with the interrupt-driven nature of what i work on, if i don't make time for non-openstack things first, i frequently find that it's suddenly evening and i never showered and ate breakfast | 11:34 |
| SergeyLukjanov | sdague, morning | 11:34 |
| fungi | which i'm going to go do quickly now while i wait for these running jobs to wrap up | 11:34 |
| * fungi will brb | 11:34 | |
| sdague | SergeyLukjanov: morning | 11:35 |
| sdague | SergeyLukjanov: so what are the changes you need landed, and are they before or after the rename? | 11:36 |
| SergeyLukjanov | sdague, fungi, heh, I have inversed day schedule ;) wake up around 5am UTC (9am local) and creep to the bad around 10pm UTC | 11:36 |
| SergeyLukjanov | sdague, all CRs needed to be landed before the rename already landed | 11:37 |
| sdague | ok | 11:37 |
| SergeyLukjanov | sdague, after repos rename - d-g update + devstack update | 11:37 |
| SergeyLukjanov | and I hope it'll just work | 11:37 |
| sdague | sounds good, point me to them once we are done | 11:37 |
| SergeyLukjanov | probably several force pushed commits :) | 11:37 |
| SergeyLukjanov | sdague, sure, thanks | 11:37 |
| *** CaptTofu has joined #openstack-infra | 11:38 | |
| *** johnthetubaguy1 has joined #openstack-infra | 11:39 | |
| *** johnthetubaguy has quit IRC | 11:42 | |
| *** fifieldt has quit IRC | 11:42 | |
| *** gokrokve has joined #openstack-infra | 11:43 | |
| *** sarob has joined #openstack-infra | 11:44 | |
| *** mbacchi has joined #openstack-infra | 11:46 | |
| *** gokrokve has quit IRC | 11:47 | |
| *** talluri has joined #openstack-infra | 11:48 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard: Missing security decorators added https://review.openstack.org/79895 | 11:52 |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Add n-novnc and n-cauth https://review.openstack.org/55721 | 11:52 |
| *** talluri has quit IRC | 11:52 | |
| *** mkerrin has quit IRC | 11:53 | |
| *** lcestari has joined #openstack-infra | 11:55 | |
| *** mwagner_lap has quit IRC | 11:59 | |
| *** jamespage has joined #openstack-infra | 11:59 | |
| fungi | okay, bacvk | 11:59 |
| fungi | looks like there are some savanna-related changes which ended up approved into the gate. not sure what will happen to those when the jobs no longer exist... guess we'll find out | 12:00 |
| *** spm has left #openstack-infra | 12:00 | |
| SergeyLukjanov | fungi, they aren't important, I'll reapprove them if needed | 12:01 |
| *** mkerrin has joined #openstack-infra | 12:01 | |
| fungi | i'm ready to stop gerrit, but i'm going to rename your gerrit groups first so that manage-projects doesn't recreate them when it imports the updated acls | 12:01 |
| *** rlandy_ has joined #openstack-infra | 12:02 | |
| *** aysyd has joined #openstack-infra | 12:02 | |
| SergeyLukjanov | fungi, ok | 12:03 |
| *** rlandy has quit IRC | 12:04 | |
| *** mburned_out is now known as mburned | 12:04 | |
| *** eharney has joined #openstack-infra | 12:05 | |
| fungi | now to test whether statusbot will still correctly update topics when changing an alert without cancelling it first | 12:05 |
| fungi | #status gerrit on review.openstack.org is down for maintenance (eta to resume is 12:30 utc) | 12:06 |
| openstackstatus | fungi: unknown command | 12:06 |
| fungi | duh | 12:06 |
| fungi | #status alert gerrit on review.openstack.org is down for maintenance (eta to resume is 12:30 utc) | 12:06 |
| openstackstatus | fungi: sending alert | 12:06 |
| fungi | it's not actually down yet, but the last jobs are about wrapped up | 12:08 |
| *** jgallard has joined #openstack-infra | 12:09 | |
| *** talluri has joined #openstack-infra | 12:12 | |
| fungi | and statusbot seems not to have updated correctly (or it's still trying) | 12:12 |
| *** e0ne has joined #openstack-infra | 12:13 | |
| SergeyLukjanov | fungi, I see "[16:06:38] <openstackstatus> NOTICE: gerrit on review.openstack.org is down for maintenance (eta to resume is 12:30 utc)" in some channels | 12:14 |
| fungi | okay, hopefully it's just taking its time | 12:14 |
| *** julim has joined #openstack-infra | 12:17 | |
| *** sarob has quit IRC | 12:17 | |
| *** dprince has joined #openstack-infra | 12:18 | |
| *** yassine has quit IRC | 12:19 | |
| *** yassine has joined #openstack-infra | 12:19 | |
| *** yassine has quit IRC | 12:19 | |
| *** yassine has joined #openstack-infra | 12:19 | |
| *** yassine has quit IRC | 12:20 | |
| *** yassine has joined #openstack-infra | 12:20 | |
| *** enikanorov_ has quit IRC | 12:21 | |
| fungi | i'm killing these last few long-running jobs because they're all in check and either non-voting or associated with changes which have a bunch of other failed jobs | 12:22 |
| fungi | and now we're ready | 12:23 |
| fungi | #status alert gerrit on review.openstack.org is down for maintenance (revised eta to resume is 13:00 utc) | 12:23 |
| openstackstatus | fungi: sending alert | 12:23 |
| *** david-lyle has quit IRC | 12:24 | |
| openstackstatus | NOTICE: gerrit on review.openstack.org is down for maintenance (revised eta to resume is 13:00 utc) | 12:24 |
| *** ChanServ changes topic to "gerrit on review.openstack.org is down for maintenance (revised eta to resume is 13:00 utc)" | 12:24 | |
| openstackstatus | fungi: finished sending alert | 12:24 |
| *** enikanorov_ has joined #openstack-infra | 12:26 | |
| fungi | db updates in step #5 done. working on filesystem modifications now | 12:27 |
| * SergeyLukjanov crossed fingers | 12:30 | |
| *** bogdando has quit IRC | 12:33 | |
| fungi | filesystem updates done. going to skip steps 8-12 and do step 13 first since i just realized that otherwise i may end up with a mess courtesy of manage-projects | 12:33 |
| *** saju_m has joined #openstack-infra | 12:37 | |
| *** esker has joined #openstack-infra | 12:39 | |
| afazekas | On https://review.openstack.org/ I got 503 | 12:39 |
| fungi | afazekas: see /topic | 12:40 |
| afazekas | :) | 12:40 |
| *** jasondotstar has joined #openstack-infra | 12:40 | |
| fungi | okay, #13 done, dropping back to 8 now | 12:41 |
| *** ArxCruz has joined #openstack-infra | 12:42 | |
| fungi | gerrit is back up, working on remaining steps niow | 12:42 |
| *** gokrokve has joined #openstack-infra | 12:43 | |
| *** jhesketh__ has quit IRC | 12:44 | |
| *** jhesketh_ has quit IRC | 12:45 | |
| fungi | step 9 done, zuul shows jobs running again, cancelling alert before proceeding | 12:45 |
| fungi | #status ok | 12:45 |
| openstackstatus | fungi: sending ok | 12:45 |
| tristanC | Good work! | 12:45 |
| SergeyLukjanov | fungi, yay! I'm going to send .gitreview updates | 12:45 |
| openstackstatus | fungi: finished sending ok | 12:45 |
| fungi | SergeyLukjanov: not done yet | 12:46 |
| fungi | SergeyLukjanov: your projects still aren't done being renamed until i get to the end | 12:46 |
| *** hashar has joined #openstack-infra | 12:46 | |
| fungi | i merely ended the alert because gerrit and jenkins are back online again | 12:46 |
| fungi | okay, 79097 and 64788 have been approved | 12:46 |
| SergeyLukjanov | fungi, yup | 12:47 |
| *** CaptTofu has quit IRC | 12:47 | |
| *** zhiyan_ is now known as zhiyan | 12:47 | |
| *** CaptTofu has joined #openstack-infra | 12:47 | |
| fungi | to speed this along, i'm preparing to pull an early update on ci-puppetmaster once they get through the gate in a moment | 12:48 |
| *** gokrokve has quit IRC | 12:48 | |
| fungi | otherwise we'll be waiting on the 13:00 cron pulse before i can update the puppet config on review.o.o | 12:49 |
| *** johnthetubaguy1 has quit IRC | 12:49 | |
| openstackgerrit | A change was merged to openstack-infra/config: First step of renaming Savanna to Sahara https://review.openstack.org/79097 | 12:49 |
| SergeyLukjanov | fungi, ok | 12:49 |
| *** rlandy__ has joined #openstack-infra | 12:50 | |
| fungi | SergeyLukjanov: also, i expect at least some of your jenkins jobs will be broken for the next few minutes too until jenkins-job-builder runs on the jenkins masters | 12:50 |
| SergeyLukjanov | fungi, git it | 12:51 |
| *** thuc_ has joined #openstack-infra | 12:51 | |
| SergeyLukjanov | :) | 12:51 |
| fungi | but it should sort itself out within about 15 minutes | 12:51 |
| SergeyLukjanov | got* | 12:51 |
| openstackgerrit | A change was merged to openstack-infra/config: Rename a StackForge project https://review.openstack.org/64788 | 12:51 |
| fungi | okay, updating the puppetmaster so i can continue | 12:51 |
| *** johnthetubaguy has joined #openstack-infra | 12:51 | |
| *** ociuhandu has quit IRC | 12:52 | |
| *** CaptTofu has quit IRC | 12:52 | |
| *** esker has quit IRC | 12:52 | |
| *** esker has joined #openstack-infra | 12:53 | |
| *** rlandy_ has quit IRC | 12:53 | |
| *** rlandy has joined #openstack-infra | 12:53 | |
| *** dkranz has joined #openstack-infra | 12:54 | |
| *** openstackgerrit has quit IRC | 12:54 | |
| *** openstackgerrit has joined #openstack-infra | 12:54 | |
| *** rlandy__ has quit IRC | 12:54 | |
| *** bogdando has joined #openstack-infra | 12:55 | |
| fungi | also looks like statusbot never updated the topics back, so something like 54 channels we need to reset topics on | 12:55 |
| *** johnthetubaguy has quit IRC | 12:55 | |
| *** weshay has quit IRC | 12:55 | |
| *** johnthetubaguy has joined #openstack-infra | 12:55 | |
| SergeyLukjanov | fungi, heh :( | 12:57 |
| *** changbl has quit IRC | 12:57 | |
| fungi | manage-projects is running for the 9 renamed repositories now | 12:57 |
| fungi | looks like it ran clean... checking behind it | 12:58 |
| AJaeger | fungi, IRC worked on openstack-doc - so only 53 ;) | 12:59 |
| fungi | AJaeger: nice! | 12:59 |
| fungi | spot checks of renamed projects' acls look correct | 13:01 |
| fungi | also puppet agent is restarted on review.o.o now | 13:02 |
| SergeyLukjanov | fungi, looks like chanserv start returning old topics, is it you? | 13:02 |
| fungi | SergeyLukjanov: oh, is it still trying? | 13:02 |
| SergeyLukjanov | fungi, yup | 13:02 |
| fungi | hasn't updated in here yet | 13:02 |
| *** dcramer__ has joined #openstack-infra | 13:02 | |
| SergeyLukjanov | fungi, in #heat for e | 13:02 |
| SergeyLukjanov | [16:45:28] ChanServ changed the topic to support @ https://ask.openstack.org | developer wiki @ https://wiki.openstack.org/wiki/Heat | development @ https://launchpad.net/heat | logged @ http://eavesdrop.openstack.org/irclogs/%23heat/ | 13:02 |
| *** rlandy_ has joined #openstack-infra | 13:03 | |
| fungi | SergeyLukjanov: yeah, but back at 12:45 utc it also responded to me that it was done sending updates to channels | 13:03 |
| fungi | so i don't think it fixed any more of them after that | 13:03 |
| fungi | for example, the topic in here hasn't been restored to normal yet | 13:04 |
| fungi | i haven't checked its logs, but will once we're settled on the final renaming tasks | 13:04 |
| SergeyLukjanov | fungi, ok, thank you | 13:04 |
| fungi | SergeyLukjanov: i suppose you can send your .gitreview updates to those projects now and we'll make sure they run jobs and whatnot | 13:05 |
| SergeyLukjanov | fungi, already done | 13:05 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Hide "Create Project" for non-superusers https://review.openstack.org/79939 | 13:05 |
| *** rlandy has quit IRC | 13:05 | |
| fungi | zuul reports "Last reconfigured: Wed Mar 12 2014 12:57:46 GMT+0000 (UTC)" so it should know what to do with them if they got submitted to gerrit after that | 13:05 |
| SergeyLukjanov | fungi, yup | 13:07 |
| fungi | if not, recheck no bug | 13:07 |
| SergeyLukjanov | fungi, I've send them ~13:00 | 13:07 |
| SergeyLukjanov | fungi, I'll track them | 13:07 |
| fungi | great | 13:07 |
| fungi | thanks! | 13:07 |
| AJaeger | fungi, 79932,1 has one "NOT_REGISTERED" | 13:07 |
| SergeyLukjanov | fungi, could you, please, approve https://review.openstack.org/#/c/78960/ too | 13:08 |
| AJaeger | fungi, same with 79333 - the ones afterwards look fine so far | 13:08 |
| SergeyLukjanov | fungi, to make us able to move to the channel | 13:08 |
| fungi | AJaeger: yeah, that probably means that jjb hadn't run on any of the jenkins masters to create it yet | 13:08 |
| SergeyLukjanov | fungi,yeah, sounds like jjb isnt created jobs yet | 13:08 |
| *** openstackgerrit has quit IRC | 13:08 | |
| *** openstackgerrit has joined #openstack-infra | 13:08 | |
| fungi | done. doesn't look like there were any meetings going on yet | 13:09 |
| SergeyLukjanov | fungi, thx | 13:10 |
| fungi | i'm going to start analyzing the statusbot problems | 13:10 |
| *** rcarrillocruz1 has joined #openstack-infra | 13:12 | |
| *** rcarrillocruz has quit IRC | 13:12 | |
| fungi | 2014-03-12 12:45:39,086 INFO statusbot.bot: Setting topic on #openstack-infra to Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/ | 13:13 |
| fungi | except... it didn't | 13:13 |
| *** talluri has quit IRC | 13:13 | |
| SergeyLukjanov | fungi, heh | 13:14 |
| *** dkliban has joined #openstack-infra | 13:14 | |
| fungi | nothing useful in the debug log either. we may need to add a belt and braces so it checks behind itself and retries topic changes multiple times | 13:15 |
| *** sarob has joined #openstack-infra | 13:15 | |
| fungi | in the meantime, at least the log says what it *thought* it did, so i'll check behind it and fix what it didn't | 13:15 |
| *** rlandy__ has joined #openstack-infra | 13:15 | |
| *** jeckersb_gone is now known as jeckersb | 13:15 | |
| *** esker has quit IRC | 13:16 | |
| SergeyLukjanov | fungi, are you ok if I self-approve such cleanup https://review.openstack.org/#/c/79543/ for sahara renaming when env will be ready? | 13:16 |
| *** ociuhandu has joined #openstack-infra | 13:16 | |
| fungi | SergeyLukjanov: i'll review it real quick | 13:17 |
| fungi | SergeyLukjanov: keep in mind that if the group name is the same as the project name (after the /) then it's not actually needed | 13:17 |
| *** rlandy_ has quit IRC | 13:18 | |
| SergeyLukjanov | fungi, yup, just for visibility | 13:18 |
| fungi | okay | 13:18 |
| SergeyLukjanov | fungi, or you think it's better to remove? | 13:18 |
| *** mriedem has joined #openstack-infra | 13:18 | |
| fungi | i think it's likely to cause a lot of other people to not realize it's an override and start doing the same thing on all new project additions | 13:18 |
| fungi | but we can fix it in a cleanup pass. i'd like to clear out a lot of the old cruft in that file anyway (remove unneeded upstream parameters too) | 13:19 |
| fungi | i also want to normalize the gerrit acls and collapse some of them, but i've not been finding the time | 13:19 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/devstack-gate: WIP : Try patched libvirt from @hallyn - Not for review https://review.openstack.org/79816 | 13:19 |
| fungi | because right now a lot of people cargo cult what they see in existing files into their new projects, which just adds new cruft | 13:20 |
| SergeyLukjanov | fungi, yeah, it's a big issue for acls | 13:20 |
| *** johnthetubaguy1 has joined #openstack-infra | 13:21 | |
| SergeyLukjanov | fungi, I'll take a look on projects list and try to clenup groups/upstream | 13:21 |
| *** johnthetubaguy has quit IRC | 13:21 | |
| fungi | SergeyLukjanov: apprecuiated! there's no hurry on that though... you've got your hands full with your renaming | 13:21 |
| SergeyLukjanov | fungi, now, I should wait for some time to see how jobs are running :) | 13:22 |
| SergeyLukjanov | fungi, I've retriggered https://review.openstack.org/#/c/79099/ | 13:23 |
| AJaeger | SergeyLukjanov, time for a coffee or two - and send a virtual one to fungi ;) | 13:23 |
| SergeyLukjanov | AJaeger, yeah :) | 13:23 |
| fungi | thanks! | 13:24 |
| *** sweston has quit IRC | 13:25 | |
| SergeyLukjanov | fungi, heh, looks like we'll wait for a quite long time before sahara will be unblocked :( queue isn't empty :) | 13:25 |
| *** rlandy_ has joined #openstack-infra | 13:25 | |
| *** ryanpetrello has joined #openstack-infra | 13:27 | |
| sdague | fungi: the fails in the gate right now are weird - https://jenkins07.openstack.org/job/gate-tempest-dsvm-full/2000/console | 13:27 |
| sdague | something wrong with nodepool? | 13:27 |
| sdague | or the images | 13:27 |
| sdague | https://jenkins07.openstack.org/job/gate-tempest-dsvm-full/2000/console | 13:27 |
| fungi | sdague: not that anybody's told me yet. care to enlighten? | 13:27 |
| sdague | 2014-03-12 13:12:35.413 | /opt/stack/new/devstack-gate/devstack-vm-gate.sh: line 269: cd: /opt/stack/new/devstack: No such file or directory | 13:27 |
| fungi | looking | 13:28 |
| *** rlandy__ has quit IRC | 13:28 | |
| SergeyLukjanov | sdague, fungi, probably it's because of job id 2000 (millennium failure)? :) | 13:28 |
| sdague | so I think review 80k will get allocated today | 13:29 |
| sdague | will be interesting to see what it is | 13:29 |
| fungi | http://logs.openstack.org/53/79053/2/gate/gate-tempest-dsvm-full/4e8262b/logs/devstack-gate-setup-workspace-new.txt.gz | 13:29 |
| fungi | "fatal: https://git.openstack.org/openstack/savanna-dashboard/info/refs not found: did you run git update-server-info on the server? | 13:29 |
| sdague | ah, so everything is borked until we merge those things | 13:29 |
| fungi | i guess we need to cram the d-g change through SergeyLukjanov | 13:30 |
| sdague | ok, where are the rename patches? | 13:30 |
| SergeyLukjanov | https://review.openstack.org/#/c/79099/ | 13:30 |
| SergeyLukjanov | sdague, fungi ^^ | 13:30 |
| *** thuc_ has quit IRC | 13:30 | |
| fungi | remaining patches in that topic are https://review.openstack.org/#/q/status:open+branch:master+topic:savanna-sahara,n,z | 13:30 |
| *** thuc has joined #openstack-infra | 13:31 | |
| sdague | fungi: so I think you shuold just ninja merge - https://review.openstack.org/#/c/79099/ | 13:31 |
| fungi | sdague: yep | 13:31 |
| sdague | you have my +2 on it | 13:31 |
| sdague | for backup | 13:31 |
| SergeyLukjanov | yeah, d-g is non-conditional | 13:31 |
| sdague | fungi: then can we clear dtroyer's -2 via ninja means in https://review.openstack.org/#/c/79353/3 ? | 13:32 |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Rename Savanna to Sahara https://review.openstack.org/79099 | 13:32 |
| fungi | sdague: i've spirited him away | 13:32 |
| *** CaptTofu has joined #openstack-infra | 13:34 | |
| SergeyLukjanov | fungi, sdague, I think that we shouldn't approve https://review.openstack.org/#/c/79353/ before rechecking - it'll not affects anyone except sahara | 13:34 |
| fungi | also reshuffled the gate to start retesting anything which hadn't already been kicked out | 13:34 |
| *** esker has joined #openstack-infra | 13:34 | |
| *** rlandy__ has joined #openstack-infra | 13:34 | |
| *** thuc has quit IRC | 13:35 | |
| sdague | SergeyLukjanov: sure, but clean check handles than anyway | 13:35 |
| fungi | back to checking channel topics, but ping me again if anything else looks horribly broken | 13:36 |
| sdague | oh, I guess it doesn't quite, because saraha is nv | 13:36 |
| fungi | mmm, #openstack-bacon | 13:36 |
| *** thomasem has joined #openstack-infra | 13:36 | |
| SergeyLukjanov | heh, the one positive thing for me that we catch name issue earlier... renaming being in the sync gate is much painful | 13:36 |
| *** rlandy_ has quit IRC | 13:37 | |
| *** smarcet has joined #openstack-infra | 13:37 | |
| *** dcramer__ has quit IRC | 13:37 | |
| sdague | agreed | 13:39 |
| *** jgrimm has quit IRC | 13:40 | |
| fungi | given that all devstack jobs probably want to clone savanna (well, now sahara), i should probably at a minimum start rebuilds of the devstack-precise images in all providers so we don't ddos our git servers again | 13:41 |
| SergeyLukjanov | fungi, that's correct | 13:41 |
| sdague | fungi: good call | 13:41 |
| sdague | man, who needs git servers anyway | 13:42 |
| sdague | we should just do this with smoke signals | 13:42 |
| *** rlandy__ is now known as rlandy | 13:42 | |
| dims_ | lol | 13:43 |
| *** nosnos has quit IRC | 13:43 | |
| *** gokrokve has joined #openstack-infra | 13:43 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 13:44 |
| *** adalbas has joined #openstack-infra | 13:47 | |
| *** sarob has quit IRC | 13:47 | |
| fungi | "Failed to fetch http://nov.ec2.archive.ubuntu.com/ubuntu/pool/universe/q/qpid-python/python-qpid_0.12-2_all.deb ..." huh? | 13:48 |
| fungi | http://logs.openstack.org/65/77965/9/gate/gate-tempest-dsvm-large-ops/87dc2df/logs/devstacklog.txt.gz | 13:48 |
| *** weshay has joined #openstack-infra | 13:48 | |
| *** gokrokve has quit IRC | 13:48 | |
| fungi | does hpcloud seriously fetch debian packages from amazon? | 13:48 |
| fungi | well, rather does archive.ubuntu.com seriously serve their packages from amazon? | 13:49 |
| *** talluri has joined #openstack-infra | 13:49 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/nodepool: Adding the possibility to pass a cleanup timeout to nodepool https://review.openstack.org/71261 | 13:49 |
| fungi | but yeah, i can't resolve nov.ec2.archive.ubuntu.com either | 13:49 |
| *** matsuhashi has joined #openstack-infra | 13:50 | |
| *** wchrisj has joined #openstack-infra | 13:51 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add logging to the new #openstack-sahara channel https://review.openstack.org/78960 | 13:52 |
| sdague | fungi: it wouldn't surprise me | 13:53 |
| sdague | or at least that it's a valid mirror | 13:53 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Initial support to a custom puppet master server https://review.openstack.org/77587 | 13:53 |
| fungi | apparently cloud-init picks your package mirror for you | 13:53 |
| *** talluri has quit IRC | 13:54 | |
| *** matsuhashi has quit IRC | 13:56 | |
| sdague | I wonder where it picks it from | 13:56 |
| *** yamahata has joined #openstack-infra | 13:57 | |
| fungi | that's one of those ubuntu mysteries i try not to look to deeply into | 13:57 |
| fungi | because it will probably just send me running screaming if i discover the truth | 13:58 |
| *** vhoward- has joined #openstack-infra | 13:58 | |
| *** zhiwei has left #openstack-infra | 13:59 | |
| *** matsuhashi has joined #openstack-infra | 14:00 | |
| *** mwagner_lap has joined #openstack-infra | 14:01 | |
| *** markmcclain has joined #openstack-infra | 14:01 | |
| openstackgerrit | A change was merged to openstack-infra/config: Rename savanna group to sahara https://review.openstack.org/79543 | 14:02 |
| *** dcramer__ has joined #openstack-infra | 14:02 | |
| *** e0ne_ has joined #openstack-infra | 14:03 | |
| *** dims_ has quit IRC | 14:03 | |
| *** openstack has joined #openstack-infra | 14:04 | |
| *** e0ne has quit IRC | 14:05 | |
| *** ChanServ sets mode: +o fungi | 14:06 | |
| *** fungi changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 14:06 | |
| *** ChanServ sets mode: -o fungi | 14:06 | |
| *** dkranz has quit IRC | 14:07 | |
| *** rcleere has joined #openstack-infra | 14:09 | |
| *** talluri has joined #openstack-infra | 14:10 | |
| *** dims_ has joined #openstack-infra | 14:15 | |
| *** jasondotstar has quit IRC | 14:16 | |
| *** jepoy has joined #openstack-infra | 14:16 | |
| openstackgerrit | Antoine Musso proposed a change to openstack-infra/zuul: cloner to easily clone dependent repositories https://review.openstack.org/70373 | 14:17 |
| *** mfer has joined #openstack-infra | 14:17 | |
| *** talluri has quit IRC | 14:20 | |
| jeblair | fungi: sitrep? | 14:22 |
| *** chandan_kumar has quit IRC | 14:22 | |
| fungi | still updating topics in channels, but otherwise all sitnorm | 14:22 |
| sdague | jeblair: I think things are generally good on the rename | 14:22 |
| *** dkranz has joined #openstack-infra | 14:22 | |
| fungi | biggest hiccup was that we forgot that devstack-gate would die if it couldn't update savanna repos from git.o.o after the rename | 14:23 |
| fungi | so had to cram the d-g change sooner in to stop the bleeding | 14:23 |
| *** matsuhashi has quit IRC | 14:23 | |
| *** talluri has joined #openstack-infra | 14:24 | |
| *** chandan_kumar has joined #openstack-infra | 14:25 | |
| SergeyLukjanov | yup | 14:25 |
| jeblair | fungi: ok. i see you did decide to road test statusbot :) | 14:25 |
| *** gokrokve has joined #openstack-infra | 14:25 | |
| fungi | yep, it... mostly worked? | 14:25 |
| fungi | oh, and i have nodepool image updates running for devstack-precise in every provider so that we don't ddos ourselves with every single devstack job cloning a fresh sahara repo | 14:26 |
| jeblair | fungi: good idea | 14:26 |
| *** krotscheck has joined #openstack-infra | 14:26 | |
| *** homeless has quit IRC | 14:27 | |
| *** dkranz has quit IRC | 14:28 | |
| *** johnthetubaguy1 is now known as johnthetubaguy | 14:28 | |
| *** e0ne_ has quit IRC | 14:30 | |
| *** matsuhashi has joined #openstack-infra | 14:30 | |
| *** david-lyle has joined #openstack-infra | 14:30 | |
| jeblair | 2014-03-12 12:45:55,077 DEBUG irc.client: FROM SERVER: :ChanServ!ChanServ@services. NOTICE openstackstatus :+You have triggered services flood protection. | 14:31 |
| jeblair | 2014-03-12 12:45:55,077 DEBUG irc.client: FROM SERVER: :ChanServ!ChanServ@services. NOTICE openstackstatus :+This is your last warning. You will be ignored for 30 seconds. | 14:31 |
| jeblair | there we go | 14:31 |
| SergeyLukjanov | heh | 14:31 |
| hashar | hopefully freenode can whitelist your bot | 14:32 |
| *** zns has joined #openstack-infra | 14:32 | |
| jeblair | hashar: oh they can do that? | 14:32 |
| fungi | oh! i overlooked that because i was digging for failure responses on the /topic commands | 14:32 |
| hashar | jeblair: we have a bunch of very spammy bots and I am pretty sure we are not encountering flood silencing. | 14:33 |
| hashar | jeblair: so either we have our bot throttle their output or we got them whitelisted somehow. | 14:33 |
| fungi | hashar: do they try to update topics on >50 channels at once? that might be a separate flood protection setting for them | 14:34 |
| hashar | ah that might be too spammy indeed :] | 14:34 |
| jeblair | fungi: i think you should be able to set the topics via chanserv too, which might be faster/easier (you don't have to op yourself) | 14:34 |
| *** e0ne has joined #openstack-infra | 14:35 | |
| fungi | jeblair: ooh, good idea | 14:35 |
| *** nati_ueno has joined #openstack-infra | 14:36 | |
| *** eharney has quit IRC | 14:37 | |
| *** sweston has joined #openstack-infra | 14:37 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Sleep 1 second https://review.openstack.org/79956 | 14:37 |
| *** chandankumar_ has joined #openstack-infra | 14:40 | |
| *** dkranz has joined #openstack-infra | 14:40 | |
| *** chandankumar_ has quit IRC | 14:40 | |
| *** jgrimm has joined #openstack-infra | 14:44 | |
| *** sarob has joined #openstack-infra | 14:45 | |
| *** rfolco has joined #openstack-infra | 14:49 | |
| *** eharney has joined #openstack-infra | 14:50 | |
| *** thedodd has joined #openstack-infra | 14:50 | |
| *** wenlock has joined #openstack-infra | 14:51 | |
| *** andreaf has quit IRC | 14:52 | |
| zns | ilyashakhat: thank you for merging my stackalytics update. | 14:54 |
| ilyashakhat | zns: you are welcome :) | 14:55 |
| hashar | jeblair: I might have ported devstack fallback system to python with latest patch of https://review.openstack.org/70373 :] | 14:56 |
| hashar | jeblair: looking for a way to test the zuul cloner know:] | 14:56 |
| zns | We got PEP-257 updated and patched pep257 and the flake8 plugins so we can check and gate on that. We're ready to try that out on satori, but we need https://review.openstack.org/#/c/79054 merged in to update the requirements files. Any chance we can get that in? | 14:57 |
| sdague | zns: so actually, I'm confused, satori is in stackforge, why would we change global requirements for that | 14:59 |
| *** changbl has joined #openstack-infra | 15:00 | |
| *** sc68cal_ has quit IRC | 15:00 | |
| *** sc68cal has joined #openstack-infra | 15:00 | |
| *** gokrokve has quit IRC | 15:02 | |
| zns | sdague: I think there's two part answer to that. For OpenStack in general, adding those in will allow us to start checking existing projects for PEP-257. For satori, I can't get https://review.openstack.org/#/c/79013/ to pass... but from your comment it sounds like maybe we are gating on something we should not be? Excuse my lack of familiarity with some of this setup.. | 15:02 |
| openstackgerrit | Malini Kamalambal proposed a change to openstack-infra/config: Add experimental job for Marconi https://review.openstack.org/78392 | 15:03 |
| *** gokrokve has joined #openstack-infra | 15:03 | |
| sdague | yeh, I have no idea why a stackforge project is gatting on global requirements | 15:03 |
| *** atiwari has joined #openstack-infra | 15:03 | |
| *** saju_m has quit IRC | 15:03 | |
| *** malini has joined #openstack-infra | 15:03 | |
| *** bogdando has quit IRC | 15:03 | |
| sdague | I think there was an overreach there in job setup | 15:03 |
| *** bogdando has joined #openstack-infra | 15:03 | |
| sdague | when an openstack project wants to add this, we should consider it | 15:03 |
| sdague | but not for stackforge projects | 15:03 |
| *** saju_m has joined #openstack-infra | 15:04 | |
| *** gokrokve_ has joined #openstack-infra | 15:04 | |
| zns | sdague: that would be my mistake in copying another project's setup. I didn't understand what the requirements gate was for. I'll remove that from our setup. | 15:04 |
| fungi | right, satori has a gate-satori-requirements job in zuul's layout.yaml, but does not actually need that | 15:04 |
| sdague | yeh, I think that's the crux of things | 15:05 |
| sdague | zns: sorry I didn't catch this on the first round of reviews | 15:05 |
| sdague | it only sunk in | 15:05 |
| fungi | it's okay if you want to force your project to work with only openstack's specified requirements list, but don't expect that to mean that you get to add requirements to openstack if you do | 15:05 |
| jeblair | it's a fine idea if a stackforge project wants to make sure they are compatible | 15:05 |
| *** krotscheck has quit IRC | 15:06 | |
| jeblair | but yeah, if you need something openstack doesn't (yet) need, then it's best to turn off the job and try to keep things in sync manually | 15:06 |
| sdague | jeblair: sure, but we need to not extend the scope of global requirements to include content that's not in openstack projects | 15:06 |
| sdague | speaking of which, it might be worth and audit to figure out if there are things in g-r that aren't in any project | 15:06 |
| zns | sdague: np. That unblocks satori. What about having the pep257 tools available for other projects to add to other style checks? Per this thread: https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg17780.html | 15:07 |
| *** mfer has quit IRC | 15:07 | |
| sdague | zns: so if an integrated project proposes a patch that needs it, then that's different | 15:07 |
| *** gokrokve has quit IRC | 15:08 | |
| sdague | but lacking that, that list is basically a "must package" list for distributors | 15:08 |
| sdague | so we don't want to add things not actually used | 15:08 |
| zns | sdague: OK. I can approach it from that end, then. | 15:09 |
| ArxCruz | clarkb: I'm getting https://git.openstack.org/openstack/savanna-dashboard/' not found | 15:09 |
| fungi | ArxCruz: http://lists.openstack.org/pipermail/openstack-dev/2014-March/029770.html | 15:09 |
| *** pcrews has joined #openstack-infra | 15:10 | |
| ArxCruz | fungi: thanks! I need to pay more attention on mail lists | 15:10 |
| zns | Where can I find out what the jobs actually do? | 15:10 |
| *** changbl has quit IRC | 15:11 | |
| fungi | ArxCruz: we've updated devstack-gate to clone the right repos, but there was a brief window where it was looking for the old ones which were no longer there, so if you're using d-g and deploying it continuously, you might have hit a patch there where it broke | 15:11 |
| ArxCruz | fungi: I'm using d-g but with a few changes by my own, I'm planning to send patches to work more generic | 15:12 |
| fungi | zns: i think we currently lack good documentation describing the jobs themselves, but you can see the yaml-based definitions for them all at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config | 15:13 |
| *** changbl has joined #openstack-infra | 15:14 | |
| russellb | SergeyLukjanov: I did edit the file you mentioned, https://review.openstack.org/#/c/79900/3 | 15:14 |
| fungi | zns: the job in question is defined at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml#n218 | 15:14 |
| fungi | zns: and the script you see it calling there is http://git.openstack.org/cgit/openstack-infra/config/tree/modules/jenkins/files/slave_scripts/project-requirements-change.py | 15:15 |
| zns | fungi: ahhhh..... that helps! | 15:16 |
| *** jasond has joined #openstack-infra | 15:16 | |
| jasond | does this situation call for a recheck or a reverify? https://review.openstack.org/#/c/77677/ | 15:16 |
| *** sarob has quit IRC | 15:17 | |
| *** branen has joined #openstack-infra | 15:17 | |
| zns | Do you have a job that checks the coverage or pylint change for a patch? I.e. did it add pylint errors or decrease coverage? That would be a cool check (we have one on an internal project that maybe I could port over). Any interest? | 15:17 |
| fungi | jasond: yes, you were one of the unlucy (hopefully) few mentioned in http://lists.openstack.org/pipermail/openstack-dev/2014-March/029770.html | 15:17 |
| *** VijayTripathi has joined #openstack-infra | 15:18 | |
| fungi | jasond: i'll open a bug for that real quick you can reverify against | 15:18 |
| jasond | fungi: thank you | 15:18 |
| *** matsuhashi has quit IRC | 15:18 | |
| *** rcarrillocruz has joined #openstack-infra | 15:18 | |
| fungi | zns: we do have jobs which run coverage reports (we stopped running pylint mostly in favor of the pyflakes backend to flake8) but nothing which calculates deltas for those i don't think | 15:19 |
| fungi | it's been discussed before, and i think there were at least a few people interested in seeing something along those lines | 15:19 |
| *** miqui has joined #openstack-infra | 15:19 | |
| *** afazekas has quit IRC | 15:19 | |
| *** homeless has joined #openstack-infra | 15:20 | |
| *** rcarrillocruz1 has quit IRC | 15:20 | |
| *** sarob has joined #openstack-infra | 15:22 | |
| *** mgagne has joined #openstack-infra | 15:24 | |
| fungi | jasond: reverify bug 1291440 | 15:24 |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Correcting the right path to mysql log in Fedora 19 https://review.openstack.org/79967 | 15:24 |
| SergeyLukjanov | russellb, I'll take a look on it soon | 15:24 |
| jasond | fungi: thanks! | 15:25 |
| *** rpodolyaka has joined #openstack-infra | 15:25 | |
| russellb | SergeyLukjanov: ok no rush, thanks | 15:25 |
| *** chandan_kumar has quit IRC | 15:26 | |
| *** sarob has quit IRC | 15:28 | |
| *** chandan_kumar has joined #openstack-infra | 15:28 | |
| *** chandan_kumar has quit IRC | 15:28 | |
| *** sarob has joined #openstack-infra | 15:28 | |
| rcarrillocruz | jeblair: hi there, I'm looking at low-hanging-fruit https://bugs.launchpad.net/openstack-ci/+bug/1061929 , but I can't seem to find where in the code we check for the recheck/reverify review comments, i'd be grateful if you could point me in the right track | 15:30 |
| rcarrillocruz | actually i meant https://bugs.launchpad.net/openstack-ci/+bug/1235335 , they are similar | 15:30 |
| *** gyee has joined #openstack-infra | 15:31 | |
| *** johnthetubaguy1 has joined #openstack-infra | 15:31 | |
| *** sarob has quit IRC | 15:33 | |
| *** johnthetubaguy has quit IRC | 15:33 | |
| jeblair | rcarrillocruz: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n15 | 15:35 |
| *** johnthetubaguy1 has quit IRC | 15:35 | |
| jeblair | rcarrillocruz: the recheck/reverify triggers are in regexes like that ^ | 15:35 |
| jeblair | rcarrillocruz: i'm not sure i would consider that a low-hanging-fruit bug | 15:35 |
| jeblair | rcarrillocruz: i have no good ideas about how to implement that :/ | 15:35 |
| jeblair | rcarrillocruz: ...maybe.... | 15:36 |
| jeblair | rcarrillocruz: it could be done like the 'require-approval' was done... | 15:37 |
| jeblair | rcarrillocruz: where we say "in order for this trigger to match, we also require that the commit message match this regex" | 15:37 |
| jeblair | rcarrillocruz: but then we'de have to actually get regex group data from the comment filter regex into the commit message regex... | 15:38 |
| *** mfer has joined #openstack-infra | 15:38 | |
| jeblair | rcarrillocruz: it seems very complicated, a bit messy, and probably too much work for the benefit... | 15:38 |
| rcarrillocruz | ahh...i was looking straight into the trigger python code... | 15:39 |
| *** johnthetubaguy has joined #openstack-infra | 15:39 | |
| *** saju_m has quit IRC | 15:40 | |
| *** rpodolyaka has quit IRC | 15:40 | |
| rcarrillocruz | that explains, heh | 15:40 |
| jeblair | rcarrillocruz: i key aspect of zuul's design is that it's all _very_ simple building blocks. we don't put any openstack-specific stuff in there -- instead, we break problems down until we can describe them in simpler terms. | 15:40 |
| jeblair | so zuul has no idea what a reverify or recheck comment is -- it's all just configuration | 15:41 |
| *** eharney has quit IRC | 15:41 | |
| *** wenlock has quit IRC | 15:41 | |
| rcarrillocruz | so that's why you say in the bug 'This would be a good thing to add to the recheck magic. Unfortunately, it may require adding some openstack specific sauce to zuul' . I infere from that sentence that you want to be zuul openstack independent... so no python checks in there for stuff that pertains to an openstack issue, like reported on bug/1235335 | 15:42 |
| rcarrillocruz | > | 15:42 |
| rcarrillocruz | ? | 15:42 |
| *** smarcet has left #openstack-infra | 15:42 | |
| rcarrillocruz | i will think about your suggestion | 15:43 |
| jeblair | right; we'd need to think of a way to describe the problem abstractly, like the two-regex idea above (which would work but is very complicated) | 15:43 |
| rcarrillocruz | if it gets hard to do, i'll move on to other low hanging fruit | 15:43 |
| rcarrillocruz | thx jeblair | 15:43 |
| malini | SergeyLukjanov, sdague: Can you please take look at this https://review.openstack.org/#/c/78392/ when you have a minute ? | 15:44 |
| SergeyLukjanov | malini, added to the giant length backlog | 15:46 |
| malini | SergeyLukjanov: hope the other giants are small :) | 15:47 |
| *** esker has quit IRC | 15:47 | |
| malini | if any other core reviewers need good karma, here is a one line change https://review.openstack.org/#/c/78392/ !! | 15:48 |
| *** johnthetubaguy1 has joined #openstack-infra | 15:50 | |
| jeblair | rcarrillocruz, clarkb: i left my suggetion in https://bugs.launchpad.net/openstack-ci/+bug/1235335 for the record, but i don't think trying to implement that is low-hanging-fruit. | 15:50 |
| rcarrillocruz | cool, thx sir, i'll try it and depending on how hard it gets i'll go forward or ditch | 15:51 |
| *** yamahata has quit IRC | 15:52 | |
| *** johnthetubaguy has quit IRC | 15:53 | |
| jeblair | zaro, hashar: is current jjb HEAD a good place to tag a release? | 15:53 |
| *** salv-orlando_ has joined #openstack-infra | 15:54 | |
| * hashar wishes Gerrit could let us review tagging | 15:54 | |
| jeblair | we all wish that :/ | 15:55 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: satori: Drop requirements, pypi, and old upstream https://review.openstack.org/79975 | 15:55 |
| mordred | hashar: ++ | 15:56 |
| hashar | jeblair: the current jjb is nice. We got a few nasty changes pending such as using threads and pbr | 15:56 |
| *** salv-orlando has quit IRC | 15:56 | |
| *** salv-orlando_ is now known as salv-orlando | 15:56 | |
| hashar | so maybe cutting 0.7.0 before the nasty changes land in is a good idea | 15:56 |
| *** sweston has quit IRC | 15:56 | |
| mordred | hashar: I am highly supportive of any patches to gerrit which add tag reviewing | 15:56 |
| hashar | plus last release was back in November | 15:56 |
| hashar | mordred: I am java illiterate and Wikimedia mostly tag mediawiki/core which is already pair reviewed via a release process. So unlikely to happen on our side | 15:57 |
| hashar | :-( | 15:57 |
| *** wenlock has joined #openstack-infra | 15:57 | |
| zns | sdague, fungi: thanks for the help. I'm updating satori's config accordingly: https://review.openstack.org/#/c/79975/ | 15:57 |
| *** rpodolyaka has joined #openstack-infra | 15:58 | |
| mordred | hashar: sniff. sniff | 15:58 |
| hashar | jeblair: maybe ping openstack-infra list about jjb releasing? I am sure all core maintainers are subscribed to it. | 15:58 |
| *** che-arne has joined #openstack-infra | 15:58 | |
| jeblair | hashar: zaro asked for it last week (he got a request from a packager i think) | 15:58 |
| jeblair | so i'm pretty sure he's on board | 15:58 |
| *** gyee has quit IRC | 15:59 | |
| *** rpodolyaka has quit IRC | 16:00 | |
| *** reed has joined #openstack-infra | 16:00 | |
| *** rpodolyaka has joined #openstack-infra | 16:00 | |
| hashar | for logging purpose: +1 on releasing a new JJB version :-] | 16:00 |
| *** CaptTofu has quit IRC | 16:00 | |
| *** pblaho has quit IRC | 16:00 | |
| hashar | one day we would have to craft a nice ChangeLog for end users consumption | 16:01 |
| *** CaptTofu has joined #openstack-infra | 16:01 | |
| jeblair | sent email | 16:01 |
| *** Sukhdev has joined #openstack-infra | 16:02 | |
| *** salv-orlando has quit IRC | 16:02 | |
| *** hogepodge has quit IRC | 16:02 | |
| jeblair | also, i -1'd the idea of a jjb-ptl group that can tag before, but that was when i was able to keep more current on what's going on with it. it's clear that i'm not as up to date now, so i'm okay with expanding that group now. | 16:03 |
| *** yassine has quit IRC | 16:03 | |
| hashar | Khai is doing an awesome job at reviewing/approving the changes | 16:04 |
| jeblair | https://review.openstack.org/#/c/56823/ has my +2 now | 16:04 |
| *** CaptTofu has quit IRC | 16:05 | |
| *** SumitNaiksatam has quit IRC | 16:07 | |
| *** talluri has quit IRC | 16:07 | |
| hashar | jeblair: added the rest of JJB approver to the change and +1ed it :-] | 16:11 |
| hashar | I like seeing delegation | 16:11 |
| sdague | jeblair: what is your feeling on making a functional test in the gate which only applied to a single project which did active clones outside of zuul? | 16:16 |
| *** yolanda has quit IRC | 16:16 | |
| sdague | because it turns out that grenade ends up breaking for non gate cases quite a bit | 16:16 |
| sdague | because we do so much overwriting of config | 16:17 |
| sdague | realizing that we'd have a failure rate based on real cloning associated with it | 16:17 |
| sdague | but it would apply to only the grenade project | 16:17 |
| jeblair | sdague: can you describe the problem a bit more? i'm not sure i understand | 16:17 |
| sdague | grenade master doesn't work | 16:18 |
| sdague | because it actually sets up a set of git clones that don't work | 16:18 |
| sdague | because in the gate all of that is done by d-g | 16:18 |
| *** vkozhukalov has joined #openstack-infra | 16:18 | |
| jeblair | how don't they work? | 16:19 |
| sdague | they turn out to be completely invalid git urls | 16:19 |
| *** dstanek_afk has joined #openstack-infra | 16:19 | |
| sdague | because novnc isn't on git.openstack.org, for instance | 16:20 |
| *** dstanek has quit IRC | 16:20 | |
| sdague | and all of devstack trees are controled by a GIT_BASE | 16:20 |
| sdague | which got changed to git.openstack.org in devstack master | 16:21 |
| sdague | and grenade master | 16:21 |
| sdague | but defaults to github.com in devstack stable/havana | 16:21 |
| sdague | so things go funny | 16:22 |
| sdague | also, turns out TARGET_BRANCH was defaulting to stable/icehouse | 16:22 |
| *** pbelanyi has quit IRC | 16:22 | |
| sdague | which, also, never tested, because d-g sets all that | 16:22 |
| jeblair | sdague: so in order to test what you want, the test would need to clone from github, since you want to make sure the novnc url works? | 16:23 |
| sdague | jeblair: yep | 16:23 |
| *** nati_ueno has quit IRC | 16:23 | |
| sdague | which is why it would be limited to grenade, because the failure rate that would introduce would be non zero | 16:23 |
| jeblair | sdague: having been on the receiving end of "why did this test break" thanks to github, i have to admit, i'm not excited about that. | 16:23 |
| sdague | sure, however the alternative is "why hasn't grenade worked for normal people in 4 months" | 16:24 |
| jeblair | sdague: could we unit test it instead? (make sure these urls or git commands look sane?) | 16:24 |
| *** dstanek_afk has quit IRC | 16:24 | |
| sdague | jeblair: probably, but that just means we know the right assumptions, not that things actually work | 16:25 |
| sdague | so if upstream trees change, we break | 16:25 |
| *** melwitt has joined #openstack-infra | 16:25 | |
| *** gokrokve_ has quit IRC | 16:25 | |
| sdague | I think the crux of this is a non negligle part of grenade is to actually set up the tree structures | 16:25 |
| sdague | which overlaps with d-g | 16:25 |
| sdague | so is untested on it's own | 16:26 |
| jeblair | so maybe a non-voting test? | 16:26 |
| sdague | sure | 16:26 |
| jeblair | so that we're not signing up the gate to be broken by github | 16:26 |
| sdague | jeblair: well, it wouldn't be in the gate | 16:26 |
| jeblair | (i mean, basically, you would not have been able to land _any_ changes to grenade yesterday) | 16:26 |
| sdague | it would be on check only for grenade | 16:27 |
| jeblair | sdague: check ~= gate now, at least for this purpose | 16:27 |
| *** rpodolyaka has quit IRC | 16:27 | |
| sdague | well, check doesn't kill someone else | 16:27 |
| SergeyLukjanov | Houston, we have a problem.... | 16:27 |
| SergeyLukjanov | https://git.openstack.org/cgit/openstack/sahara is empty O_O | 16:27 |
| jeblair | i mean, continuing to use yesterday as an example, no grenade check tests would have passed if this were voting | 16:27 |
| sdague | so it's more limitted pain | 16:27 |
| *** wenlock has quit IRC | 16:27 | |
| sdague | jeblair: this job would not have passed, agreed | 16:28 |
| SergeyLukjanov | fungi, jeblair, clarkb, could you please take a look on gerrit mirroring? https://git.openstack.org/cgit/openstack/sahara is empty... | 16:28 |
| mgagne | SergeyLukjanov: showing fine to me here | 16:28 |
| jeblair | SergeyLukjanov: i will trigger gerrit replication | 16:28 |
| jeblair | SergeyLukjanov: you might want to check each of "git01" .. "git05" to narrow down the problem | 16:28 |
| *** rpodolyaka has joined #openstack-infra | 16:29 | |
| *** wenlock has joined #openstack-infra | 16:29 | |
| SergeyLukjanov | jeblair, yup, /me checking | 16:29 |
| fungi | that's definitely weird. i did mv the backing directory for it on all the git servers | 16:29 |
| jeblair | SergeyLukjanov: like http://git01.openstack.org:8080/cgit/openstack/sahara | 16:29 |
| fungi | oh! | 16:29 |
| fungi | i forgot git05 | 16:29 |
| fungi | i forgot _we had_ git05 :/ | 16:29 |
| * fungi sighs | 16:29 | |
| jeblair | fungi: yep that's the one | 16:30 |
| jeblair | http://git05.openstack.org:8080/cgit/openstack/sahara | 16:30 |
| SergeyLukjanov | there are no sahara repos on git05 | 16:30 |
| *** CaptTofu has joined #openstack-infra | 16:30 | |
| *** alexpilotti has joined #openstack-infra | 16:30 | |
| SergeyLukjanov | and savanna too | 16:30 |
| SergeyLukjanov | :) | 16:30 |
| fungi | at this point the repos have all been created on there, but if you haven't triggered replication yet i can remove them and move the old ones to the right nams | 16:31 |
| fungi | names | 16:31 |
| jeblair | fungi: i have not triggered replication | 16:31 |
| fungi | doing it now--thanks! | 16:31 |
| SergeyLukjanov | fungi, jeblair, thank you! | 16:31 |
| *** Ryan_Lane1 has joined #openstack-infra | 16:33 | |
| fungi | all removed/moved. retriggering replication for them now | 16:35 |
| *** dstanek_afk has joined #openstack-infra | 16:35 | |
| *** jlibosva has quit IRC | 16:36 | |
| *** Ryan_Lane has quit IRC | 16:36 | |
| *** Ryan_Lane1 is now known as Ryan_Lane | 16:36 | |
| *** Ryan_Lane has joined #openstack-infra | 16:36 | |
| *** Ryan_Lane1 has joined #openstack-infra | 16:37 | |
| fungi | looks like it should all be replicated and up to date now? | 16:39 |
| pleia2 | clarkb: just had one comment re: 79814 (not actually a blocker, but might consider it) | 16:40 |
| *** bhuvan has joined #openstack-infra | 16:41 | |
| *** bhuvan_ has joined #openstack-infra | 16:41 | |
| *** khyati has joined #openstack-infra | 16:41 | |
| *** amcrn has joined #openstack-infra | 16:42 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/publications: update slides https://review.openstack.org/79993 | 16:44 |
| *** reed has quit IRC | 16:44 | |
| *** dizquierdo has quit IRC | 16:45 | |
| *** gyee has joined #openstack-infra | 16:49 | |
| *** jasond has quit IRC | 16:50 | |
| *** gokrokve has joined #openstack-infra | 16:50 | |
| *** rcarrillocruz1 has joined #openstack-infra | 16:51 | |
| *** SumitNaiksatam has joined #openstack-infra | 16:51 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack/requirements: Rename Savanna to Sahara https://review.openstack.org/79994 | 16:51 |
| SergeyLukjanov | fungi, clarkb, jeblair, sdague, folks, could you please fast approve https://review.openstack.org/79994 to make sahara jobs running using pypi.o.o | 16:52 |
| *** thuc has joined #openstack-infra | 16:52 | |
| *** lcostantino has joined #openstack-infra | 16:53 | |
| *** rcarrillocruz has quit IRC | 16:53 | |
| fungi | SergeyLukjanov: approved | 16:53 |
| *** thuc_ has joined #openstack-infra | 16:53 | |
| SergeyLukjanov | fungi, thank you! | 16:53 |
| clarkb | morning | 16:54 |
| SergeyLukjanov | clarkb, good morning | 16:54 |
| fungi | SergeyLukjanov: i'm just sorry i didn't think to suggest a patch to that file too | 16:54 |
| clarkb | pleia2: lookig thanks | 16:54 |
| *** e0ne has quit IRC | 16:54 | |
| SergeyLukjanov | fungi, my bad, absolutely forget about the dep on it for choosing mirrors | 16:54 |
| fungi | SergeyLukjanov: i think it would probably have also broken requirements update proposals too | 16:55 |
| SergeyLukjanov | fungi, yeah | 16:55 |
| *** thuc has quit IRC | 16:56 | |
| *** thuc has joined #openstack-infra | 16:57 | |
| *** cody-somerville has quit IRC | 16:58 | |
| *** thuc_ has quit IRC | 17:00 | |
| *** cody-somerville_ has joined #openstack-infra | 17:03 | |
| *** cody-somerville_ has quit IRC | 17:03 | |
| *** cody-somerville_ has joined #openstack-infra | 17:03 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Remove unneeded package sources from nodepool node https://review.openstack.org/79814 | 17:03 |
| clarkb | pleia2: ^ I think that should do it | 17:03 |
| *** harlowja_away is now known as harlowja | 17:04 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/statusbot: Sleep 1 second https://review.openstack.org/79956 | 17:05 |
| *** nati_ueno has joined #openstack-infra | 17:06 | |
| jeblair | oh wow, i just learned you can "yum install /path/to/binary/you/want" | 17:07 |
| jeblair | and that will install the package that provides it | 17:07 |
| clarkb | jeblair: and you don't even need to do a package list update first :) | 17:08 |
| SergeyLukjanov | jeblair, oh, interesting | 17:08 |
| SergeyLukjanov | clarkb, bah, fantastic | 17:09 |
| *** afazekas has joined #openstack-infra | 17:09 | |
| *** afazekas has quit IRC | 17:09 | |
| fungi | neat. better than using apt-file search | 17:09 |
| *** gokrokve_ has joined #openstack-infra | 17:09 | |
| *** jasondotstar has joined #openstack-infra | 17:09 | |
| *** sweston has joined #openstack-infra | 17:09 | |
| *** reed has joined #openstack-infra | 17:10 | |
| dstufft | jeblair: that is awesome | 17:11 |
| *** andreaf has joined #openstack-infra | 17:11 | |
| clarkb | dtroyer: Alex_Gaynor for the openstack sdks' ptl group who should be the initial member | 17:12 |
| *** gokrokve has quit IRC | 17:12 | |
| fungi | clarkb: added a few more comments on 79814 | 17:12 |
| Alex_Gaynor | clarkb: Jesse Noller I suppose | 17:13 |
| *** morganfainberg_Z is now known as morganfainberg | 17:13 | |
| clarkb | Alex_Gaynor: thanks | 17:13 |
| Alex_Gaynor | clarkb: sure | 17:13 |
| fungi | dstufft: did you get my ping the other night that i put your keys on those sample servers where we're seeing discrepancies in pip behavior wrt https://github.com/pypa/pip/issues/1632 | 17:13 |
| dstufft | fungi: oh I missed it | 17:14 |
| dstufft | and then forgot | 17:14 |
| dstufft | :[ | 17:14 |
| *** VijayTripathi has quit IRC | 17:14 | |
| dstufft | sorry! | 17:14 |
| clarkb | fungi: responded. I can fix the comment but we actually need sudo there because distros | 17:14 |
| Alex_Gaynor | dstufft: lol. Have fun debugging that | 17:15 |
| fungi | dstufft: 'ssh jenkins@23.253.82.112' is where we're seeing it break (rackspace) and 'ssh jenkins@15.185.123.189' is where we're seeing it work normally (hpcloud) | 17:15 |
| clarkb | fungi: can you take a look before I push a new patchset? | 17:15 |
| fungi | clarkb: looking | 17:15 |
| *** esker has joined #openstack-infra | 17:16 | |
| zaro | clarkb: i tried verifying this on my personal VM but i don't know how to setup a reverse proxy. https://review.openstack.org/#/c/60893/ | 17:16 |
| *** saju_m has joined #openstack-infra | 17:16 | |
| clarkb | zaro: just install gerrit with that puppet change | 17:17 |
| dstufft | fungi: Ok, i'm fixing my devenv since homebrew and apple decided to conspire to destroy my productivity today | 17:17 |
| clarkb | zaro: that will setup apache and everything | 17:17 |
| dstufft | I'll look at it after that | 17:17 |
| fungi | dstufft: appreciated--and good luck with the fruit | 17:17 |
| dstufft | hmm | 17:18 |
| dstufft | you know what though | 17:18 |
| dstufft | I could totally use my eldritch-horror branch on pip and make it so pip just assumes setuptools is a setup_requires and installs it in a temp location :V | 17:18 |
| *** kmartin has quit IRC | 17:18 | |
| dstufft | then this shouldn't be possible to happen | 17:18 |
| fungi | clarkb: lgtm | 17:19 |
| zaro | clarkb: i don't think i even have apache installed. don't even have the /etc/apache folder. | 17:19 |
| fungi | dstufft: i wanted to compliment you on your choice of branch naming there, if nothing else. very nice ;) | 17:19 |
| clarkb | fungi: thanks | 17:19 |
| dstufft | fungi: the first revision was worse than the current revision :V | 17:20 |
| zaro | clarkb: is it just a straight forward install of apache with those changes? | 17:20 |
| dstufft | the current one is not entirely super horrible | 17:20 |
| fungi | dstufft: i assume while it's still named eldritch-horror, i should expect to be driven mad simply by a mere glimpse of the source code there | 17:20 |
| clarkb | zaro: mostly. but you can have puppet do it all for you | 17:21 |
| *** afazekas has joined #openstack-infra | 17:21 | |
| dstufft | fungi: https://github.com/dstufft/pip/compare/pypa:develop...eldritch-horror monkeypatching via python -c "" is the best ever | 17:21 |
| zaro | clarkb: i forgot how. i'll be in tomorrow, just i'll just postpone until then. | 17:22 |
| fungi | dstufft: you know now that i've clicked, i'm going to live out the remainder of my days gibbering in a corner, mumbling incoherently about ancient computer gods from other dimensions | 17:23 |
| zaro | clarkb: have you taken a look at jeblair comment on https://review.openstack.org/#/c/61321/ | 17:23 |
| andreaf | sdague: ping | 17:24 |
| clarkb | zaro: not yet, still digging out of the remains of bug day | 17:24 |
| * clarkb is going to fix release two more bugs | 17:24 | |
| fungi | dstufft: also always a fan of commit messages like "Make this less bad" | 17:24 |
| dstufft | fungi: :3 | 17:25 |
| dstufft | fungi: it's cool though, it actually works (not options like --cert are passed through yet), no more setuptools touching the network with that branch | 17:25 |
| fungi | nice that pip source is growing a "hacks" module too | 17:26 |
| fungi | says a lot right there about the fright level of monkey-patching there | 17:27 |
| dstufft | a lot of pip could justifably be moved into that module | 17:27 |
| openstackgerrit | A change was merged to openstack-infra/statusbot: Sleep 1 second https://review.openstack.org/79956 | 17:27 |
| clarkb | jeblair: fungi SergeyLukjanov mordred I have two nodepool script changes that need an additional +2 and +A. I can babysit some image builds if I can get reviewers | 17:28 |
| SergeyLukjanov | clarkb, you get one | 17:29 |
| fungi | clarkb: i think i already +2'd them both | 17:29 |
| *** sarob has joined #openstack-infra | 17:30 | |
| clarkb | fungi: you did thanks | 17:30 |
| clarkb | zaro: I think jeblair is right. We need to make sure that line 30 of https://review.openstack.org/#/c/61321/12/modules/openstack_project/templates/nodepool/nodepool-dev.yaml.erb runs a script that will install the correct key for jenkins-dev/nodepool-dev | 17:31 |
| *** jcoufal has quit IRC | 17:31 | |
| *** krotscheck has joined #openstack-infra | 17:32 | |
| jeblair | clarkb: aprvd | 17:32 |
| clarkb | jeblair: awesome thank you | 17:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Remove unneeded package sources from nodepool node https://review.openstack.org/79814 | 17:32 |
| *** dprince has quit IRC | 17:34 | |
| *** kmartin has joined #openstack-infra | 17:34 | |
| *** sarob has quit IRC | 17:34 | |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Add an import whitelist helper factory https://review.openstack.org/80018 | 17:37 |
| *** jepoy has quit IRC | 17:37 | |
| *** jepoy has joined #openstack-infra | 17:38 | |
| *** openstackstatus has quit IRC | 17:38 | |
| *** openstackstatus has joined #openstack-infra | 17:38 | |
| *** julienvey has quit IRC | 17:39 | |
| *** jepoy has quit IRC | 17:42 | |
| malini | thanks SergeyLukjanov for the review | 17:44 |
| malini | fungi: Can you take a look at this if you have a minute https://review.openstack.org/#/c/78392/ ? | 17:45 |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Add an import whitelist helper factory https://review.openstack.org/80018 | 17:47 |
| fungi | malini: getting sdague or dtroyer to weigh in on that would be good too, before we possibly approve it | 17:48 |
| malini | sure fungi! | 17:48 |
| malini | dtroyer, sdague: Can you take a look please https://review.openstack.org/#/c/78392/ ? | 17:49 |
| fungi | i can evaluate the (very trivial) change on its technical correctness, but they're the ones who know whether that's going to cause any inconvenience for them | 17:49 |
| malini | sure..makes sense | 17:49 |
| malini | dtroyer, sdague: This is to make the marconi devstack troubleshooting easier | 17:50 |
| dtroyer | malini, fung: lgtm | 17:51 |
| malini | thanks dtroyer!! | 17:51 |
| malini | fungi: ^ | 17:51 |
| *** jpich has quit IRC | 17:52 | |
| *** rpodolyaka has quit IRC | 17:52 | |
| *** bhuvan__ has joined #openstack-infra | 17:53 | |
| *** bhuvan___ has joined #openstack-infra | 17:53 | |
| *** dstanek_afk is now known as dstanek | 17:53 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Rename migration revisions https://review.openstack.org/79869 | 17:54 |
| *** bhuvan_ has quit IRC | 17:54 | |
| *** bhuvan has quit IRC | 17:54 | |
| clarkb | uh | 17:54 |
| *** hashar has quit IRC | 17:54 | |
| clarkb | so you may as well use migrate if you are going to do ^ | 17:54 |
| openstackgerrit | A change was merged to openstack-infra/config: Add experimental job for Marconi https://review.openstack.org/78392 | 17:55 |
| clarkb | iirc alembic uses proper uuid like ids because it can build arbitrary graphs | 17:55 |
| clarkb | rather than doing the "human readable" thing that migrate does | 17:55 |
| *** rpodolyaka has joined #openstack-infra | 17:56 | |
| *** rpodolyaka has quit IRC | 17:57 | |
| *** reed has quit IRC | 17:57 | |
| *** reed has joined #openstack-infra | 17:57 | |
| *** rlandy_ has joined #openstack-infra | 17:57 | |
| fungi | clarkb: jeblair: mordred: you'll be amused to know that i'm getting eoferror (or sometimes connection reset) repeatedly retrying to update the devstack-precise image in rax-iad now ;) | 17:58 |
| clarkb | fungi: ya I ran into that when sudo went sideways | 17:58 |
| *** vkozhukalov has quit IRC | 17:58 | |
| *** bhuvan_ has joined #openstack-infra | 17:58 | |
| *** bhuvan has joined #openstack-infra | 17:58 | |
| fungi | got it once or twice in dfw and ord too but they eventually completed after once or twice | 17:59 |
| fungi | completed after one or two tries i mean | 17:59 |
| jeblair | :/ | 17:59 |
| *** bhuvan___ has quit IRC | 17:59 | |
| *** bhuvan__ has quit IRC | 17:59 | |
| clarkb | fungi: I think we may have leaked nodes too. I meant to look at that toay | 18:00 |
| fungi | i was actually about to do the same, and noticed that my iad image still wasn't updated | 18:00 |
| *** rlandy has quit IRC | 18:00 | |
| *** rpodolyaka has joined #openstack-infra | 18:01 | |
| clarkb | there are a bunch of ~140 hour old ready nodes | 18:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Limit non tripleo nodepool nodes to 8GB of RAM https://review.openstack.org/78440 | 18:01 |
| *** bhuvan___ has joined #openstack-infra | 18:01 | |
| *** bhuvan__ has joined #openstack-infra | 18:01 | |
| fungi | clarkb: yeah, i'm looking at those. some in other states past 3 hours as well | 18:01 |
| fungi | i'll give 'em all a quick death | 18:01 |
| clarkb | nodepool list | grep ready | sort -t'|' -k11 -n | 18:02 |
| fungi | the 140-hour-old ones are dsvm-precise-krnl | 18:03 |
| *** bhuvan_ has quit IRC | 18:03 | |
| *** bhuvan has quit IRC | 18:03 | |
| fungi | more of those than i would have expected | 18:03 |
| fungi | i omitted any in a hold state and am deleting any others which are >3 hours in any given state | 18:05 |
| *** Sukhdev has quit IRC | 18:06 | |
| clarkb | woot ram change merged. I am going to spin up a new hpcloud region b devstack-precise-check image | 18:06 |
| clarkb | fungi: that doesn't interfere with what you are doing does it? | 18:06 |
| fungi | clarkb: not in the least | 18:06 |
| *** derekh has quit IRC | 18:07 | |
| *** jgallard has quit IRC | 18:09 | |
| SergeyLukjanov | folks, I've just find that we need to have etc/sahara/sahara.conf in repo for d-g, will symlink work well or I need to copy-paste conf sample? | 18:09 |
| SergeyLukjanov | https://review.openstack.org/80038 | 18:09 |
| clarkb | SergeyLukjanov: I think copy paste then you can delete the old one after a release or two, but then you have to double account which sucks | 18:10 |
| clarkb | SergeyLukjanov: also symlinks aren't a thing on windows. not sure if that matteers | 18:10 |
| sdague | given how pegged we are on dsvm nodes, thoughts on merging this - https://review.openstack.org/#/c/78687/ | 18:11 |
| fungi | clarkb: windows has "shortcuts" instead (don't ask why i know this) | 18:11 |
| sdague | which drops jobs we don't want | 18:11 |
| sdague | jogo: can you rebase that? | 18:11 |
| SergeyLukjanov | clarkb, we're planning to not keep backward compat for such stuff, so, I'll add temp copy-paste | 18:11 |
| SergeyLukjanov | clarkb, thanks | 18:11 |
| *** zns has quit IRC | 18:11 | |
| SergeyLukjanov | clarkb, we just need it to merged updated devstack integration and then rename all internals | 18:12 |
| clarkb | SergeyLukjanov: wfm | 18:12 |
| SergeyLukjanov | clarkb, could I ask you to forse push it? https://review.openstack.org/80038 | 18:13 |
| clarkb | SergeyLukjanov: is it not possible to run it through the gate normally? (would be good to see it actually fix the problem) | 18:14 |
| SergeyLukjanov | clarkb, devstack integration update isn't merged yet :( | 18:14 |
| SergeyLukjanov | clarkb, here is an issue - http://logs.openstack.org/53/79353/3/check/check-tempest-dsvm-sahara-full-nv/f4580cf/logs/devstacklog.txt.gz#_2014-03-12_17_27_25_560 | 18:15 |
| *** bhuvan has joined #openstack-infra | 18:15 | |
| *** bhuvan_ has joined #openstack-infra | 18:15 | |
| clarkb | SergeyLukjanov: right but won't 80038 fix ^ without needing a force push? | 18:15 |
| clarkb | jgriffith: you got change 80000 I am jealous | 18:16 |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Add an import whitelist helper factory https://review.openstack.org/80018 | 18:16 |
| *** wenlock has quit IRC | 18:16 | |
| *** bhuvan___ has quit IRC | 18:16 | |
| *** bhuvan__ has quit IRC | 18:16 | |
| jeblair | sdague: lgtm | 18:16 |
| jeblair | SergeyLukjanov: where's the devstack change? | 18:16 |
| SergeyLukjanov | clarkb, I can't merge it to sahara because d-g job is voting | 18:16 |
| SergeyLukjanov | https://review.openstack.org/#/c/79353/ | 18:16 |
| SergeyLukjanov | jeblair, ^^ | 18:16 |
| clarkb | oh I see 79353 depends on 80038 and vice versa | 18:17 |
| openstackgerrit | Nikola Dipanov proposed a change to openstack-dev/hacking: Add an import whitelist helper factory https://review.openstack.org/80018 | 18:17 |
| *** bhuvan_ has quit IRC | 18:17 | |
| *** bhuvan has quit IRC | 18:17 | |
| clarkb | I think I am ok force pushing the sahara change since it is just copy pasta. jeblair fungi mordred any dissent? | 18:17 |
| jeblair | https://review.openstack.org/#/c/79353/ depends on 80038? | 18:17 |
| SergeyLukjanov | clarkb, yeah, we've added all needed aliases except conf file I hope | 18:18 |
| clarkb | jeblair: that is my understanding since it is trying to setup sahara etc files not savanna etc files | 18:18 |
| *** bhuvan has joined #openstack-infra | 18:18 | |
| *** bhuvan_ has joined #openstack-infra | 18:18 | |
| clarkb | jeblair: we could probably split it out into a couple changes that use savanna etc fiels with sahara then sahara etc files with sahara | 18:18 |
| jeblair | is 79353 overreaching then? | 18:18 |
| jeblair | clarkb: yeah | 18:18 |
| clarkb | jeblair: ya it may be ^ | 18:18 |
| SergeyLukjanov | clarkb, that's an option too | 18:19 |
| fungi | clarkb: seems fine, though note that this morning before we started the rename SergeyLukjanov had already discovered that the devstack fix to start properly overriding from global-requirements.txt may have put savanna in a broken state, so until it's determined what is broken there there may be more forced merges to come | 18:19 |
| clarkb | SergeyLukjanov: I think we should go that route then | 18:19 |
| clarkb | SergeyLukjanov: its more CD friendly and all that | 18:19 |
| SergeyLukjanov | clarkb, ok, it's fair | 18:19 |
| SergeyLukjanov | heh, looks like +10h of looking on zuul status page ;) | 18:19 |
| *** sarob has joined #openstack-infra | 18:20 | |
| *** rpodolyaka has quit IRC | 18:20 | |
| clarkb | hpcloud region b image update is running now | 18:21 |
| jeblair | SergeyLukjanov: ? | 18:22 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Minor fixes in the trigger documentation https://review.openstack.org/79854 | 18:22 |
| SergeyLukjanov | jeblair, you're about "+10h of looking on zuul status page ;)"? | 18:22 |
| jeblair | yeah | 18:22 |
| jeblair | i didn't understand that | 18:22 |
| *** rpodolyaka has joined #openstack-infra | 18:23 | |
| SergeyLukjanov | jeblair, I mean that I'll need to wait for several more CRs to be checked by Jenkins :) | 18:23 |
| *** bhuvan has quit IRC | 18:24 | |
| *** bhuvan_ has quit IRC | 18:24 | |
| jeblair | SergeyLukjanov: yeah; jobs at the top of the check queue look like they waited about 30 mins to get nodes | 18:24 |
| *** ildikov_ has quit IRC | 18:25 | |
| mtreinish | jogo: so on https://review.openstack.org/#/c/79123/ should I respin it to just be nova, infra, and qa? | 18:28 |
| *** SumitNaiksatam has quit IRC | 18:28 | |
| *** e0ne has joined #openstack-infra | 18:29 | |
| *** saju_m has quit IRC | 18:30 | |
| *** esker has quit IRC | 18:30 | |
| *** bhuvan has joined #openstack-infra | 18:30 | |
| *** afazekas has quit IRC | 18:30 | |
| *** esker has joined #openstack-infra | 18:30 | |
| *** sarob has quit IRC | 18:30 | |
| SergeyLukjanov | jeblair, done, could you please take a look if you have some time https://review.openstack.org/79353 | 18:31 |
| *** zns has joined #openstack-infra | 18:32 | |
| *** bhuvan has quit IRC | 18:32 | |
| jeblair | SergeyLukjanov: lgtm; sdague ^ | 18:33 |
| reed | where can I find a list of all the web properties managed by the CI team? | 18:33 |
| clarkb | reed: openstack-infra/config/manifests/site.pp is probably the closest thing | 18:34 |
| reed | clarkb, wonderful, that's what I was looking for | 18:34 |
| jeblair | clarkb: a possibly incomplete list is in our docs (it omits some smaller things): http://ci.openstack.org/ | 18:34 |
| jeblair | reed: ^ | 18:34 |
| *** homeless_ has joined #openstack-infra | 18:35 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/devstack-gate: WIP: test caching-forwarding nameserver https://review.openstack.org/80045 | 18:35 |
| reed | yep, thanks | 18:35 |
| *** homeless has quit IRC | 18:35 | |
| *** sarob has joined #openstack-infra | 18:36 | |
| SergeyLukjanov | jeblair, thx | 18:36 |
| SergeyLukjanov | russellb, re https://review.openstack.org/#/c/79900/3 | 18:36 |
| clarkb | http://paste.openstack.org/show/73279/ huh | 18:37 |
| clarkb | I am going to retry the image build to see if this is consistent. May just be that puppet forge said go away | 18:37 |
| SergeyLukjanov | russellb, you should add section about nova-docker to modules/openstack_project/files/jenkins_job_builder/config/porjects.yaml to generate jobs in jenkins | 18:37 |
| jeblair | clarkb: yeah | 18:38 |
| *** wenlock_demo has joined #openstack-infra | 18:38 | |
| jogo | mtreinish: that was my thinking | 18:38 |
| *** sweston has quit IRC | 18:39 | |
| *** cody-somerville_ is now known as cody-somerville | 18:39 | |
| *** wenlock_demo has quit IRC | 18:39 | |
| *** rlandy__ has joined #openstack-infra | 18:39 | |
| *** SumitNaiksatam has joined #openstack-infra | 18:39 | |
| jeblair | jogo, mtreinish: do other channels want it? | 18:39 |
| *** wenlock has joined #openstack-infra | 18:40 | |
| *** jasondotstar has quit IRC | 18:40 | |
| mtreinish | jeblair: dunno, jogo asked me to push out the patch | 18:40 |
| mtreinish | jeblair: do you want it for infra? | 18:40 |
| jeblair | i ask because we could have had gerritbot announcing failures for years, but no one has fixed that... that makes me think people aren't interested in failure announcements | 18:40 |
| *** lcostantino has quit IRC | 18:40 | |
| *** sdake has quit IRC | 18:41 | |
| jeblair | i mean, i do think it's extremely useful for people tending e-r, _especially_ the unclassified announcements | 18:41 |
| clarkb | well someone did try, but then it got more complicated than we realized | 18:41 |
| jamespage | dims_, was really hoping to have libvirt 1.2.2 + patch into the icehouse CA today | 18:41 |
| mtreinish | jeblair: well this is a bit more specific it will only announce the failures with a known bug and only if the bug matches the projects listed in the yaml for the channel | 18:41 |
| jamespage | however debugging a unit test failure in it right now | 18:41 |
| jamespage | so lagging a bit | 18:41 |
| *** wenlock has quit IRC | 18:42 | |
| *** sdake_ has quit IRC | 18:42 | |
| jeblair | mtreinish: oh, so it's like "hey this nova bug keeps hitting stuff" | 18:42 |
| *** lcostantino has joined #openstack-infra | 18:42 | |
| fungi | paste.o.o is grinding. i bet it's swapping hard again | 18:42 |
| mtreinish | jeblair: yeah | 18:42 |
| *** sweston has joined #openstack-infra | 18:43 | |
| *** rlandy_ has quit IRC | 18:43 | |
| jogo | jeblair: my thinking was others will but push this out on phases | 18:43 |
| fungi | actually it looks fine | 18:43 |
| clarkb | fungi: I think we should make it read only and let it die | 18:43 |
| jogo | and I as nova want it | 18:43 |
| clarkb | fungi: I think drizzle is just slow | 18:43 |
| *** krotscheck has quit IRC | 18:43 | |
| clarkb | because old drizzle and so on | 18:43 |
| mtreinish | jogo: well I was thinking the first patch it might make sense to include neutron and cinder too | 18:44 |
| jeblair | mtreinish: i could see that being useful then; though that makes me think we might not want it in infra -- we have a lot of unfixable bugs, so much of what it would say to us is unactionable | 18:44 |
| jogo | mtreinish: your call | 18:44 |
| *** sdake_ has joined #openstack-infra | 18:44 | |
| jeblair | mtreinish: (example: ubuntu mirror is wonky) | 18:44 |
| clarkb | there are so many paste services out that that all support basically the same functionality I am not sure we need our own. The only reason to keep it would be to avoid a paste provider going away causing everyone to not have paste but there are a lot of options (I suppose running an open version of a thing is valuable too) | 18:44 |
| jogo | jeblair: good point about infra bugs. mtreinish ^ | 18:44 |
| mtreinish | jeblair: you don't want e-r to spam the channel for every patch when there is a mirror outage :) | 18:45 |
| *** sdake has joined #openstack-infra | 18:45 | |
| clarkb | jeblair: the puppet module thing happened again. I think I am going to get lunch and try again after | 18:45 |
| *** andreaf has quit IRC | 18:45 | |
| jeblair | clarkb: ack. also, i think it's worth spending a few minutes fixing paste. | 18:45 |
| *** weshay has quit IRC | 18:45 | |
| clarkb | mordred: ^ have you seen hpcloud region b connectivty issues to puppet forge | 18:45 |
| mordred | uhm. no | 18:45 |
| clarkb | jeblair: definitely. I don't think we should kill that service without some headroom | 18:45 |
| mtreinish | jogo: yeah I didn't think about that, we probably should exclude infra then | 18:45 |
| clarkb | jeblair: just throwing it out there that I think the value of having our own is minimal | 18:46 |
| clarkb | mordred: jeblair actually I got more error message this time. I think we may have conflicting requirements somewhere | 18:46 |
| clarkb | :( I will look into fixing that after lunch | 18:46 |
| *** rpodolyaka has quit IRC | 18:47 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/config: Elastic recheck irc spam everywhere https://review.openstack.org/79123 | 18:48 |
| mtreinish | jogo: ^^^ just nova, cinder, and neutron | 18:48 |
| fungi | those most deserving of its ire? | 18:48 |
| mtreinish | russellb, markmcclain, jgriffith: ^^^ do you guys have an issue with that? | 18:48 |
| *** e0ne has quit IRC | 18:49 | |
| markmcclain | +1 | 18:49 |
| dims_ | jamespage, very cool | 18:49 |
| mtreinish | fungi: heh, yeah something like that :) | 18:49 |
| dims_ | jamespage, the 0.9.8+ deb(s) are holding up very well. need to run it another 5-8 times (i have 5-6 good runs so far) | 18:49 |
| mtreinish | fungi: that or most likely to respond to an e-r comment in irc | 18:50 |
| fungi | mtreinish: ooh, burn ;) | 18:50 |
| jeblair | fungi, clarkb: so my thinking is recursive resolvers on all servers, and forwarding resolvers on nodepool nodes. how does that sound? | 18:52 |
| jeblair | fungi, clarkb: https://review.openstack.org/#/c/80045/ is an attempt to see if that would mess up devstack | 18:52 |
| fungi | jeblair: *caching* forwarding resolvers on nodepool nodes? if so, yes perfect | 18:53 |
| jeblair | fungi: yes | 18:53 |
| *** johnthetubaguy1 has quit IRC | 18:53 | |
| jeblair | (bind will cache in both cases) | 18:53 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove group for the projects with the same name https://review.openstack.org/80049 | 18:53 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove unneeded upstream links https://review.openstack.org/80050 | 18:53 |
| fungi | jeblair: though i think we need a little extra bootstrapping in jobs once that's in place. some initial retries to lookup things we know we'll use and "prime the pump" so to speak | 18:53 |
| SergeyLukjanov | fungi, ^^ | 18:53 |
| fungi | jeblair: which hopefully will help guard against stray initial cache misses timing out | 18:54 |
| clarkb | jeblair: out of curiousity why bind? | 18:54 |
| jgriffith | mtreinish: reading back | 18:55 |
| jeblair | clarkb: "yum install caching-nameserver" and "apt-get install bind9" are all that are needed to get a caching recursive resolver on both os's. | 18:55 |
| *** thuc has quit IRC | 18:55 | |
| fungi | i personally am fine with bind. at least since 9.x it's not the bloated monstrosity of spaghetti it was in the post-4 days of 8.x | 18:55 |
| jeblair | clarkb: i'd be pretty coll with unbound though. | 18:55 |
| jeblair | cool. | 18:55 |
| jgriffith | mtreinish: I'm fine with trying it | 18:55 |
| *** thuc has joined #openstack-infra | 18:55 | |
| jgriffith | mtreinish: if it's overwhelming we can revisit | 18:56 |
| *** SumitNaiksatam_ has joined #openstack-infra | 18:56 | |
| clarkb | jeblair: I don't have any thing against bind it just seems overcomplicated for what we need | 18:56 |
| jgriffith | mtreinish: it could get pretty distracting | 18:56 |
| fungi | when you run bind as a recursive resolver, most of the sharp edges and moving parts are safely out of harm's way | 18:56 |
| fungi | but i'm also fine with something even more lightweight and designed not to act as an authoritative resolver at all | 18:57 |
| jeblair | unbound is available on both os's. | 18:57 |
| fungi | i do agree that if we pick something, it should be the same something on all platforms with consistent configuration | 18:57 |
| clarkb | ++ to consistency | 18:57 |
| *** SumitNaiksatam has quit IRC | 18:57 | |
| clarkb | looks like unbound will only do recursive | 18:58 |
| jeblair | clarkb: unbound can forward | 18:58 |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 18:58 | |
| clarkb | great. So either one works for me | 18:58 |
| clarkb | was curious if there was a specific reason for bind | 18:58 |
| dims_ | jamespage, @hallyn had provided a url that i am wget-ing for the 0.9.8+ debs. how much work/time is it move to proposed? | 18:59 |
| fungi | i take it nobody's a huge fan of nscd (i've used it very little myself, so it very well may be a hideous idea) | 18:59 |
| *** thuc has quit IRC | 19:00 | |
| clarkb | fungi: nscd is that service you restart anytime something else breaks | 19:00 |
| jeblair | unbound does use like 1/5 the ram of named... | 19:00 |
| fungi | clarkb: got it | 19:00 |
| fungi | clarkb: i thought that was pulseaudio | 19:00 |
| *** e0ne has joined #openstack-infra | 19:00 | |
| sdague | mtreinish: I think you need russellb sign off before doing it | 19:00 |
| jeblair | more like 1/3 | 19:01 |
| *** weshay has joined #openstack-infra | 19:01 | |
| clarkb | jeblair: that might be a good reason to use it | 19:01 |
| *** mark0n has joined #openstack-infra | 19:01 | |
| fungi | what percentage of our 8gb of ram does that work out to? ;) | 19:01 |
| clarkb | fungi: jeblair: does centos not do nscd? that might be the simplest to start with | 19:02 |
| fungi | but yeah, unbound sounds like a great choice | 19:02 |
| jeblair | does nscd actually cache dns records? | 19:03 |
| clarkb | ya it caches host lookups | 19:03 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Don't run large-ops test on repos that it doesn't touch https://review.openstack.org/78687 | 19:03 |
| sdague | ok, that's the rebase | 19:03 |
| mark0n | clarkb: remember my problem with JJB and multiple Git remotes? I prepared a patch for that. I also want to add a simple unit test. But I have problems running "tox -epy27". Are you familiar with tox? | 19:03 |
| clarkb | mark0n: yup | 19:03 |
| *** thuc has joined #openstack-infra | 19:03 | |
| fungi | the trick with nscd, as i understand it, is that it hooks into the local resolver library and then forwards and caches based on your resolv.conf | 19:04 |
| fungi | rather than putting your loopback in resolv.conf | 19:04 |
| fungi | but i always thought the c in nscd stood for "cache" | 19:04 |
| jeblair | http://stackoverflow.com/questions/11020027/dns-caching-in-linux | 19:05 |
| mark0n | clarkb: I'm running it on Ubuntu 13.10. It crashes with an exception: https://gist.github.com/mark0n/9513987 | 19:05 |
| *** rpodolyaka has joined #openstack-infra | 19:05 | |
| jeblair | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=335476 | 19:05 |
| clarkb | huh its disabled | 19:05 |
| clarkb | good to know | 19:05 |
| fungi | oho, "You should install this package only if you use slow services like LDAP, NIS or NIS+." | 19:06 |
| jeblair | yah, cause it's broken | 19:06 |
| clarkb | jeblair: I wonder if unscd has the same problem. but unbound seems like a great options if we want simple and bind if we need to do more complicated things | 19:06 |
| jeblair | i'll hack out a change to use unbound after lunch | 19:06 |
| fungi | awesome | 19:07 |
| clarkb | mark0n: huh, let me try reproducing | 19:07 |
| jeblair | that d-g change should still give us the answers we need about running a caching nameserver locally and pointing /etc/hosts at it in general | 19:07 |
| clarkb | ++ | 19:07 |
| fungi | agreed | 19:07 |
| *** sweston has quit IRC | 19:08 | |
| *** sweston has joined #openstack-infra | 19:09 | |
| clarkb | mark0n: works for me. Can you try `tox -repy27` that will rebuild the virtualenv for you. I wonder if the virtualenv is missing something | 19:09 |
| mark0n | clarkb: that gives me the same output | 19:10 |
| mark0n | clarkb: what OS are you using? | 19:10 |
| clarkb | mark0n: the same | 19:10 |
| clarkb | but my tox is version 1.6.1 | 19:11 |
| *** caleb_ has joined #openstack-infra | 19:11 | |
| mtreinish | jgriffith: yeah I could see it getting distracting, but how many gate failures does e-r attribute to cinder bugs? | 19:11 |
| clarkb | mark0n: is pip installed? | 19:12 |
| mark0n | clarkb: did you install tox from the Ubuntu repo or using pip? | 19:12 |
| clarkb | mark0n: using pip | 19:12 |
| clarkb | mark0n: I think what is happening there is tox is trying to fork and exec pip | 19:13 |
| clarkb | but not finding pip | 19:13 |
| *** salv-orlando has joined #openstack-infra | 19:14 | |
| mark0n | clarkb: I tried 1.7.0 - seems to have a different issue, see gist | 19:18 |
| clarkb | mark0n: yea 1.7.0 won't work | 19:18 |
| clarkb | mark0n: they broke backward compat | 19:18 |
| *** thuc_ has joined #openstack-infra | 19:18 | |
| *** thuc has quit IRC | 19:18 | |
| clarkb | mark0n: I would install 1.6.1, pip install --upgrade tox==1.6.1 | 19:18 |
| *** thuc has joined #openstack-infra | 19:19 | |
| *** hogepodge has joined #openstack-infra | 19:19 | |
| *** zhiyan is now known as zhiyan_ | 19:20 | |
| mark0n | clarkb: hmm, with 1.6.1 I have the first issue again... | 19:20 |
| *** mburned is now known as mburned_out | 19:21 | |
| clarkb | mark0n: weird | 19:21 |
| clarkb | mark0n: are you doing this on tip of master or on your patch? can you try doing it on tip of master to see if that is different? | 19:21 |
| clarkb | I wonder if something in the local tree is confusing tox | 19:21 |
| mark0n | clarkb: without my patch | 19:21 |
| *** thuc_ has quit IRC | 19:22 | |
| *** jooools has quit IRC | 19:23 | |
| *** mburned_out is now known as mburned | 19:24 | |
| mark0n | clarkb: I also purged all auto-generated files (including git-ignored files) | 19:24 |
| clarkb | mark0n: ok, let us try debugging this step by step using the virtualenv tox creates. `source .tox/py27/bin/activate ; pip install -r/home/marko/work/automated builds/vagrant/jenkins-job-builder/tools/pip-requires -r/home/marko/work/automated builds/vagrant/jenkins-job-builder/tools/test-requires ; python setup.py testr --slowest` | 19:24 |
| clarkb | oh after pip install -r's you should pip install $PATH_TO_JENKINS_JOB_BUILDER | 19:25 |
| *** krotscheck has joined #openstack-infra | 19:26 | |
| *** melwitt has quit IRC | 19:26 | |
| clarkb | that is basically what tox should be doing for us, and hopefully gives us an idea of what is breaking | 19:26 |
| *** melwitt has joined #openstack-infra | 19:27 | |
| *** malini is now known as malini_afk | 19:27 | |
| clarkb | but I think tox can't find pip for some reason | 19:27 |
| clarkb | I need to grab lunch now, but will be back in a bit others in here should be able to help if you find something new before I get back | 19:28 |
| mordred | kiall: we should actually probably have that conversation here | 19:30 |
| *** zns has quit IRC | 19:30 | |
| mordred | kiall: because even if you do all the work over the weekend because you think it's fun, we'll want jeblair and clarkb and fungi and SergeyLukjanov to be on board | 19:30 |
| *** SumitNaiksatam_ has joined #openstack-infra | 19:31 | |
| *** SumitNaiksatam has quit IRC | 19:31 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 19:31 | |
| *** rpodolyaka has quit IRC | 19:31 | |
| *** rpodolyaka has joined #openstack-infra | 19:31 | |
| fungi | i would hate for someone to waste their time doing work we can't make use of, if a little input from us would result in work we *can* make use of. what's the work? | 19:32 |
| mordred | fungi: the idea is to be able to support non-irc output for some of our bots who are generall useful -such as gerritbot | 19:33 |
| *** afazekas has joined #openstack-infra | 19:33 | |
| kiall | mordred: was AFK | 19:33 |
| fungi | non-irc = rss? | 19:33 |
| mordred | fungi: kiall has pointed out errbot, which supports both xmpp and irc backends | 19:33 |
| kiall | XMPP .. HipChat | 19:33 |
| kiall | we use HipChat at HP | 19:33 |
| fungi | oh, you mean aol instant messenger! | 19:33 |
| fungi | i have seen that before | 19:34 |
| mordred | fungi: so as best I can tell, options would be either adding jabber support to gerritbot, or porting gerritbot to errbot, or writing a duplicate of the gerritbot functionality for something else | 19:34 |
| kiall | fungi: lol.. I wish HP Cloud would get onboard with IRC .. But.. The animated gif support got too many people | 19:34 |
| *** ildikov_ has joined #openstack-infra | 19:34 | |
| clarkb | can it do ssl connections? | 19:34 |
| mordred | I'm a fan of options 1 and 2 because they lead to less forking of things | 19:34 |
| fungi | honestly, i think gerritbot may be too fragile for much of that, and suspect it would be a porting/rewriting effort | 19:35 |
| kiall | It uses the same underlying IRC lib as gerritbot does near as I can see | 19:35 |
| dstufft | or abstract away chat protocols and make that pluggable :D | 19:35 |
| kiall | so .. SSL is probably a yes | 19:35 |
| kiall | dstufft: like this? https://github.com/gbin/err/tree/master/errbot/backends ;) | 19:35 |
| mordred | haha | 19:36 |
| mordred | clarkb: errbot usees irc.bot as its irc stuff | 19:36 |
| mordred | so - I'd say it supports the exact same irc features :) | 19:36 |
| kiall | well.. irc.connection, not irc.bot.. same lib then | 19:36 |
| mordred | https://github.com/gbin/err/blob/master/errbot/backends/irc.py#L11 | 19:36 |
| kiall | humm | 19:37 |
| kiall | I only noticed L5 ;) | 19:37 |
| kiall | Which .. Funny enough.. Looks like a bug. That try import will neverh it | 19:37 |
| kiall | never hit* | 19:38 |
| mordred | it won't? | 19:38 |
| fungi | i'll admit i haven't stared at gerritbot's code much, so it might be amenable to some refactoring to make that abstraction happen... looking it over with an eye for that now | 19:38 |
| *** yolanda has joined #openstack-infra | 19:38 | |
| kiall | yea - L5 will raise an ImportError before the L11 import runs | 19:38 |
| mordred | kiall: oh right- if the irc lib was not there | 19:39 |
| kiall | Yep | 19:39 |
| *** locke105 has joined #openstack-infra | 19:39 | |
| mark0n | clarkb: a white-space in my path broke it... | 19:40 |
| mark0n | clarkb: thanks | 19:40 |
| clarkb | np | 19:40 |
| dstufft | kiall: possibly! | 19:41 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Go back to alembic migrations https://review.openstack.org/79208 | 19:42 |
| mordred | fungi: so - in any case, I didn't want kiall to go have fun this weekend porting gerritbot to be an errbot plugin only to have us tell him to go away | 19:42 |
| fungi | it might not actually be too terribly ugly to pass GerritBot() to inherit from something other than irc.bot.SingleServerIRCBot | 19:42 |
| mordred | fungi: if, otoh, we tell him to go away before hand and he does that work anyway - it's at least a conscious choice | 19:43 |
| fungi | come up with some sort of translation for the common methods supported by both that and the xmpp backend... | 19:43 |
| *** thuc has quit IRC | 19:43 | |
| clarkb | I'd be onboard with it if we can add ssl support | 19:43 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Missing security decorators added https://review.openstack.org/79895 | 19:44 |
| *** thuc has joined #openstack-infra | 19:44 | |
| fungi | mordred: well, my biggest concern, as with a lot of these sorts of contributions of features we don't use, is that gerritbot has no testing so we could easily break that feature and not know about it if it didn't break our very specific use case | 19:44 |
| kiall | clarkb: SSL is baked into it as of 4 months ago.. https://github.com/gbin/err/commit/93ff296483d13c0097341101637bd3c5d1776489 | 19:44 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Fill the last_login field https://review.openstack.org/79887 | 19:44 |
| mordred | kiall: well that certainly seems handy | 19:45 |
| fungi | and one thing gerritbot *does* have going for it, is that it's not using twisted-python | 19:45 |
| dstufft | lifeless: halp, what does "db type could not be determined" mean besides the obvious, I assume it's testrepository but it might also be subunit or some other thing | 19:45 |
| dstufft | fungi: twisted is pretty good for writing network things | 19:45 |
| fungi | dstufft: yes i know. but it also comes with a lot of baggage | 19:46 |
| mordred | and I, for one, don't know how to debug twisted applications. I hear great thigns about them though | 19:46 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Logging in with a pre-existing user saves details https://review.openstack.org/79705 | 19:46 |
| clarkb | and no ipv6 | 19:47 |
| fungi | dstufft: and things like, you can't use ipv6 and ssl/tls together | 19:47 |
| fungi | that | 19:47 |
| kiall | fungi: I don't see twisted as a dep? At least for the IRC parts (no clue what the XMPP etc backends rely on) | 19:47 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Added paging to list endpoints https://review.openstack.org/79757 | 19:47 |
| fungi | kiall: it isn't for gerritbot (it is for supybot which we use in other bots we run) | 19:48 |
| *** SumitNaiksatam_ has joined #openstack-infra | 19:48 | |
| *** nicedice has joined #openstack-infra | 19:48 | |
| kiall | Oh.. Fair enough. supybot always felt like eggdrop to me.. Extended beyond it's design without a re-factoring.. ;) | 19:48 |
| fungi | i was just taking the opportunity to make an undeserved dig at twisted | 19:49 |
| fungi | eggdrop's fine as long as you have a time machine which can take your brain back to tcl syntax | 19:49 |
| clarkb | fungi did you catch the ghub fix? | 19:49 |
| dstufft | mordred: heh | 19:49 |
| clarkb | we need to add read:orgs perms to the token | 19:50 |
| dstufft | mordred: debugging twisted apps is like 100x easier thand ebugging eventlet/gevent at least :| | 19:50 |
| *** SumitNaiksatam has quit IRC | 19:50 | |
| *** SumitNaiksatam_ is now known as SumitNaiksatam | 19:50 | |
| fungi | clarkb: oh! did they add that as a new permission? | 19:50 |
| clarkb | apparently | 19:51 |
| fungi | clarkb: because we haven't touched the perms for that account on the github side in, like, well and embarrassingly long time i think | 19:51 |
| clarkb | dstufft lol | 19:51 |
| clarkb | our brains must work differently | 19:51 |
| fungi | dstufft: in twisted-python, apps debug you! | 19:51 |
| dstufft | clarkb: the problem with eventlet/grevent is that the context switch can happen at any point, so you get a combinatorial number of states the software can be | 19:51 |
| dstufft | with twisted the context switch happens at very defined points | 19:52 |
| clarkb | dstufft thats true of any threads | 19:52 |
| dstufft | yes | 19:52 |
| dstufft | threads are horrible to debug too | 19:52 |
| clarkb | which means I have lots of practice | 19:52 |
| dstufft | that just means your brain is pre-broken :D | 19:52 |
| clarkb | they arent that bad. hard yes but not terrible | 19:52 |
| * fungi thought it was all so much easier in the pvm/mpich days with explicit message-passing between processes | 19:53 | |
| dstufft | It's probably a result of my primary language being Python, and before that PHP but i've done next to nothing with threads | 19:53 |
| mordred | dstufft: you know we write a bunch of threaded python code aroudn here, yeah? | 19:53 |
| clarkb | dstufft ah. /me did C lots of it | 19:54 |
| dstufft | mordred: crazy people | 19:54 |
| dstufft | pip had some threads | 19:54 |
| mordred | dstufft: works pretty well... | 19:54 |
| dstufft | until I ripped them out | 19:54 |
| *** jp_at_hp has quit IRC | 19:54 | |
| mordred | I don't know if you know about the scalable gating system controller called zuul we use... | 19:54 |
| fungi | pip does seem like the sort of tool which doesn't need any threading | 19:54 |
| mordred | ++ | 19:54 |
| clarkb | there are airplanes flying around with a radio I helped build in them. scary | 19:54 |
| fungi | clarkb: as long as someone else built the black box, they should eventually be able to blame your work | 19:55 |
| dstufft | well | 19:55 |
| dstufft | historically pip used threading because | 19:55 |
| kiall | fungi: I'm not sure that's comforting ;) | 19:55 |
| dstufft | it has an unbounded number of urls it would hit to discover stuff | 19:56 |
| * kiall starts using the ferry more often | 19:56 | |
| clarkb | dont worry its not certified for commercial jets | 19:56 |
| dstufft | or did previously | 19:56 |
| fungi | dstufft: right, eww | 19:56 |
| kiall | dstufft: Sure, but pip etc should have stopped what they were doing. Immediately. Once they realized they were building a crawler. | 19:57 |
| dstufft | kiall: pip is by defintion a crawler :] | 19:57 |
| kiall | ;) | 19:57 |
| dstufft | that's how you install stuff | 19:57 |
| dstufft | well find stuff to install | 19:57 |
| clarkb | fungi we have another case of unfamiliar with gerrit but stackforging... | 19:57 |
| dstufft | :/ | 19:57 |
| clarkb | fungi I can respond to it | 19:58 |
| kiall | clarkb: no canned response ready to go? ;) | 19:58 |
| fungi | clarkb: thank you for leaping onto that grenade. remind me to beer you in atlanta or something | 19:58 |
| clarkb | fungi let me know if you need more eyes on github token stuff | 19:58 |
| fungi | clarkb: pulling up the bug now, but if i can confirm it, patch on the way | 19:58 |
| jeblair | kiall: can we make gerritbot and statusbot errbot plugins and run them in the same bot? | 19:58 |
| jeblair | kiall: and then, does errbot have a meetbot plugin? if not, could we make one of those too? | 19:59 |
| kiall | jeremyb: yea.. they could be melded into 2 plugins of 1 bot | 19:59 |
| fungi | meetbot too, while we're talking pie in the sky | 19:59 |
| jeblair | kiall: and finally, can errbot do eavesdrop-style channel logging? | 19:59 |
| kiall | lol - that's more than a weekend right there jeblair ;) | 19:59 |
| jeblair | kiall: there's lots of weekends! | 19:59 |
| *** unicell has quit IRC | 20:00 | |
| clarkb | fungi patch? you should be able to update the token in github I think | 20:00 |
| fungi | i agree it would be beyond awesome not to be managing bots from several disparate codebases | 20:00 |
| jeblair | kiall: because other irc related problems that would be great to solve are: we have a lot of bots, each of which requires configuration and account management... | 20:00 |
| mordred | jeblair: from what I see - I betcha all of those things could be put on a roadmpa - so perhaps this would turn in to the beginning of projet "one bot to rule them all" | 20:00 |
| fungi | clarkb: oh, on the github end. got it. will take a look | 20:00 |
| jeblair | kiall: and also 'we can't restart meetbot (which is also eavesdrop) to pick up a new configuration during meetings' | 20:00 |
| kiall | I'm going with No - not because it's not possible.. But because it's way more work than I can offer ;) But.. Yea.. You can load 100 plugins doing different things into it | 20:00 |
| *** unicell has joined #openstack-infra | 20:00 | |
| mordred | jeblair: also - apparently - each of those could be plugins, and they coudl all still live in diffrent repos if we wanted | 20:00 |
| fungi | i think they would almost certainly live in separate repos | 20:01 |
| jeblair | kiall: let's pretend i didn't say "could you" in those questions, and instead am just asking for your insight to decide if this is a good road to head down | 20:01 |
| jeblair | (which should be easy because i didn't say that ;) | 20:01 |
| mordred | as the unit of plugin installation in errbot is a git repo | 20:01 |
| fungi | heh | 20:01 |
| *** VijayTripathi has joined #openstack-infra | 20:01 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Remove logging of sahara projects to #savanna https://review.openstack.org/79545 | 20:02 |
| jeblair | i would have suggested moving our other bots into supybot a while ago, but for the ipv6/ssl thing | 20:02 |
| jeblair | twisted seems not at all interested in fixing that, so i kind of think that makes supybot a dead end | 20:02 |
| kiall | jeblair: Sure ;) So .. yea, it probably could.. It'd boil down to it's internals of if it's capable of watching N channels at once and actually doing something with the input I guess | 20:02 |
| jeblair | kiall: okay, so that's a question we should probably answer early on... | 20:02 |
| mordred | jeblair: I _believe_ we'll have to answer that to do gerritbot at all | 20:03 |
| kiall | i.e. It's got a bunch of threading stuff in there, but I'm not sure what it's used for (never looked..) | 20:03 |
| mordred | since gerritbot needs to watcha stream | 20:03 |
| jeblair | kiall, mordred: i'm much more interested in a technology shift here if we move to something that puts us in a better place overall | 20:03 |
| mordred | jeblair: I agree | 20:03 |
| *** Sukhdev has joined #openstack-infra | 20:04 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: only care about unclassifieds in gate queue https://review.openstack.org/80066 | 20:05 |
| mordred | from what I can tell - since gerritbot needs to run a thread to watch the gerrit stream - it means that using errbot for gerrit at all will necessarily have to figure out if it can do multiple things in threads | 20:06 |
| *** sarob has quit IRC | 20:06 | |
| *** SumitNaiksatam has quit IRC | 20:07 | |
| jeblair | mordred: yeah, it doesn't necessarily have to have a huge thread infrastructure, but just having things like the a "send privmsg" method be thread-safe (even better if it has thread-safe flood control) would be good. | 20:07 |
| jeblair | mordred, kiall: so open questions are: a) thread safety; b) on-the-fly reconfiguration; c) sensible configuration for multiple plugins; d) existence of meetbot/eavesdrop plugins | 20:09 |
| sdague | so honestly, what the suggestion that fungi made off hand earlier | 20:09 |
| sdague | to gerrit -> rss | 20:09 |
| jeblair | sdague: like openstackwatch? | 20:09 |
| fungi | clarkb: found the permission on the token and added it, rerunning the script now... hasn't died yet either | 20:10 |
| sdague | I guess, I don't know what that is :) | 20:10 |
| *** caleb_ has quit IRC | 20:10 | |
| sdague | but regardless then bots pull that and off we go | 20:10 |
| jeblair | sdague: it's a 95% completed project in the config repo to make an rss feed for gerrit. afaik it just needs some finishing touches | 20:10 |
| jeblair | sdague, fungi: maybe i don't understand the suggestion... | 20:10 |
| *** sarob has joined #openstack-infra | 20:11 | |
| sdague | and would solve having everyone talking to gerrit | 20:11 |
| kiall | jeblair: noted .. Will eye up the code to see if it's capable of handle those | 20:11 |
| sdague | or was that not part of what was trying to be solved | 20:11 |
| jeblair | sdague: you want to go from a real-time push notification system to a delayed polling based pull system? | 20:11 |
| fungi | jeblair: i was indeed making an illusion to openstackwatch | 20:11 |
| *** jasond` has left #openstack-infra | 20:11 | |
| fungi | allusion | 20:11 |
| jeblair | sdague: afaik the problem kiall wants to solve is that his company doesn't use irc for their internal copy of infra, so they want a gerritbot that talks to their proprietary thing | 20:12 |
| *** weshay has quit IRC | 20:12 | |
| kiall | jeblair: Yep - bang on | 20:12 |
| kiall | But if the choice of bot/lib/etc has other benefits, cool. | 20:13 |
| jeblair | kiall: ++ | 20:13 |
| kiall | jeblair: .. seeing code like this suggests on-the-fly reconfig won't be an issue.. https://github.com/gbin/err/blob/master/errbot/plugin_manager.py#L140-154 | 20:15 |
| *** esker has quit IRC | 20:15 | |
| sdague | yeh, it just makes me wonder about the other bots as well. because I imagine the desire for another elastic recheck at some point. | 20:16 |
| kiall | seems to handle reloading the plugin's code without a restart.. So, promising it can handle re-config | 20:16 |
| jeblair | sdague: yeah, i'd love it if all our bots could be plugins to one bot; including er; to reduce the overhead of what we really want to do which is "add functionality to irc" | 20:16 |
| sdague | jeblair: yeh that would be nice | 20:17 |
| kiall | jeblair: s/irc/irc+hipchat/ ;)) | 20:18 |
| sdague | I expect the complexities build up pretty quickly here though | 20:18 |
| *** resker has joined #openstack-infra | 20:18 | |
| fungi | sdague: the openstackwatch module in openstack-infra/jeepyb makes http://rss.cdn.openstack.org/nova.xml | 20:19 |
| fungi | just basically a query of open reviews | 20:20 |
| fungi | and uploads to a swift container periodically | 20:20 |
| sdague | cool | 20:20 |
| *** dprince has joined #openstack-infra | 20:20 | |
| fungi | chmouel wrote it and anteaya puppeted it a while back | 20:20 |
| *** rpodolyaka has quit IRC | 20:20 | |
| *** rpodolyaka has joined #openstack-infra | 20:21 | |
| *** rpodolyaka has quit IRC | 20:21 | |
| *** rpodolyaka1 has joined #openstack-infra | 20:21 | |
| fungi | i think the idea was that once we have more control over gerrit themes, it could get linked as an rss feed from an out-of-the-way part of the ui | 20:21 |
| *** Sukhdev has quit IRC | 20:21 | |
| *** mrda_away is now known as mrda | 20:21 | |
| fungi | clarkb: close-pull-requests completed its run with no errors | 20:22 |
| sdague | coolio | 20:23 |
| *** thuc has quit IRC | 20:24 | |
| *** thuc has joined #openstack-infra | 20:24 | |
| *** sweston has quit IRC | 20:25 | |
| jeblair | yeah, forwarding everything in unbound is trivial | 20:26 |
| *** changbl has quit IRC | 20:26 | |
| sdague | jeblair / fungi: I rebased jogo's patch - https://review.openstack.org/#/c/78687/ to free up some dsvm nodes | 20:27 |
| *** mrodden has quit IRC | 20:28 | |
| clarkb | fungi woot | 20:28 |
| jeblair | lgtm | 20:28 |
| fungi | clarkb: spammed us with about 30-60 pr closures | 20:29 |
| *** rossella_s has quit IRC | 20:29 | |
| clarkb | fungi was the token thing through their webui? | 20:29 |
| fungi | clarkb: yep! | 20:29 |
| fungi | under settings->applications->personal token access | 20:29 |
| *** rossella_s has joined #openstack-infra | 20:29 | |
| fungi | there's a whole array of permissions you can add/remove in there now | 20:29 |
| jeblair | looks like check node wait time is about 1.5 hours | 20:29 |
| fungi | but that was the only one i needed to add | 20:29 |
| *** afazekas has quit IRC | 20:29 | |
| jogo | sdague: thanks! | 20:30 |
| *** MarkAtwood has joined #openstack-infra | 20:30 | |
| *** che-arne has quit IRC | 20:30 | |
| fungi | jogo: lgtm too... aprv'd | 20:31 |
| *** mburned is now known as mburned_out | 20:32 | |
| jeblair | of course the dnssec trust file is /var/lib/unbound/root.anchor on centos and /var/lib/unbound/root.key on ubuntu. | 20:32 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't run large-ops test on repos that it doesn't touch https://review.openstack.org/78687 | 20:33 |
| fungi | they're almost the same... what's a file suffix between frenemies? | 20:33 |
| lifeless | dstufft: it mens python changed defaults for dbm between minor versions and screwed everyone | 20:34 |
| lifeless | dstufft: *means* | 20:34 |
| dstufft | lifeless: how can I fix it | 20:34 |
| dstufft | it happens on the first run even if I delete the .testrepository dir :[ | 20:35 |
| lifeless | dstufft: rm .testrepository/timing.db | 20:35 |
| lifeless | dstufft: I'm going to guess that you're using tox to run several versions of python at once | 20:35 |
| dstufft | lifeless: yes tox, just 1 version | 20:35 |
| jogo | http://logs.openstack.org/67/57967/23/gate/gate-tempest-dsvm-neutron/0de62a7/logs/devstack-gate-setup-workspace-new.txt.gz | 20:35 |
| jogo | is there a bug filed for that | 20:35 |
| lifeless | dstufft: just one version? really? | 20:35 |
| lifeless | dstufft: colour me skeptical :) | 20:36 |
| *** e0ne has quit IRC | 20:36 | |
| dstufft | lifeless: actually I remember the symtpoms slightly wrong, it works the first time but fails every other time after that | 20:36 |
| *** e0ne has joined #openstack-infra | 20:36 | |
| dstufft | unless I delete | 20:36 |
| dstufft | lifeless: well i'm using tox -e py26 | 20:37 |
| dstufft | so if it's executing something other than py26 ima be angry | 20:37 |
| lifeless | dstufft: you may need to be angry; anyhow what I'd do is just not use testr within the venv; use it from outside. | 20:37 |
| lifeless | dstufft: then you won't be subject to python version vaguaries | 20:37 |
| jogo | fungi: ^^ | 20:38 |
| *** emagana has joined #openstack-infra | 20:38 | |
| jogo | saw a whole slew of failures like this http://logs.openstack.org/59/62859/1/gate/gate-tempest-dsvm-neutron/7025cba/console.html | 20:38 |
| mordred | lifeless: there's no good way to do that in a tox context really | 20:38 |
| jogo | wnat to add a e-r fingerprint to remove them from the unclassificed list | 20:38 |
| fungi | jogo: 1291440 | 20:38 |
| dstufft | lifeless: did I ever mention that software is the worst | 20:39 |
| jeblair | OverLimit: Quota exceeded for ram: Requested 16384, but already used 944128 of 960000 ram (HTTP 413) (Request-ID: req-368f5ce0-8211-4acd-9acb-ba002baea11b) | 20:39 |
| jeblair | fungi, clarkb, lifeless: ^ | 20:39 |
| *** VijayTripathi has quit IRC | 20:39 | |
| clarkb | jeblair: which provider is that from? | 20:39 |
| fungi | jeblair: in tripleo? | 20:39 |
| jeblair | does anyone understand why it is requesting 16384 when it's configured for 8192 | 20:39 |
| clarkb | oh right becaseu 16G nodes | 20:39 |
| fungi | (i assume because you pinged lifeless) | 20:39 |
| jeblair | tripleo, yes | 20:39 |
| clarkb | jeblair: there is no 8192 flavor, so it picks the next bigger | 20:39 |
| jeblair | that's q.1 | 20:40 |
| jeblair | ok | 20:40 |
| *** e0ne has quit IRC | 20:40 | |
| jeblair | so q.2: 960000/16384 = 58.593750 | 20:40 |
| *** ociuhandu has quit IRC | 20:40 | |
| *** thuc_ has joined #openstack-infra | 20:40 | |
| lifeless | mordred: just remove testrepository from the test-requirements, no? | 20:40 |
| lifeless | mordred: use binddep instead | 20:40 |
| jeblair | our actual limit is 58 servers; lifeless: should we adjust nodepool's config to match, or do you want to increase the quota? | 20:41 |
| jogo | fungi: how does 'message:"savanna" AND message: "fatal\: https://git.openstack.org/openstack" AND filename:"logs/devstack-gate-setup-workspace-new.txt"' sounds for a fingeprint | 20:41 |
| lifeless | jeblair: memory quota? let me check hypervisor stats | 20:41 |
| jeblair | lifeless: yes | 20:41 |
| lifeless | free_ram_mb | 107624 | 20:41 |
| fungi | clarkb: on that last update to 1291479, you'd think they'd be working more closely with the heat developer community (they did request sbaker as a member of their ptl group) | 20:42 |
| lifeless | | memory_mb | 966760 | | 20:42 |
| lifeless | | memory_mb_used | 859136 | | 20:42 |
| clarkb | fungi: you would think... | 20:42 |
| clarkb | fungi: I am going to respond now | 20:42 |
| clarkb | then figure out puppet module installs | 20:42 |
| *** e0ne has joined #openstack-infra | 20:42 | |
| ttx | fungi, jeblair: howdy! Is the world still turning ? | 20:42 |
| fungi | ttx: still turning | 20:42 |
| lifeless | room for 6 more in there | 20:43 |
| mordred | lifeless: what? | 20:43 |
| lifeless | so I think reduce nodepool config | 20:43 |
| lifeless | mordred: what what | 20:43 |
| fungi | lifeless: have we maybe leaked nodes in that tenant? | 20:43 |
| ttx | fungi: damn, that hurts my narcissism | 20:43 |
| *** thuc has quit IRC | 20:43 | |
| fungi | ttx: become a solipsist and then it won't matter ;) | 20:44 |
| lifeless | fungi: there's 859GB used, which is ~53 servers | 20:44 |
| ttx | fungi, jeblair: it nothing urgent is needed from me, i'll go back to ignoring what happens this week | 20:44 |
| lifeless | fungi: so hitting 58 seems about right | 20:44 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Adjust tripleo nodepool config to actual values https://review.openstack.org/80078 | 20:44 |
| lifeless | since 53 + 6 = 59 | 20:44 |
| *** jasondotstar has joined #openstack-infra | 20:45 | |
| jeblair | ttx: go back to skiing. or start skiing if you've been snowboarding. ;P | 20:45 |
| lifeless | jeblair: dunno if you saw https://review.openstack.org/#/c/79091/ | 20:45 |
| *** wenlock has joined #openstack-infra | 20:45 | |
| lifeless | jeblair: which I put up when I changed the flavor | 20:45 |
| jeblair | lifeless: nope, hadn't gotten to that yet, though it looks like it's still to high, yeah? | 20:45 |
| sdague | ttx: everythings pretty solid, nothing crazy on fire | 20:46 |
| jeblair | lifeless: i went with 50 since i calculate the max at 58 | 20:46 |
| lifeless | jeblair: it is, I can abandon happily enough | 20:46 |
| *** yolanda has quit IRC | 20:46 | |
| *** mrodden has joined #openstack-infra | 20:46 | |
| fungi | jogo: "savanna-dashboard/info/refs not found" maybe | 20:46 |
| fungi | jogo: i think they all failed on the same point in the list | 20:46 |
| fungi | jogo: and that's one which definitely won't recur exactly the same way again | 20:46 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291440 https://review.openstack.org/80080 | 20:47 |
| clarkb | ttx: enjoy the snow | 20:47 |
| *** Sukhdev has joined #openstack-infra | 20:47 | |
| * fungi is popping out for dinner... bbl | 20:47 | |
| jogo | fungi: thanks | 20:47 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add fingerprint for bug 1291440 https://review.openstack.org/80080 | 20:48 |
| SergeyLukjanov | jogo, agreed with fungi | 20:48 |
| jogo | SergeyLukjanov: ^ | 20:48 |
| SergeyLukjanov | jogo, thanks for fingerprint | 20:48 |
| SergeyLukjanov | jogo, are there many affected changes? | 20:49 |
| jogo | SergeyLukjanov: once that lands I think we should have a pretty high classifcation rate http://status.openstack.org/elastic-recheck/data/uncategorized.html | 20:49 |
| jogo | SergeyLukjanov: there were a good number in gate | 20:49 |
| jogo | 258 hits in gate | 20:50 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: fix formatting on 'and' https://review.openstack.org/80083 | 20:50 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: only care about unclassifieds in gate queue https://review.openstack.org/80066 | 20:51 |
| ttx | alrighty then | 20:51 |
| * ttx signs off | 20:51 | |
| *** ArxCruz has quit IRC | 20:52 | |
| openstackgerrit | Martin Konrad proposed a change to openstack-infra/jenkins-job-builder: SCM module: Add support for multiple Git remotes. https://review.openstack.org/80084 | 20:52 |
| SergeyLukjanov | jogo, oh, quite a lot | 20:52 |
| fungi | a lot of those didn't report though | 20:52 |
| jogo | fungi: that makes sense | 20:53 |
| clarkb | nibalizer: Module 'puppetlabs-inifile' (v1.0.3) is already installed | 20:53 |
| fungi | as soon as sdague pointed them out, we force-merged the fix immediately and i reshuffled the gate to get everything which was still in there but about to die to retest | 20:53 |
| jogo | I think that caused the recent spike http://jogo.github.io/gate/ | 20:53 |
| *** thuc__ has joined #openstack-infra | 20:53 | |
| clarkb | nibalizer: puppet module install should install the modules that it needs as dependencies right? but you want ot install it as no deps? | 20:53 |
| clarkb | nibalizer: I am slowly debugging this but can't install git on my VM because my apt repo is gone apparently | 20:53 |
| *** thuc_ has quit IRC | 20:54 | |
| clarkb | nibalizer: I will know a lot more in about 5 minutes | 20:54 |
| nibalizer | clarkb: okay | 20:54 |
| fungi | jogo: probably so. shouldn't have happened outside of 12:45-13:30 if memory serves | 20:54 |
| * fungi is really at dinner now ;) | 20:54 | |
| nibalizer | puppet module install brings in deps, yes | 20:54 |
| SergeyLukjanov | https://review.openstack.org/#/c/80080/ lgtm | 20:54 |
| nibalizer | clarkb: is this changes that have already landed or a change that I can look at? | 20:54 |
| clarkb | nibalizer: so I think something is installing 1.0.3 then the specific 1.0.0 version dies | 20:55 |
| nibalizer | also i'm about to go into a work meeting so will be of limited use for an hour or two | 20:55 |
| clarkb | nibalizer: its tip of master | 20:55 |
| nibalizer | greaaat | 20:55 |
| clarkb | nibalizer: no worries I am debugging now | 20:55 |
| nibalizer | uh well just rollback to 1.0.0 and screw puppetdb, i can fix it later :P | 20:55 |
| clarkb | I can't rollback that is what it is trying to do :P | 20:55 |
| clarkb | and failing | 20:55 |
| *** khyati has quit IRC | 20:57 | |
| *** malini_afk is now known as malini | 20:57 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Periodic purge of fixed and unseen bugs https://review.openstack.org/80086 | 20:59 |
| *** skraynev is now known as skraynev_afk | 21:00 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova unit test bug 1291605 https://review.openstack.org/80087 | 21:02 |
| *** vkozhukalov has joined #openstack-infra | 21:02 | |
| *** rlandy_ has joined #openstack-infra | 21:04 | |
| *** rpodolyaka1 has quit IRC | 21:05 | |
| *** smarcet has joined #openstack-infra | 21:05 | |
| *** thuc has joined #openstack-infra | 21:05 | |
| *** gyee has quit IRC | 21:05 | |
| *** ociuhandu has joined #openstack-infra | 21:06 | |
| *** nati_ueno has quit IRC | 21:07 | |
| *** rlandy__ has quit IRC | 21:07 | |
| *** mburned_out is now known as mburned | 21:08 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: fix formatting on 'and' https://review.openstack.org/80083 | 21:08 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: only care about unclassifieds in gate queue https://review.openstack.org/80066 | 21:08 |
| *** thuc__ has quit IRC | 21:08 | |
| *** nati_ueno has joined #openstack-infra | 21:08 | |
| *** rpodolyaka has joined #openstack-infra | 21:09 | |
| *** weshay has joined #openstack-infra | 21:09 | |
| *** dprince has quit IRC | 21:11 | |
| mattoliverau | Morning | 21:11 |
| *** mrodden has quit IRC | 21:11 | |
| *** mrodden has joined #openstack-infra | 21:12 | |
| *** rlandy__ has joined #openstack-infra | 21:13 | |
| *** jeckersb is now known as jeckersb_gone | 21:13 | |
| jeblair | mattoliverau: good morning | 21:13 |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Make email-ext's AlwaysTrigger the default. https://review.openstack.org/79337 | 21:14 |
| *** denis_makogon_ has joined #openstack-infra | 21:14 | |
| *** rlandy_ has quit IRC | 21:15 | |
| *** e0ne has quit IRC | 21:16 | |
| *** dkranz has quit IRC | 21:17 | |
| *** e0ne has joined #openstack-infra | 21:17 | |
| clarkb | nibalizer: huh it worked fine on my test machine | 21:19 |
| *** rfolco has quit IRC | 21:19 | |
| nibalizer | weird | 21:19 |
| clarkb | nibalizer: is there a reson for explicitly listing 1.0.0 in install_modules? is that so you can install puppetdb module without deps? | 21:20 |
| clarkb | I am going to give it a try with nodepool again | 21:20 |
| *** mfer has quit IRC | 21:20 | |
| clarkb | (I used the same region and base image and everything to test) | 21:20 |
| *** nati_ueno has quit IRC | 21:20 | |
| *** denis_makogon_ is now known as denis_makogon | 21:21 | |
| *** zns has joined #openstack-infra | 21:23 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for nova unit test bug 1291620 https://review.openstack.org/80091 | 21:23 |
| *** hogepodge has quit IRC | 21:23 | |
| *** rlandy__ is now known as rlandy | 21:24 | |
| clarkb | I will try a different provider too | 21:24 |
| *** hogepodge has joined #openstack-infra | 21:24 | |
| nibalizer | clarkb: no inifile can have any 1.x release | 21:25 |
| nibalizer | as far as puppetdb is concerned | 21:25 |
| *** sweston has joined #openstack-infra | 21:25 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 21:26 |
| nibalizer | clarkb: you can bump inifile to 1.0.3 if that makes problems go away | 21:26 |
| clarkb | nibalizer: ok it just worked | 21:26 |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Fix typo in ircbot's default matrix-notifier. https://review.openstack.org/79328 | 21:26 |
| clarkb | nibalizer: I think puppet forge may haev been cranky with us | 21:26 |
| *** cody-somerville- has joined #openstack-infra | 21:26 | |
| nibalizer | puppetdb just wants 1.something | 21:26 |
| nibalizer | https://github.com/puppetlabs/puppetlabs-puppetdb/blob/master/Modulefile | 21:26 |
| *** jasondotstar has quit IRC | 21:26 | |
| clarkb | nibalizer: so I think we are good, thank you | 21:26 |
| jeblair | clarkb, fungi, pleia2, mattoliverau: https://review.openstack.org/80092 could use some scrutiny as it has the capability to break everything. everywhere. all at once. :) | 21:26 |
| clarkb | jeblair: looking | 21:27 |
| nibalizer | okay | 21:27 |
| clarkb | jeblair: is that the sort of thing we merge friday afternoon? | 21:27 |
| jeblair | clarkb: i'm thinking so. | 21:27 |
| jeblair | after some dry runs | 21:28 |
| *** rcarrillocruz has joined #openstack-infra | 21:28 | |
| *** caleb_ has joined #openstack-infra | 21:28 | |
| *** caleb_ has quit IRC | 21:29 | |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Fill the last_login field https://review.openstack.org/79887 | 21:29 |
| *** caleb_ has joined #openstack-infra | 21:29 | |
| *** caleb_ has quit IRC | 21:29 | |
| *** rcarrillocruz1 has quit IRC | 21:29 | |
| *** sweston has quit IRC | 21:29 | |
| *** sweston has joined #openstack-infra | 21:30 | |
| *** caleb_ has joined #openstack-infra | 21:30 | |
| *** caleb_ has quit IRC | 21:30 | |
| *** sweston has quit IRC | 21:30 | |
| *** sweston has joined #openstack-infra | 21:31 | |
| *** caleb_ has joined #openstack-infra | 21:31 | |
| *** nati_ueno has joined #openstack-infra | 21:33 | |
| clarkb | jeblair: reviewed | 21:35 |
| openstackgerrit | Randy Coulman proposed a change to openstack-infra/jenkins-job-builder: Fix typo in ircbot's default matrix-notifier. https://review.openstack.org/79328 | 21:38 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Logging in with a pre-existing user saves details https://review.openstack.org/79705 | 21:42 |
| * clarkb wait for vmdk to finish downloading so that we can have new hpcloud region b image | 21:43 | |
| clarkb | sdague: ^ about that | 21:44 |
| *** thedodd has quit IRC | 21:44 | |
| clarkb | sdague: I think the discussion yesterday was that anyone can have bigger images if they want them | 21:44 |
| *** CaptTofu has quit IRC | 21:44 | |
| sdague | clarkb: I think that whatever parses devstack should be smarter about that | 21:44 |
| clarkb | sdague: and we do download an ubuntu image. None of that takes forever | 21:44 |
| clarkb | the vmdk takes forever | 21:44 |
| sdague | because there is no reason that vmdk is in that image at all | 21:45 |
| clarkb | sdague: its there because devstack may useit | 21:45 |
| *** mriedem has quit IRC | 21:45 | |
| clarkb | because its in the devstack config | 21:45 |
| sdague | but you know which hypervisors you are going to support in that image | 21:45 |
| sdague | and all of those are hidden behind hypervisor ifs | 21:46 |
| *** smarcet has left #openstack-infra | 21:46 | |
| sdague | so instead of parsing that, eval it | 21:46 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add fingerprint for bug 1291440 https://review.openstack.org/80080 | 21:46 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Periodic purge of fixed and unseen bugs https://review.openstack.org/80086 | 21:46 |
| clarkb | sdague: the image creator can't eval it with a high degree of certainty about which case will be hit | 21:46 |
| clarkb | sdague: because we could xen or kvm cirros or ubuntu | 21:46 |
| openstackgerrit | A change was merged to openstack-infra/storyboard: Missing security decorators added https://review.openstack.org/79895 | 21:47 |
| *** khyati has joined #openstack-infra | 21:47 | |
| mark0n | I'm using gerrit for the first time. I submitted a patch and jenkins discovered a minor style issues. How am I supposed to resubmit? git commit --amend && git review or git commit && git review? | 21:47 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard: Logging in with a pre-existing user saves details https://review.openstack.org/79705 | 21:47 |
| *** thomasem has quit IRC | 21:48 | |
| clarkb | mark0n: if you need to update a change you already pushed do git commit --amend && git review | 21:48 |
| clarkb | mark0n: git commit && git review will give you a second change | 21:48 |
| mark0n | clarkb: ok, thanks | 21:48 |
| mattoliverau | jeblair: looking now, sorry, had a lot of scroll back to read :) | 21:50 |
| jeblair | sdague: yeah, so i suggested mordred propose that change to spark a conversation about that... | 21:51 |
| *** rlandy has quit IRC | 21:51 | |
| jeblair | sdague: my understanding is that devstack has been shrinking to core functionality with good support for plugins/addons | 21:51 |
| *** rpodolyaka has quit IRC | 21:52 | |
| jeblair | sdague: so i thought with that in mind, it was worth considering whether vmware support is still something that should be in core | 21:52 |
| jeblair | sdague: or if it makes better sense as add-ons | 21:52 |
| sdague | jeblair: so that's probably a can of worms in politics that will take us a while to get there | 21:53 |
| sdague | because it's an in tree driver for nova | 21:53 |
| sdague | which is fair game | 21:53 |
| sdague | clarkb: and I disagree with the high degree of certainty issue | 21:54 |
| clarkb | sdague: how can nodepool image builder know if it needs cirros or ubunut image or xen | 21:54 |
| sdague | http://paste.openstack.org/show/73317/ | 21:54 |
| clarkb | sdague: it can't | 21:54 |
| *** fifieldt has joined #openstack-infra | 21:54 | |
| clarkb | the distinguishing features between cases aren't strong enough | 21:54 |
| sdague | you should have a config for nodepool to decide what hypervisor list you want to support | 21:54 |
| sdague | and just drop vmware from that list | 21:55 |
| clarkb | whcih means we need to udpate devstack-gate anytime you want a new hypervisor | 21:55 |
| clarkb | which we don't need today | 21:55 |
| sdague | sure | 21:55 |
| jeblair | clarkb: not devstack-gate, nodepool config | 21:55 |
| clarkb | jeblair: oh right this is nodepool config now | 21:55 |
| clarkb | I am thinking of the old thing | 21:55 |
| sdague | but your alternative is to impose policy into devstack because you don't want something in the gate | 21:56 |
| sdague | which I think is the wrong thing | 21:56 |
| jeblair | sdague: whoah whoah | 21:56 |
| jeblair | sdague: no one is imposing policy | 21:56 |
| *** dkliban has quit IRC | 21:56 | |
| *** rpodolyaka has joined #openstack-infra | 21:56 | |
| sdague | I really think it should be fine for nodepool to decide the list of hypervisors it's going to support and eval devstack to get all those images. Then the vmware thing won't impact things. | 21:57 |
| sdague | or whatever part of infra wants to make that decision | 21:57 |
| jeblair | sdague: we're just having a discussion about where the configuration of this kind of environment belongs in the developer deployment tool... | 21:58 |
| openstackgerrit | Martin Konrad proposed a change to openstack-infra/jenkins-job-builder: SCM module: Add support for multiple Git remotes. https://review.openstack.org/80084 | 21:58 |
| sdague | sure | 21:58 |
| jeblair | sdague: mordred suggested that we just ignore the vmdk | 21:59 |
| *** rossella_s has quit IRC | 21:59 | |
| sdague | I strongly believe that having a functional base image for your hypervisor that you installed is an important starting point | 21:59 |
| jeblair | sdague: i said i thought it would be nice if we actually talked to some devstack developers about it before we just worked around the problem | 21:59 |
| sdague | honestly I think devstack should provide more of those | 21:59 |
| jeblair | sdague: i'm starting to regret that now that i've been accused of trying to 'impose policy' | 21:59 |
| sdague | jeblair: ok, well mordred's patch was basically delete the vmdk because it's slow :) | 22:00 |
| sdague | which is what I heard from clarkb as well | 22:00 |
| sdague | so maybe I misheard | 22:00 |
| mordred | sdague: two different patches | 22:00 |
| mordred | sdague: I made a patch to devstack which was "delete the ref to the vmdk" - and I did that to start the discussion | 22:00 |
| mordred | sdague: I also made a patch to devstack-gate which trims the vmdk out of the list of images we pre-download | 22:01 |
| mordred | sdague: which would work, but puts more not less logic in d-g | 22:01 |
| clarkb | mordred: nodepool | 22:01 |
| mordred | clarkb: yah | 22:01 |
| clarkb | mordred: I keep confusingthem too because d-g was the cacher. One day I will remember | 22:01 |
| mordred | sdague: my view, fwiw, is that if it's not tested, it's broken - and we don't test the vmdk in the gate, so I dont' know why it's in devstack | 22:02 |
| lifeless | finally! | 22:02 |
| lifeless | Cloud Images is currently in Limited Availability. It will allow you to import properly prepared images. We expect to open it to all customers before the end of March. | 22:02 |
| lifeless | You can take a look at the FAQ to see exactly what’s coming: http://www.rackspace.com/knowledge_center/article/frequently-asked-questions-about-using-glance-and-the-images-api | 22:02 |
| mordred | lifeless: yup. excited about that | 22:02 |
| *** jhesketh__ has joined #openstack-infra | 22:02 | |
| *** jhesketh__ has quit IRC | 22:02 | |
| *** jhesketh_ has joined #openstack-infra | 22:02 | |
| *** jhesketh__ has joined #openstack-infra | 22:03 | |
| *** jhesketh_ has quit IRC | 22:03 | |
| *** rpodolyaka has quit IRC | 22:03 | |
| *** jhesketh has joined #openstack-infra | 22:03 | |
| jhesketh__ | Morning | 22:03 |
| mordred | mornign jhesketh__ | 22:03 |
| *** mwagner_lap has quit IRC | 22:04 | |
| *** zul has joined #openstack-infra | 22:04 | |
| clarkb | sdague: we noticed the thing because it is slow to cache | 22:04 |
| clarkb | I think more than half our image build time is downloadingthat image | 22:05 |
| *** zul has quit IRC | 22:05 | |
| clarkb | not saying you can't have large images. (and the ubuntu image is evidence of that unless I am missing something) | 22:06 |
| jeblair | why is it a different image in the first place? | 22:07 |
| jeblair | i mean, can you not run cirros on esx? | 22:08 |
| jeblair | (and sure, if you want to have a full os instead of cirros that's fine -- couldn't you run the same full os on kvm and esx?) | 22:09 |
| mtreinish | fungi, clarkb, jeblair: have you guys seen this before?: http://logs.openstack.org/72/79572/1/check/gate-tempest-python27/f099c44/console.html#_2014-03-11_10_02_35_372 | 22:09 |
| jeblair | mtreinish: unit tests aren't allowed to sudo | 22:09 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: If not posting IRC comment for unrecognized error, log something https://review.openstack.org/78801 | 22:10 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Make IRC bot list which failures were seen in which job. https://review.openstack.org/78790 | 22:10 |
| jogo | mtreinish: this was for you ^ | 22:10 |
| *** rpodolyaka has joined #openstack-infra | 22:10 | |
| mtreinish | jeblair: ahh ok, I missed that the proposed patch was actually trying to run something with sudo instead of mocking it | 22:12 |
| * mtreinish leaves a -1 comment | 22:12 | |
| jeblair | mtreinish: yay the system works! :) | 22:13 |
| mtreinish | jogo: it was't my thing I was fine without the tests. I was just trying to get mriedman to stand up for himself :) | 22:13 |
| *** maxbit has joined #openstack-infra | 22:14 | |
| *** rcleere has quit IRC | 22:15 | |
| clarkb | how do we feel about merging mordred nodepool change? (I am still waiting for this new image, makes it really hard to iterate on this stuff) | 22:15 |
| clarkb | I don't understand if garyk is saying they are using nodepool or only devstack | 22:15 |
| jogo | mtreinish: well it was a valid point, that code needed tests | 22:15 |
| mtreinish | jogo: yeah but we've kind of just ignored it for most things in the past | 22:16 |
| jogo | mtreinish: heh i noticed | 22:17 |
| jeblair | clarkb: wfm | 22:17 |
| *** lcostantino has quit IRC | 22:18 | |
| sdague | sorry, got pulled away | 22:19 |
| clarkb | jeblair: were you able to parse garyk's statement? I can't tell if the nodepool change will breka him or not | 22:19 |
| sdague | mordred: I think it's in devstack because the vmware system does test it | 22:19 |
| sdague | in their CI | 22:19 |
| sdague | we can confirm and call them on that | 22:20 |
| mordred | sdague: do they use devstack in their CI? | 22:20 |
| sdague | for nova, probably | 22:20 |
| *** aysyd has quit IRC | 22:20 | |
| sdague | I know the hyperv guys do | 22:20 |
| mordred | ok. then I can buy it being decently valid - and I'd vote for us filtering in d-g - since they are one of the first ones who stepped up to the 3rd party gate strongly | 22:20 |
| mordred | if it was "a dev might want to do vmware stuff" my opinion is different | 22:21 |
| sdague | mordred: ok, fair. | 22:21 |
| clarkb | is garyk usually on irc? /me would like to clarify comment on https://review.openstack.org/#/c/79537/1 before approving | 22:22 |
| *** yassine has joined #openstack-infra | 22:22 | |
| sdague | so I guess it begs a second question, which is I'd like to add fedora20 and ubuntu(something) cloud images into devstack to make it more useful to people for dev (as that's one of the first things I add) | 22:23 |
| sdague | so we should figure out if that's going to cause issues here, or if there is some other way we want to mark the images | 22:23 |
| clarkb | sdague: I don't think it is going to cause issues. see my earlier statements | 22:23 |
| sdague | ok | 22:24 |
| clarkb | the existing ubuntu image is fine | 22:24 |
| sdague | the lxc one? | 22:24 |
| *** lukego has quit IRC | 22:24 | |
| clarkb | openvz | 22:24 |
| sdague | ok | 22:24 |
| sdague | now, openvz, that I could get behind deleting entirely :) | 22:24 |
| clarkb | ++ | 22:24 |
| sdague | because, no one is using it in the gate | 22:24 |
| sdague | but we'll wait until the docker drop | 22:25 |
| *** e0ne has quit IRC | 22:25 | |
| *** maxbit_ has joined #openstack-infra | 22:25 | |
| *** e0ne has joined #openstack-infra | 22:25 | |
| *** e0ne has quit IRC | 22:25 | |
| *** jgrimm has quit IRC | 22:26 | |
| clarkb | ermagherd | 22:26 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: If not posting IRC comment for unrecognized error, log something https://review.openstack.org/78801 | 22:26 |
| clarkb | the image cacher is downloading the same image over and over | 22:26 |
| sdague | hah | 22:26 |
| * clarkb looks at fixing that | 22:26 | |
| sdague | so, I do think it is probably worth while to figure out why vmware needs a custom image. I will put that on a list, because I'd really like to get us to people pulling fairly standard upstream cloud images (fedora and ubuntu) | 22:27 |
| sdague | as that seems more useful | 22:27 |
| *** mgagne has quit IRC | 22:27 | |
| clarkb | ++ | 22:27 |
| *** rpodolyaka has quit IRC | 22:28 | |
| sdague | clarkb: do you have the vmware bot id ? | 22:29 |
| sdague | so I could find some runs in gerrit to confirm what they are doing | 22:29 |
| *** rpodolyaka has joined #openstack-infra | 22:30 | |
| mordred | clarkb, sdague how about we figure out a way to flag images that the gate doesn't need to download some how? | 22:30 |
| *** harlowja has quit IRC | 22:30 | |
| jgriffith | fungi: that wasn't nice BTW :) | 22:31 |
| sdague | mordred: right, well I was thinking about that in terms of evaling with hypervisor | 22:31 |
| *** harlowja has joined #openstack-infra | 22:31 | |
| jgriffith | fungi: re the comment about "most deserving of it's ire" | 22:31 |
| sdague | mordred: http://paste.openstack.org/show/73317/ | 22:31 |
| sdague | every image add in devstack is behind some conditional | 22:31 |
| sdague | so if you just set those conditionals, you could eval out all the images | 22:32 |
| jeblair | sdague: keep in mind that we have to do this for every branch of devstack, so we need a fairly stable api | 22:32 |
| sdague | jeblair: that's been pretty stable | 22:32 |
| sdague | VIRT_DRIVER has been there forever | 22:32 |
| jeblair | sdague: yeah, but 'functions' hasn't | 22:33 |
| sdague | we'll make sure to make it more stable if we know it's something that will be externally called. I don't have an issue with that | 22:33 |
| jeblair | sdague: cool | 22:33 |
| sdague | jeblair: sure, well we could also provide the script in devstack | 22:33 |
| mordred | hell - I'd say we just do that snippet above ^^ | 22:33 |
| mordred | and just do it for libvirt - since d-g doesn't do xenserver either | 22:33 |
| clarkb | mordred: no | 22:34 |
| sdague | mordred: yeh, that was my suggestion. There is probably one other variable for heat that's needed as well | 22:34 |
| mordred | and as soon as d-g does xenserver or lxc or anything else, we can add a thing to VIRT_DRIVERS | 22:34 |
| jeblair | mordred: not yet; it might possibly could, and actually that could get a lot easier with glance | 22:34 |
| mordred | yah | 22:34 |
| clarkb | I don't like it I don't want to update nodepool each time we need a new hypervisor | 22:34 |
| jeblair | there are people working on that | 22:34 |
| mordred | but as long as it's just a list | 22:34 |
| clarkb | docker, xen, etc are all waiting | 22:34 |
| clarkb | and if they have to wait for new images thats a huge delay | 22:34 |
| mordred | jeblair: I'm quite excited about glance | 22:34 |
| clarkb | if we can just download some sane images then we are set | 22:34 |
| sdague | jeblair: so what if we provided a devstack tool that let you declare a list of hypervisors you care about on the command line | 22:34 |
| sdague | and it dumps image urls yuo need | 22:34 |
| jeblair | sdague: and make that tool name/command like argument the stable api? | 22:35 |
| sdague | instead of the parser, which I know I broke you guys | 22:35 |
| sdague | jeblair: yep | 22:35 |
| mordred | ++ | 22:35 |
| jeblair | sdague: sounds lovely if we can get clarkb on board with the idea :) | 22:35 |
| sdague | we'll put it in devstack and backport it | 22:35 |
| clarkb | that still requires a huge turn around delay on getting support for new stuff | 22:35 |
| mordred | clarkb: what don't you like it? | 22:35 |
| sdague | so, I've got something else which will make clarkb happy immediately | 22:35 |
| clarkb | mordred: beacuse you are talking days of waiting, get image in devstack, get nodepool to update, wait for images | 22:35 |
| *** jergerber has joined #openstack-infra | 22:35 | |
| jeblair | clarkb: it's worth noting that devstack does work with a cache miss in this system | 22:36 |
| sdague | turns out vmware doesn't use this image in their CI - http://208.91.1.172/logs/77736/5/console.txt | 22:36 |
| clarkb | with the current system docker/xen/whoever could just start doing stuff | 22:36 |
| sdague | they use a cirros image from the same url | 22:36 |
| sdague | which will be tons smaller | 22:36 |
| mordred | clarkb: what? | 22:36 |
| mordred | clarkb: I'm not trying to troll - I am not following what you are talking about at all | 22:36 |
| jeblair | clarkb: so you can actually start doing your devstack work etc before updating the image cacher to add your hypervisor | 22:36 |
| clarkb | jeblair: with a potentially massive delay as we are seeing | 22:36 |
| jeblair | clarkb: and, tbh, in 3 years we still haven't added a hypervisor. | 22:37 |
| clarkb | jeblair: thats fair | 22:37 |
| clarkb | jeblair: but, docker and xen are wanting to | 22:37 |
| clarkb | and I expect them to in juno | 22:37 |
| jeblair | clarkb: we could go ahead and put them in our initial list | 22:37 |
| sdague | clarkb: https://review.openstack.org/80101 | 22:39 |
| *** malini is now known as malini_afk | 22:39 | |
| sdague | so that will make you immediately more happy. | 22:39 |
| jeblair | clarkb: moreover, with glance we might end up having 'devstack-foo-xen' 'devstack-foo-kvm' images anyway, with the nodepool scripts having a bit more knowledge of hypervisors... | 22:39 |
| *** jamielennox|away is now known as jamielennox | 22:40 | |
| sdague | ok, so I need to run away. if we want a tool in devstack that takes hypervisors and spits out images, I am happy to do that | 22:41 |
| *** jergerber has quit IRC | 22:41 | |
| sdague | but I'll let folks decide | 22:41 |
| sdague | regardless we should be able to fix the giant debian image issue in the short term | 22:41 |
| sdague | per patch above | 22:41 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Download each cached image once. https://review.openstack.org/80102 | 22:42 |
| clarkb | jeblair: mordred ^ you will like that | 22:42 |
| clarkb | and I think that fixes a majority of the problem here | 22:43 |
| sdague | clarkb: well dtroyer just approved the switch to cirros as well | 22:45 |
| sdague | so that will also help | 22:45 |
| clarkb | woot | 22:45 |
| *** MarkAtwood has quit IRC | 22:45 | |
| sdague | ok, running away for real, have a good night | 22:45 |
| *** sarob__ has joined #openstack-infra | 22:46 | |
| *** Daviey has quit IRC | 22:46 | |
| *** Sukhdev has quit IRC | 22:47 | |
| *** changbl has joined #openstack-infra | 22:47 | |
| *** mrodden has quit IRC | 22:48 | |
| *** sarob has quit IRC | 22:49 | |
| *** sarob__ has quit IRC | 22:50 | |
| *** emagana has quit IRC | 22:51 | |
| *** hogepodge has quit IRC | 22:53 | |
| *** MarkAtwood has joined #openstack-infra | 22:53 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 22:55 |
| clarkb | jeblair: mordred fungi SergeyLukjanov I held 15.126.230.111 it booted off of the hpcloud-region-b image. It came up with 8GB of RAM and no non standard apt lists | 22:55 |
| clarkb | figured I would give folks an opportunity to take a look but it seems happy | 22:55 |
| jeblair | clarkb, pleia2, mattoliverau: ^ i responded to your comments; maybe we should have further discussion here? | 22:55 |
| clarkb | jeblair: let me look | 22:55 |
| *** alex-out is now known as Alexandra | 22:56 | |
| jeblair | oh, i guess gerrit doesn't order those things... | 22:56 |
| *** Daviey has joined #openstack-infra | 22:56 | |
| jeblair | um, let me know if you have trouble matching my response to your comment. :) | 22:57 |
| clarkb | jeblair: for the centos thing, puppet will set resolve.conf to 127.0.0.1 | 22:57 |
| mordred | clarkb: woot | 22:57 |
| jeblair | clarkb: my response to you was "I tested this on a centos node. Can you point me at where you're talking about?" | 22:57 |
| clarkb | jeblair: then the nodepool script addition will run and make the unbound talk to localhost instead of external resolver | 22:58 |
| jeblair | clarkb: oh, you're talking about my own change... | 22:58 |
| jeblair | that makes sense. :) | 22:58 |
| clarkb | jeblair: https://review.openstack.org/#/c/80092/1/modules/unbound/manifests/init.pp line 50 | 22:58 |
| clarkb | yup | 22:58 |
| *** _sweston_ has joined #openstack-infra | 22:58 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 22:59 |
| jeblair | clarkb: thank you, that was a very nice "doh!" catch. :) | 22:59 |
| *** sweston has quit IRC | 22:59 | |
| clarkb | np | 23:00 |
| openstackgerrit | Tom Fifield proposed a change to openstack-infra/config: Enable Welcome Message to post review comments https://review.openstack.org/80111 | 23:00 |
| jeblair | clarkb: so as to your suggestion about getting a list of nameservers... where should we get them? | 23:00 |
| jeblair | mattoliverau: ^ goes to your comment too... | 23:00 |
| clarkb | now the other thing to worry about | 23:01 |
| clarkb | will dhcp update the nameserver for us? | 23:01 |
| jeblair | clarkb: what dhcp? | 23:01 |
| clarkb | because that would be annoying | 23:01 |
| clarkb | jeblair: the dhcp run by the cloud provider to set things when VMs come up | 23:01 |
| clarkb | iirc nova has a really low lease time so we would be picking up new leases at least a few times during a test | 23:01 |
| openstackgerrit | A change was merged to openstack-infra/config: Download each cached image once. https://review.openstack.org/80102 | 23:02 |
| *** dims_ has quit IRC | 23:02 | |
| clarkb | jeblair: this is easy to test, I will change the nameserver on one of my hpcloud instances | 23:02 |
| clarkb | then wait and see if a new lease updates it (can rpobably also look in /var/log/dhclient/leases* | 23:02 |
| jeblair | clarkb: well, rackspace writes /etc/network/interfaces | 23:02 |
| jeblair | clarkb: and uses a static config | 23:02 |
| clarkb | huh no dhclient, maybe that is how hpcloud does it too | 23:03 |
| *** hogepodge has joined #openstack-infra | 23:03 | |
| clarkb | ah /var/lib | 23:03 |
| *** adalbas has quit IRC | 23:03 | |
| jeblair | clarkb: ? | 23:03 |
| clarkb | jeblair: option domain-name-servers 8.8.4.4,8.8.8.8; is set in the hpcloud lease | 23:03 |
| clarkb | jeblair: we can do something to ignore that option. I am trying to remember what; haven't had to do it in years | 23:04 |
| *** melwitt has quit IRC | 23:04 | |
| jeblair | clarkb: so hpcloud does run dhclient? | 23:04 |
| clarkb | option dhcp-rebinding-time 151200; is not as short as I thought it may be | 23:05 |
| *** mark0n has quit IRC | 23:05 | |
| clarkb | jeblair: yup | 23:05 |
| jeblair | clarkb: also, hpcloud uses google for their dns servers ?!?!?! | 23:05 |
| pleia2 | jeblair: so I just manually edited /etc/resolv.conf on an hpcloud ubuntu instance, when I rebooted it had reset back to the old one | 23:05 |
| clarkb | jeblair: apparently | 23:05 |
| clarkb | pleia2: ya because dhcp | 23:05 |
| clarkb | pleia2: check your leases in /var/lib/dhcp/ | 23:05 |
| pleia2 | yep, as I mentioned in my comment | 23:05 |
| jeblair | pleia2: your comment was about a bit of code that didn't write to /etc/resolv.conf | 23:06 |
| jeblair | pleia2: nothing i did in that patch writes to /etc/resolv.conf on ubuntu | 23:06 |
| *** atiwari has quit IRC | 23:06 | |
| jeblair | pleia2: instead, unbound tells resolvconf about itself and so resolvconf writes /etc/resolv.conf with the localhost value | 23:07 |
| pleia2 | jeblair: oh, ugh, I misread :( | 23:07 |
| pleia2 | yeah | 23:07 |
| clarkb | jeblair: /etc/dhcp/dhclient-enter-hooks.d/resolvconf pushes this information to resolvconf. So resolvconf may do the right thing if unbound wins over dhcp | 23:08 |
| clarkb | which honestly I think dhcp should win in a properly designed env | 23:08 |
| *** maxbit_ has quit IRC | 23:08 | |
| jeblair | clarkb: you think dhcp should override a locally configured resolver? | 23:08 |
| jeblair | clarkb: i strongly disagree | 23:08 |
| clarkb | jeblair: yes, because dhcp's job is to configure stuff | 23:09 |
| jeblair | clarkb: dhcp doesn't have the right context to make that decision | 23:09 |
| *** maxbit has quit IRC | 23:09 | |
| clarkb | you start having rogue devices otherwise | 23:09 |
| clarkb | jeblair: sure it does, you control dhcp server and clients | 23:09 |
| *** mriedem has joined #openstack-infra | 23:09 | |
| clarkb | usually | 23:09 |
| jeblair | clarkb: if a sysadmin installs a local recursive resolver, it _clearly_ should be used | 23:09 |
| jeblair | clarkb: you may not have noticed that we do not control the dhcp server | 23:09 |
| jeblair | clarkb: and the people who do are making really bad decisions about dns | 23:09 |
| clarkb | jeblair: right hence my comment about proper env | 23:09 |
| clarkb | cloud is special | 23:09 |
| jeblair | so, anyway... back to getting this working | 23:10 |
| jeblair | clarkb: when i look at an hpcloud centos node, i see nameserver 10.6.0.1 in resolv.conf | 23:10 |
| *** mrodden has joined #openstack-infra | 23:10 | |
| clarkb | jeblair: I can isntall unbound on my test box and reboot it to see what happens | 23:10 |
| jeblair | clarkb: that does not seem to equal 8.8.8.8 | 23:10 |
| clarkb | jeblair: right I think the lease time is long enough we will only see trouble at boot | 23:10 |
| jeblair | clarkb: i'm not following | 23:11 |
| *** nati_ueno has quit IRC | 23:11 | |
| clarkb | jeblair: when we boot off of the image you have made with that nameserver, it will ask for a dhcp lease | 23:12 |
| clarkb | if resolvconf lets dhcp win then we have a problem | 23:12 |
| jeblair | clarkb: if you see 8.8.8.8 in the lease and i see 10.6.0.1 in resolv.conf, that suggests to me that resolv.conf is not affected by dhcp at all :/ | 23:12 |
| *** nati_ueno has joined #openstack-infra | 23:12 | |
| clarkb | jeblair: but have you rebooted that machine | 23:12 |
| jeblair | clarkb: i'm not talking about an image with any of the new unbound stuff | 23:12 |
| jeblair | clarkb: i'm just talking about what we have now | 23:12 |
| clarkb | oh | 23:12 |
| clarkb | huh | 23:12 |
| *** melwitt has joined #openstack-infra | 23:13 | |
| clarkb | jeblair: my resolv.conf is google | 23:13 |
| clarkb | this is a node in hpcloud-region-b | 23:13 |
| jeblair | clarkb: oh i used azx | 23:13 |
| jeblair | i can't tell you how excited i am that we have 6 combinations to test. | 23:14 |
| jeblair | clarkb: is dhclient running on your region b node? | 23:14 |
| clarkb | jeblair: yes | 23:15 |
| * clarkb sets to dyndns and reboots | 23:15 | |
| *** dims_ has joined #openstack-infra | 23:16 | |
| clarkb | jeblair: yup dhcp reset my nameserver. Now I am going to try unbound | 23:17 |
| jeblair | okay, so it seems like hpcloud has dhclient running in both clouds, and sends 10.6.0.1 in old and google in new. | 23:17 |
| jeblair | clarkb: the answer is probably different with centos and ubuntu too -- there's a chance resolvconf will do the right thing and leave it on ubuntu; but i'm assuming dhclient writes resolv.conf directly on hpcloud centos. | 23:18 |
| fungi | jgriffith: cinder isn't deserving my *my* ire at all! i can't help if some irc bot decides to take out a vendetta ;) | 23:18 |
| jeblair | i <3 cinder | 23:19 |
| fungi | jeblair: clarkb: most dhcp clients allow you to configure which options to request, which options to ignore, and which list options you might want to apprend/prepend to | 23:19 |
| fungi | i will just about guarantee i can find you a config option to let you override the dhcp-suggested resolver list | 23:20 |
| jeblair | fungi: yeah, it seems like we might need to reconfigure dhclient on hpcloud, and do nothing on raxspace (which uses static config) | 23:20 |
| * fungi looks to see which client hpcloud is installing | 23:20 | |
| jeblair | /sbin/dhclient -1 -q -cf /etc/dhcp/dhclient-eth0.conf -lf /var/lib/dhclient/dhclient-eth0.leases -pf /var/run/dhclient-eth0.pid eth0 | 23:20 |
| clarkb | fungi: yes I know I was suggesting we do that | 23:20 |
| jeblair | dhclient3 -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1 eth0 | 23:21 |
| fungi | oh, that's what i get for skimmingh | 23:21 |
| fungi | skimming | 23:21 |
| jeblair | centos and ubuntu, respectively | 23:21 |
| clarkb | I have unbound resolving on my test box | 23:21 |
| clarkb | rebooting now to see what happens to resolv.conf | 23:21 |
| *** atiwari has joined #openstack-infra | 23:21 | |
| clarkb | jeblair: looks like unbound won on precise hpcloud | 23:22 |
| jeblair | yeah, so i think we're set on precise everywhere thanks to resolvconf. i never thought i'd say that. | 23:23 |
| *** yassine has quit IRC | 23:23 | |
| jeblair | then on rax-centos, we just write /etc/resolv.conf, and on hpcloud-centos, we reconfigure dhclient? | 23:23 |
| clarkb | jeblair: should be safe to reconfigure dhclient on all nodes and not think to hard about it | 23:24 |
| clarkb | not sure if we want to be specific or not if one node type doesn't deal with this properly | 23:24 |
| *** gyee has joined #openstack-infra | 23:24 | |
| jeblair | clarkb: would be good to avoid it since currently puppet has no idea if we're on hpcloud or rax. | 23:24 |
| fungi | dhclient-4.1.1-38.P1.el6.centos.x86_64 on centos6, config is in /etc/dhcp/dhclient-eth0.conf with a /etc/dhcp/dhclient.d run-parts dir | 23:24 |
| *** atiwari has quit IRC | 23:24 | |
| jeblair | fungi, clarkb: that same version is also installed on rax | 23:25 |
| jeblair | clarkb: so yeah, seems like the idea of 'just reconfigure dhclient everywhere' should work | 23:26 |
| jeblair | or at least, for all centos | 23:26 |
| clarkb | ya | 23:26 |
| *** wchrisj has quit IRC | 23:26 | |
| fungi | isc-dhcp-client on precise with /etc/dhcp/dhclient.conf and dhclient-enter-hooks.d and dhclient-exit-hooks.dm run-parts dirs | 23:27 |
| fungi | we want to 'prepend domain-name-servers 127.0.0.1;' | 23:27 |
| *** sweston has joined #openstack-infra | 23:28 | |
| fungi | at least that's what i've usually done with dhclient | 23:28 |
| jeblair | fungi: i'd like the other servers not to show up at all, is that possible? | 23:28 |
| fungi | precise has a commented-out example of that in the conffile | 23:28 |
| jeblair | (rather than prepend) | 23:28 |
| fungi | jeblair, pretty sure that's 'supersede domain-name-servers 127.0.0.1;' then | 23:29 |
| *** _sweston_ has quit IRC | 23:29 | |
| fungi | gotta love the twenty-five-cent words the dhclient maintainers decided on for their config options | 23:30 |
| jeblair | i'm going broke | 23:30 |
| *** dcramer__ has quit IRC | 23:30 | |
| * fungi spent all his twenty-five cent words on cheap booze | 23:31 | |
| jeblair | fungi, clarkb: do we need to append that to /etc/dhcp/dhclient-eth0.conf ? | 23:31 |
| *** denis_makogon has quit IRC | 23:32 | |
| fungi | that should work fine. anywhere in the global scope, just not inside other config scope blocks like lease{}; or alias{}; | 23:32 |
| *** maxbit has joined #openstack-infra | 23:32 | |
| jeblair | send vendor-class-identifier "anaconda-Linux 2.6.32-431.el6.x86_64 x86_64"; | 23:32 |
| jeblair | timeout 45; | 23:32 |
| *** mbacchi has quit IRC | 23:32 | |
| jeblair | that's the entirety of the existing config ^ | 23:32 |
| jeblair | different numbers on rax v hpcloud... | 23:32 |
| *** maxbit_ has joined #openstack-infra | 23:33 | |
| jeblair | i'm not sure what we should do with puppet in that case? | 23:33 |
| fungi | yeah, the centos one is wonderfully terse. the precise one has a lot more to ir | 23:33 |
| fungi | to it | 23:33 |
| clarkb | jeblair: dump a file in the .d dir? | 23:33 |
| clarkb | with one line overriding nameservers? | 23:33 |
| fungi | for centos there's just the one run-parts dir so that ought to work | 23:33 |
| jeblair | i thought that just ran scripts | 23:33 |
| jeblair | i didn't think it was included config | 23:33 |
| fungi | oh, wait. looking at the ntp example there... | 23:34 |
| jeblair | dhclient.d/ntp.sh | 23:34 |
| fungi | yep, script | 23:34 |
| fungi | so we do need to append to the config | 23:34 |
| clarkb | ah | 23:34 |
| clarkb | :/ | 23:34 |
| *** wenlock has quit IRC | 23:34 | |
| *** wenlock_ has joined #openstack-infra | 23:34 | |
| *** wenlock_ is now known as wenlock | 23:34 | |
| jeblair | so we need to do this? http://projects.puppetlabs.com/projects/puppet/wiki/Simple_Text_Patterns/5 | 23:35 |
| *** jhesketh has quit IRC | 23:36 | |
| fungi | that's why debubtu properly has the run-parts dirs called foo-hooks.d | 23:36 |
| *** maurosr has quit IRC | 23:37 | |
| *** mtreinish has quit IRC | 23:37 | |
| *** mtreinish has joined #openstack-infra | 23:37 | |
| *** maurosr has joined #openstack-infra | 23:37 | |
| fungi | jeblair: i had never seen that facility of puppet before | 23:38 |
| fungi | that looks way better than some of the places we've just slurped in entire conffiles | 23:38 |
| jeblair | fungi: i don't think it is a facility | 23:38 |
| jeblair | fungi: i think it's way better to slurp in conffiles; having config management edit files is almost always pain | 23:38 |
| fungi | s/facility/insane hack/ | 23:39 |
| *** david_lyle_ has joined #openstack-infra | 23:39 | |
| fungi | i've probably just never gotten to witness some of the myriad of ways that can go horribly wrong | 23:39 |
| *** VijayTripathi has joined #openstack-infra | 23:40 | |
| clarkb | there is parsed file | 23:40 |
| clarkb | which is clunky | 23:41 |
| *** david-lyle has quit IRC | 23:41 | |
| fungi | i wonder whether completely normalizing the dhclient config between platforms/providers would present new problems | 23:41 |
| jeblair | this is another "multi cloud is hard" moment. | 23:42 |
| *** sweston has quit IRC | 23:42 | |
| *** dcramer__ has joined #openstack-infra | 23:43 | |
| clarkb | does centos not do resolvconf? | 23:45 |
| fungi | i thought resolvconf was an ubuntuism which got up-ported into debian, though i could be wrong | 23:46 |
| *** CaptTofu has joined #openstack-infra | 23:46 | |
| jeblair | yum search resolvconf comes up empty | 23:46 |
| fungi | starting as basically a hackish dhcp client hook script which made in-place edits to your resolv.conf file | 23:46 |
| clarkb | gotcha | 23:47 |
| fungi | one of those innovations to ease linux adoption in the average joe desktop market | 23:47 |
| *** SumitNaiksatam has joined #openstack-infra | 23:48 | |
| *** wchrisj has joined #openstack-infra | 23:48 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Use unbound https://review.openstack.org/80092 | 23:49 |
| jeblair | fungi: turns out it inadvertently eased it in the server market as well.. | 23:49 |
| jeblair | okay, so that change should do the dhclient thing (i think)... | 23:49 |
| jeblair | one more thing while we're brainstorming though... | 23:50 |
| fungi | indeed | 23:50 |
| jeblair | clarkb and mattoliverau: both suggested having puppet configure the forward ips | 23:50 |
| *** CaptTofu has quit IRC | 23:50 | |
| *** yamahata has joined #openstack-infra | 23:50 | |
| jeblair | i chose not to do that because i figured reading /etc/resolv.conf would give us the ips on all os/provider combinations... | 23:50 |
| jeblair | i think that's still the case, yeah? | 23:50 |
| clarkb | jeblair: yes, that should still be the case | 23:51 |
| openstackgerrit | Arnaud Legendre proposed a change to openstack/requirements: Update oslo.vmware to be >= 0.3 https://review.openstack.org/80120 | 23:51 |
| jeblair | given what we know, would it make sense to do it in puppet? | 23:51 |
| mattoliverau | sorry, been in a meeting.. reading scroll back | 23:51 |
| *** jhesketh_ has joined #openstack-infra | 23:51 | |
| jeblair | i mean, i think the fact would have to read the dhcp lease on hpcloud and the /etc/network/interfaces or /etc/system/whatever files on rax | 23:51 |
| jeblair | right? | 23:51 |
| jeblair | so that's at least 3 different files (with different syntaxes) that we would need to parse | 23:52 |
| clarkb | jeblair: probably. I think we can start with what you have there. Mostly my concern is/was that if you run puppet on a machine without the nodepool script you will get a half configured node | 23:52 |
| clarkb | jeblair: for things related to image building thats fine (boot parameters and so on) but the unbound service is configured half in pupet and half in nodepool | 23:53 |
| jeblair | clarkb: actually we will run the puppet on all machines | 23:53 |
| *** Alexandra is now known as alex-afk | 23:53 | |
| jeblair | clarkb: that's going to run on servers too, and configure unbound, and the nodepool bit is not going to run | 23:53 |
| jeblair | clarkb: which is fine, because that will give us unbound in the default recursive resolver config, which is what we want on servers | 23:54 |
| *** resker has quit IRC | 23:54 | |
| clarkb | oh right, nodepool is just doing forwarding config for us | 23:54 |
| *** gokrokve_ has quit IRC | 23:54 | |
| jeblair | clarkb: so it's sort of like every host (nodepoor or not) gets unbound, then nodepool hosts drop in a forwarding config | 23:54 |
| clarkb | wfm | 23:54 |
| fungi | that feels safe enough | 23:55 |
| *** esker has joined #openstack-infra | 23:55 | |
| jeblair | (nodepoor is what we are right now) | 23:55 |
| *** gokrokve has joined #openstack-infra | 23:55 | |
| fungi | but soon, noderich! | 23:55 |
| fungi | profit! | 23:55 |
| *** SumitNaiksatam has quit IRC | 23:55 | |
| *** SumitNaiksatam has joined #openstack-infra | 23:56 | |
| *** gokrokve has quit IRC | 23:59 | |
| jogo | clarkb: 1291440 isn't showing up in the e-r even though the pages have regenerated | 23:59 |
| *** esker has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!