| anteaya | awesome | 00:00 |
|---|---|---|
| openstackgerrit | Tony Breeds proposed openstack-infra/project-config: Force the gate-irc-meetings-tox-ical postbuildscript to always run https://review.openstack.org/278790 | 00:00 |
| anteaya | mnaser: enjoy | 00:00 |
| mnaser | thanks for the help mordred clarkb jeblair anteaya ! :D | 00:00 |
| mordred | mnaser: any time! | 00:00 |
| mgagne | clarkb: you win all the internets sir thanks! | 00:00 |
| zaro | clarkb: what is xvdc and xvde in these graphs? http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=9&page=3 | 00:00 |
| tonyb | anteaya: commit message updated, let me knwo what you think about the other comment. | 00:00 |
| zaro | clarkb: also xvda and xvdb? | 00:01 |
| clarkb | zaro: they are block devices, xvde is the ephemeral device and xvdc is the ssd volume we put our repos on | 00:01 |
| clarkb | zaro: xvda is / and I don't know what b is maybe b is ssd and c is something else | 00:01 |
| clarkb | mordred: their ssl cert is not yet valid, is there a way with oscc/shade to specify untrusted ssl is ok? | 00:01 |
| anteaya | tonyb: will do | 00:02 |
| clarkb | mordred: not sure we have much alternative until they fix that (I guess we could hold off merging the osic change until they have proper cert and domain) | 00:02 |
| anteaya | firefox 44.0 is just not gerrit compatible | 00:02 |
| Shrews | clarkb: verify=False in clouds.yaml, iirc | 00:02 |
| zaro | clarkb: if we are using those for disks then what's sda1 and sda2? | 00:02 |
| clarkb | Shrews: danke | 00:02 |
| clarkb | I grepped for ssl and insecure but not verify | 00:03 |
| *** nwkarsten has joined #openstack-infra | 00:03 | |
| mordred | clarkb: yes | 00:03 |
| Shrews | clarkb: irc grep usually works :) | 00:03 |
| *** crinkle has quit IRC | 00:03 | |
| mordred | oh - Shrews got you | 00:03 |
| *** crinkle_ has joined #openstack-infra | 00:03 | |
| clarkb | zaro: the sda things may be historical? from when gerrit was on the legacy cloud | 00:04 |
| clarkb | zaro: not sure how cacti cleans that up | 00:04 |
| *** FallenPegasus has quit IRC | 00:04 | |
| *** crinkle_ is now known as crinkle | 00:04 | |
| clarkb | mordred: Shrews that would not go in the profile right? thats a per clouds.yaml setting? makes sense considering individual users will wnat to set what they are comfortable with for security | 00:04 |
| zaro | clarkb: and which one is the review_site on? it looks like it might be either xvdb or xvde but can't tell from the graph. | 00:04 |
| *** hdd has joined #openstack-infra | 00:05 | |
| JayF | I ^c'd git review in the middle (it had been hanging for a long time), and now my patch appears to be in a weird limbo. https://review.openstack.org/#/c/259135 doesn't have my change, but git-review fails now with ! [remote rejected] HEAD -> refs/publish/master/bug/1526918 (no new changes) (full logs: https://gist.github.com/jayofdoom/2fdcfe7751137894b70b) | 00:05 |
| mordred | clarkb: no- it should be a per-cloud setting | 00:05 |
| mordred | clarkb: in fact, it certainly is, because I know unitedstack had broken certs for a while | 00:06 |
| jeblair | JayF: yep, you'll want to amend it and push a new patchset | 00:06 |
| mordred | clarkb: ustack: | 00:06 |
| mordred | profile: unitedstack | 00:06 |
| mordred | #verify: False | 00:06 |
| clarkb | zaro: looks like xvda is /, xvde is /opt, and /home/gerrit2 is on xvdb and c via lvm | 00:06 |
| mordred | clarkb: that's from my clouds.yaml | 00:06 |
| JayF | jeblair: so just amend my change with something enough to change the sha1 and repush? WFM | 00:06 |
| jeblair | JayF: yep. the issue was gerrit database backup at 0000utc | 00:06 |
| mordred | clarkb: I think we should wait a bit before merging a verify: false in oscc itself | 00:06 |
| JayF | jeblair: aha, thanks :) | 00:07 |
| mordred | clarkb: so for now maybe yo just put it in your clouds.yaml | 00:07 |
| clarkb | mordred: agreed or never at all and just do it in clouds.yaml | 00:07 |
| mordred | yah | 00:07 |
| mordred | I'm not sure why they don't have real certs | 00:07 |
| clarkb | mordred: ok I think I have what I need, time to instll my change and openstack client and see if it works | 00:07 |
| mordred | but I won't ask that questoin just right now | 00:07 |
| clarkb | mordred: because they don't have a domain | 00:07 |
| mordred | oh | 00:07 |
| mordred | gotcha | 00:07 |
| clarkb | I am told it is in progress | 00:07 |
| jeblair | mordred: i think we may have outgrown a simple mysqldump and maybe it's time for replication or some other backup strategy. :) | 00:07 |
| *** nelsnels_ has joined #openstack-infra | 00:07 | |
| mordred | jeblair: agree | 00:07 |
| *** nelsnelson has quit IRC | 00:08 | |
| mordred | jeblair: maybe it's time to make a plan for that | 00:08 |
| jeblair | mordred: well, at this time of day, i can only make a plan to make a plan for that. | 00:09 |
| anteaya | tonyb: good enough, thank you | 00:09 |
| mordred | jeblair: can we make a plan for the plan making plan? | 00:10 |
| tonyb | anteaya: Thanks. | 00:10 |
| *** armax has joined #openstack-infra | 00:10 | |
| anteaya | :) | 00:10 |
| anteaya | mordred: you'll be on a plane | 00:10 |
| *** denisra has joined #openstack-infra | 00:12 | |
| tristanC | clarkb: what if I need multiple jobs to complete before doing a publish job for example, is this possible ? | 00:13 |
| *** Daisy has joined #openstack-infra | 00:14 | |
| *** sabeen3 has quit IRC | 00:14 | |
| tristanC | that is, in the same pipeline | 00:14 |
| clarkb | tristanC: it is not | 00:15 |
| clarkb | I believe that is one of the features that zuulv3 will add | 00:15 |
| *** rocky_g has quit IRC | 00:15 | |
| tristanC | so it's only one to many with zuulv2 | 00:15 |
| *** annegentle_ has joined #openstack-infra | 00:15 | |
| clarkb | mordred: woot I have openstackclient working with a clouds.yaml and my vendor profile so will push this up momentarily | 00:15 |
| clarkb | tristanC: yes | 00:15 |
| tristanC | clarkb: thank! | 00:15 |
| *** nwkarsten has quit IRC | 00:15 | |
| clarkb | tristanC: you can serialize them all if it is really important | 00:15 |
| tristanC | clarkb: well the use-case would be to do publish job in the gate, so that if the publish failed, the change isn't merged | 00:16 |
| *** Daisy has quit IRC | 00:18 | |
| openstackgerrit | Jamie Lennox proposed openstack-infra/project-config: Run the identity v3 only job in additional services https://review.openstack.org/271128 | 00:19 |
| *** annegentle_ has quit IRC | 00:19 | |
| tristanC | (though this also mean the publish would have to wait to be on top of the gate to actually happen) | 00:20 |
| *** annegentle_ has joined #openstack-infra | 00:20 | |
| clarkb | ok have my first devstack tempest run going on osic on a base ubuntu image | 00:23 |
| clarkb | will sort out getting one of our qcow2's up there next | 00:23 |
| *** dchen has joined #openstack-infra | 00:25 | |
| *** hashar_ has quit IRC | 00:25 | |
| anteaya | yay! | 00:25 |
| *** rbrndt_ has quit IRC | 00:26 | |
| *** rkukura has quit IRC | 00:27 | |
| clarkb | jeblair: this reproduce.sh script is like the best thing ever | 00:28 |
| *** jsavak has quit IRC | 00:28 | |
| *** xyang1_ has joined #openstack-infra | 00:28 | |
| *** xyang1 has quit IRC | 00:28 | |
| *** xyang1_ is now known as xyang1 | 00:28 | |
| clarkb | E: Unable to locate package liberasurecode-dev thats interesting | 00:29 |
| *** ganso has quit IRC | 00:29 | |
| armax | hi folks, we have a couple of graphite links bookmarked here: http://docs.openstack.org/developer/neutron/dashboards/#neutron-graphite-pages but they look a bit broken right now..doesn’t anyone have pointers? | 00:29 |
| *** ybathia has quit IRC | 00:30 | |
| clarkb | is that in backports? | 00:30 |
| *** claudiub has joined #openstack-infra | 00:31 | |
| armax | um, now I get a notfound right of the bat | 00:31 |
| *** vmbrasseur is now known as vmb_off_record | 00:32 | |
| *** vmb_off_record is now known as vmbrasseur | 00:32 | |
| clarkb | armax: first thing I would check is if a local build of the docs produce those two files | 00:32 |
| armax | clarkb: ok, it did in the past…that was working a minute ago, but the graphite graphs looked wrong | 00:33 |
| clarkb | armax: but I am guessing that the docs site has some htpassword magic preventing those two file "extensions" ? | 00:33 |
| armax | clarkb: let me double check | 00:34 |
| armax | clarkb: I guess this has worked by accident so far? | 00:34 |
| clarkb | armax: it would not surprise me if the docs team updated their htpassword rules | 00:34 |
| jeblair | clarkb: yay! | 00:35 |
| clarkb | but first thing is to confirm the builds actually make those files | 00:35 |
| armax | clarkb: checking | 00:35 |
| armax | I get a build error locally now | 00:35 |
| clarkb | I uncommented all the repos in sources.list fwiw and liberasurecode-dev showed up | 00:35 |
| armax | something merged recently and broke the docs job | 00:35 |
| clarkb | aptitude says it is in universe which I thought was enabled beofre so maybe universe backports? | 00:36 |
| jeblair | clarkb: http://packages.ubuntu.com/search?keywords=liberasurecode-dev | 00:36 |
| jeblair | clarkb: yeah, trusty-backports universe | 00:36 |
| armax | clarkb: I am still waiting on https://review.openstack.org/#/c/229951/ | 00:36 |
| armax | :) | 00:36 |
| clarkb | "Running setup.py bdist_wheel for numpy: still running..." how nice of it to tell me it is still spinning the cpu :) | 00:39 |
| clarkb | jeblair: but this should give us an upper bound on how bad a run is without wheels and cached info and so on | 00:39 |
| clarkb | I guess I could've added the wheel mirror first, oh well | 00:40 |
| mordred | clarkb: woot! | 00:40 |
| jeblair | clarkb: i think comparing the tempest runtimes as output by devstack is the best way to compare | 00:40 |
| jeblair | clarkb: eliminates all of that setup concern | 00:41 |
| clarkb | jeblair: good point, the tempest runtime should be ya that | 00:41 |
| jeblair | or rather, the tempest runtime as output by tempest | 00:41 |
| jeblair | (ran x tests in ...s) | 00:41 |
| openstackgerrit | Eric Brown proposed openstack/requirements: Revert "Only install mock on Python < 3.3" https://review.openstack.org/278814 | 00:41 |
| armax | clarkb: the page came back now: http://docs.openstack.org/developer/neutron/dashboards/gate.dashboard.html | 00:41 |
| armax | but the graph looks funny | 00:41 |
| armax | I am puzzled | 00:42 |
| clarkb | armax: that has the .html, the old links didnt | 00:42 |
| *** rkukura has joined #openstack-infra | 00:42 | |
| clarkb | which lends some weight to the htaccess theory | 00:42 |
| armax | clarkb: that’s bizzarre indeed | 00:42 |
| clarkb | fwiw, you might find using our grafana instance is a better place for that stuff? | 00:42 |
| armax | because I am simply browsing http://docs.openstack.org/developer/neutron from my browser | 00:42 |
| jeblair | armax: you can add dashboards to grafana if you want | 00:42 |
| jeblair | clarkb: jinx | 00:43 |
| *** arthus has quit IRC | 00:43 | |
| armax | jeblair: tell me more :) | 00:43 |
| jeblair | armax: http://grafana.openstack.org | 00:43 |
| jeblair | armax: i made a graph similar to what's on that page here: http://grafana.openstack.org/dashboard/db/tempest-failure-rate | 00:43 |
| jeblair | armax: it's mostly a stub -- but it's an example | 00:44 |
| armax | jeblair: thanks for the pointers | 00:44 |
| jeblair | armax: look at some of the other dashboards there for other examples | 00:44 |
| * armax looks | 00:44 | |
| jeblair | armax: and each of those has a link to the file in project-config that configured the dashboard (all yaml) | 00:44 |
| jeblair | armax: eg: https://git.openstack.org/cgit/openstack-infra/project-config/tree/grafana/tempest.yaml | 00:44 |
| *** Swami has quit IRC | 00:44 | |
| armax | jeblair: that’s excellent | 00:45 |
| armax | ! | 00:45 |
| jeblair | armax: main thing it gets you over raw graphite, aside from pretty, is zooming and correlation | 00:45 |
| armax | jeblair: plus you get the nice URLs | 00:46 |
| jeblair | yep | 00:46 |
| armax | sweet | 00:46 |
| jeblair | oh look, if you zoom out far enough on the nodepool graphs, you can actually see the devstack runtime drop from the wheel mirrors :) | 00:46 |
| jeblair | (it takes a lot of data to get that to average out into something smooth) | 00:47 |
| armax | jeblair: so I can just post a patch adding a neutron.yaml file? | 00:47 |
| jeblair | http://grafana.openstack.org/dashboard/db/nodepool-rackspace?from=1454460426870&to=1455151626870 | 00:47 |
| jeblair | armax: yep! | 00:47 |
| armax | right on | 00:48 |
| *** annegentle_ has quit IRC | 00:48 | |
| armax | the CI jobs will also show how it looks? or does it need to be deployed to become live? | 00:48 |
| *** thorst_ has joined #openstack-infra | 00:49 | |
| jeblair | armax: we haven't quite worked that out yet; though i think pabelanger had some ideas. it will validate syntax. | 00:49 |
| armax | jeblair: ok | 00:49 |
| armax | jeblair: mine was just idle curiousity | 00:49 |
| jeblair | armax: also, you can use grafyaml to make the json and actually paste it into grafana (or edit json that's already there) | 00:49 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance https://review.openstack.org/278816 | 00:50 |
| armax | jeblair: noted | 00:50 |
| *** sabari has joined #openstack-infra | 00:50 | |
| *** browne has joined #openstack-infra | 00:50 | |
| armax | dougwig: ^^ this is useful info | 00:50 |
| mtreinish | mordred: ^^^ | 00:50 |
| mtreinish | it's not really ready yet though, it needs tests (and I need to test it) | 00:51 |
| mtreinish | s/test/try | 00:51 |
| browne | hi all. i did a revert of the g-r mock change. looks like its breaking py34 jobs for some projects | 00:51 |
| *** Sukhdev has quit IRC | 00:51 | |
| browne | https://review.openstack.org/#/c/278814/ | 00:51 |
| mordred | mtreinish: neat! | 00:51 |
| mordred | mtreinish: I hear that's a thing people do | 00:51 |
| *** jlk has quit IRC | 00:54 | |
| mtreinish | I know that I've done it before :) | 00:54 |
| *** jlk has joined #openstack-infra | 00:54 | |
| *** jlk has quit IRC | 00:54 | |
| *** jlk has joined #openstack-infra | 00:54 | |
| *** [1]Thelo has joined #openstack-infra | 00:55 | |
| *** Jeffrey4l has joined #openstack-infra | 00:56 | |
| *** rpioso has quit IRC | 00:56 | |
| *** Thelo has quit IRC | 00:57 | |
| *** [1]Thelo is now known as Thelo | 00:57 | |
| mtreinish | mordred: also: http://status.openstack.org/openstack-health/#/test/devstack?groupKey=project&resolutionKey=hour I'm guessing the wheel mirror went live on the 5th | 00:58 |
| mordred | mtreinish: nice! | 00:58 |
| mordred | jeblair: ^^ | 00:58 |
| jeblair | mtreinish: yep | 00:59 |
| *** ajmiller has quit IRC | 01:00 | |
| slogan621 | anteaya: is there anything I need to do to improve https://review.openstack.org/#/c/275873/ so it can get approval by you? I see you are real busy, and I'm trying to not bother you but feeling a little unsure if you are waiting for me on something | 01:02 |
| tonyb | lifeless: again not trying to be annoying but can you look at: https://bugs.launchpad.net/python-fixtures/+bug/1542984 I think that releaseing fixtures 1.2.1 is the right compromise position | 01:02 |
| openstack | Launchpad bug 1542984 in Python Fixtures "fixtures 1.2.0 isn't compatible with testtools 2.0.0" [Undecided,New] | 01:02 |
| tonyb | lifeless: FWIW I'm drafting an email re install -U but that's a tangent to the bug above | 01:03 |
| *** eil397 has quit IRC | 01:03 | |
| *** thorst_ has quit IRC | 01:05 | |
| *** thorst_ has joined #openstack-infra | 01:06 | |
| tonyb | browne: Can you include a bug ot logs for the py34 jobs affected by the mock change? | 01:07 |
| *** Jeffrey4l has quit IRC | 01:07 | |
| *** esker has joined #openstack-infra | 01:08 | |
| *** annegentle_ has joined #openstack-infra | 01:08 | |
| *** Sukhdev has joined #openstack-infra | 01:08 | |
| browne | tonyb: sure | 01:09 |
| clarkb | jeblair: http://paste.openstack.org/show/486641/ is osic with 4 cpu | 01:11 |
| anteaya | slogan621: I'm not waiting for you on anything | 01:12 |
| clarkb | jeblair: http://logs.openstack.org/44/278344/2/check/gate-tempest-dsvm-full/9bf45fc/console.html#_2016-02-10_14_12_03_400 is the change I got the reproduce.sh script form | 01:12 |
| clarkb | jeblair: it seems to hold up extremely well | 01:12 |
| anteaya | to be honest I'm not the best person to review new project patches | 01:12 |
| anteaya | as I feel new folks make far better use of their time if they spend time on current projects | 01:12 |
| anteaya | helping with bugs and testing and documentation | 01:12 |
| anteaya | to learn our workflow | 01:13 |
| anteaya | but the fashion seems to just launch your own project | 01:13 |
| anteaya | which I can't stop | 01:13 |
| anteaya | but I'm not a supporter | 01:13 |
| anteaya | and thanks for fixing the commit message | 01:13 |
| anteaya | anyway I'm at the end of my day | 01:13 |
| anteaya | good night | 01:14 |
| *** thorst_ has quit IRC | 01:14 | |
| sabari | browne: other projects like glance didn't break because they depend on oslotest that still depends on mock. | 01:14 |
| sabari | browne: https://review.openstack.org/#/c/272823/2/requirements.txt hasn't merged btw | 01:14 |
| browne | sabari: ah! | 01:14 |
| jeblair | clarkb: not shabby | 01:15 |
| jeblair | clarkb: is that 8cpu or 4? | 01:15 |
| clarkb | jeblair: 4vpcu 8GB ram and 96GB / | 01:15 |
| jeblair | cool | 01:16 |
| jeblair | then we'll just be ip limited :) | 01:16 |
| clarkb | jeblair: well they gave us the IPs so ya | 01:16 |
| browne | tonyb: where are bugs opened on g-r? | 01:16 |
| clarkb | need that oscc change in then I can propose a chnage to our clouds.yaml | 01:16 |
| clarkb | and I need to make a mirror | 01:16 |
| jeblair | clarkb: link to change? | 01:16 |
| tonyb | browne: normally in the reporting project | 01:17 |
| clarkb | jeblair: https://review.openstack.org/278803 | 01:17 |
| *** sridhar_ram has quit IRC | 01:17 | |
| tonyb | browne: it's just helpful for others to follow the bug -> resolution path as we tent not to document that in the commit message very well. | 01:17 |
| jeblair | clarkb: are non public (or semi public) clouds going into oscc? | 01:17 |
| clarkb | jeblair: mordred said I should push it, so I guess so? | 01:18 |
| clarkb | jeblair: there is a bluebox.json too which is only private clouds | 01:18 |
| *** annegentle_ has quit IRC | 01:18 | |
| mordred | yah | 01:18 |
| *** harlowja has quit IRC | 01:18 | |
| mordred | we have one semi-public and one vendor of private clouds | 01:18 |
| browne | tonyb: understood. normally i'd go to launchpad, but doesn't appear to be anything for openstack/requirements projects | 01:18 |
| mordred | catalyst is semi-public | 01:18 |
| *** chlong has joined #openstack-infra | 01:18 | |
| mordred | they are - but also they focus on government, so there's weird ip access restrictions | 01:19 |
| jeblair | mordred: who focuses on government? | 01:19 |
| tonyb | browne: I'd use: https://bugs.launchpad.net/bandit | 01:20 |
| *** Jeffrey4l has joined #openstack-infra | 01:20 | |
| browne | tonyb: ok | 01:20 |
| *** angdraug has quit IRC | 01:20 | |
| mordred | jeblair: catalyst | 01:20 |
| *** achanda has quit IRC | 01:21 | |
| tonyb | browne: I know I butted in, and I'm not being super helpful :/ | 01:21 |
| browne | np | 01:21 |
| mordred | jeblair: http://www.catalyst.net.nz/catalyst-cloud | 01:21 |
| jeblair | anyway, i'm not sure it's the call i would have made (seems like a lot of churn for not much benefit), but i don't think there's much point arguing against it. :) | 01:21 |
| mordred | clarkb: comment on the patch - it's missing docs entries | 01:21 |
| *** annegentle_ has joined #openstack-infra | 01:22 | |
| clarkb | oh thats not all generated from the json? | 01:22 |
| clarkb | I can add | 01:22 |
| jeblair | clarkb: you can, of course, proceed without the osic oscc change and switch to it later. | 01:22 |
| clarkb | jeblair: that is true | 01:23 |
| clarkb | and its not too bad to do that since it is very defaulty | 01:23 |
| jeblair | clarkb: when you spin up the mirror -- be aware that if their trusty image is built from the server image rather than the cloud image, it may have the lts enablement packages installed; our puppet will remove them but there may be some post-trusty kernel packages installed that need to be removed for openafs to compile | 01:24 |
| clarkb | jeblair: ok | 01:24 |
| *** sflanigan has joined #openstack-infra | 01:24 | |
| *** sflanigan has joined #openstack-infra | 01:24 | |
| jeblair | (this has only happened on internap so far) | 01:24 |
| clarkb | going off of what my test box is the kernel looks older so probably not a problem | 01:24 |
| clarkb | Linux clarkb-test 3.13.0-66-generic #108-Ubuntu | 01:25 |
| jeblair | oh you used one of their images rather than upload. easy to check then :) | 01:25 |
| clarkb | yup | 01:25 |
| jeblair | version looks right for trusty (though is 10 builds behind?) | 01:26 |
| openstackgerrit | Eric Brown proposed openstack/requirements: Revert "Only install mock on Python < 3.3" https://review.openstack.org/278814 | 01:26 |
| tonyb | browne: thanks | 01:26 |
| jeblair | clarkb: this will be the first cloud where using it will not be a multi-day process... | 01:27 |
| *** esker has quit IRC | 01:27 | |
| jeblair | clarkb: all we need to do is upload existing images and create the mirror server -- we don't need to *update* the images (assuming they use one of the existing formats), and the mirror doesn't need to build anything | 01:28 |
| jeblair | you could have it all done tomorrow :) | 01:28 |
| *** annegentle_ has quit IRC | 01:28 | |
| clarkb | mordred: change updated | 01:29 |
| clarkb | jeblair: yup uses qcow2 so images should be easy | 01:29 |
| clarkb | its just a matter of lining everything up so that nodepool can talk to it and start booting things | 01:30 |
| clarkb | I will get stuff in hiera tomorrow, propose clouds.yaml and nodepool.yaml changes | 01:30 |
| clarkb | start with 1 VM and go from there | 01:30 |
| nibalizer | clarkb: you can either parent on the infracloud stuff or not | 01:31 |
| nibalizer | it hasn't landed yet but I think it can, its got positive votes at least | 01:32 |
| clarkb | in that cas emaybe I should start by reviewing infracloud stuff | 01:32 |
| clarkb | so that it can go in then do osic changes | 01:32 |
| *** lane_kong has joined #openstack-infra | 01:32 | |
| *** achanda has joined #openstack-infra | 01:33 | |
| nibalizer | I think https://review.openstack.org/#/c/275477/ is the important one, since once that is in I/we can build a mirror in that region | 01:33 |
| nibalizer | it might be cool to try to build mirrors together | 01:33 |
| *** jasondotstar has quit IRC | 01:35 | |
| *** raginbajin has quit IRC | 01:35 | |
| *** eliqiao_ has quit IRC | 01:36 | |
| clarkb | jeblair: re comment on https://review.openstack.org/#/c/275477/3/modules/openstack_project/templates/puppetmaster/ansible-clouds.yaml.erb it has been the case that we haven't been able to use the same name | 01:36 |
| clarkb | that stopped being the case with hpcloud | 01:36 |
| clarkb | so we could do that for now | 01:36 |
| *** dstufft has quit IRC | 01:36 | |
| clarkb | until some random cloud has a conflict again | 01:36 |
| clarkb | oh and for password I am not sure | 01:36 |
| jeblair | clarkb: i specifically meant for both omfra east and west | 01:36 |
| *** eliqiao_ has joined #openstack-infra | 01:36 | |
| *** dims_ has joined #openstack-infra | 01:37 | |
| clarkb | oh gotcha | 01:37 |
| *** jasondotstar has joined #openstack-infra | 01:37 | |
| jeblair | my argument is that we use the same creds for rax iad/dfw/ord | 01:37 |
| clarkb | ooh looks like horizon has built in password changing now | 01:37 |
| jeblair | so we should use the same creds for omfra east/west | 01:37 |
| clarkb | jeblair: ++ | 01:37 |
| *** raginbajin has joined #openstack-infra | 01:37 | |
| clarkb | I should maybe change the passwds before we properly use this thing | 01:37 |
| clarkb | (osic) | 01:37 |
| jeblair | i chose not to push the argument though and left a +2 | 01:37 |
| nibalizer | I'm open to flattening it, really | 01:37 |
| *** annegentle_ has joined #openstack-infra | 01:38 | |
| *** dstufft has joined #openstack-infra | 01:38 | |
| nibalizer | if I get time before Im done working today I can flatten it | 01:38 |
| jeblair | nibalizer and i are both being equal parts stubborn and gracious ;) | 01:38 |
| clarkb | but now I need to go wrangle children | 01:38 |
| *** dims has quit IRC | 01:38 | |
| *** dims_ has quit IRC | 01:41 | |
| *** gildub has quit IRC | 01:41 | |
| *** apoorvad has quit IRC | 01:41 | |
| *** zz_dimtruck is now known as dimtruck | 01:42 | |
| *** alex_xu has quit IRC | 01:43 | |
| *** alex_xu has joined #openstack-infra | 01:45 | |
| nibalizer | some of the IC nodes do not apply puppet | 01:47 |
| nibalizer | they do not apply puppet because they don'thave the secrets | 01:47 |
| nibalizer | because they don't have infracloud.yaml copied over | 01:47 |
| nibalizer | because puppet_get_hiera_file_list is not returning 'group/infracloud.yaml' in its array | 01:48 |
| nibalizer | which makes very little sense because some hosts are working just fine | 01:48 |
| *** gyee has quit IRC | 01:48 | |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Add neutron tempest run using neutron-lib master to neutron exp https://review.openstack.org/278825 | 01:48 |
| *** dims has joined #openstack-infra | 01:49 | |
| *** hdd has quit IRC | 01:49 | |
| *** pvaneck has quit IRC | 01:50 | |
| *** rlandy has quit IRC | 01:51 | |
| *** annegentle_ has quit IRC | 01:51 | |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: Add distil jobs back https://review.openstack.org/278827 | 01:52 |
| *** baoli has joined #openstack-infra | 01:53 | |
| lane_kong | ping AJaeger | 01:53 |
| *** tiswanso has joined #openstack-infra | 01:53 | |
| slogan621 | anteaya: ok, that clears things up, thanks. | 01:55 |
| slogan621 | can anyone suggest a core reviewer who can tackle the simple process of approving a new project commit? I've got a few thousand lines of code to commit to that project, once it exists, so I would appreciate some help. Not trying to be pushy, but I'm blocked until that project gets created, so appreciate some help. | 01:58 |
| slogan621 | https://review.openstack.org/#/c/275873/ is the project | 01:59 |
| *** akshai has quit IRC | 01:59 | |
| *** tphummel has quit IRC | 01:59 | |
| *** eliqiao_ has quit IRC | 02:00 | |
| *** eliqiao_ has joined #openstack-infra | 02:00 | |
| *** baoli has quit IRC | 02:00 | |
| *** Daisy has joined #openstack-infra | 02:00 | |
| *** rbrndt has joined #openstack-infra | 02:01 | |
| *** rbrndt has quit IRC | 02:02 | |
| *** rossella_s has quit IRC | 02:03 | |
| *** Daisy has quit IRC | 02:03 | |
| *** edmondsw has quit IRC | 02:03 | |
| *** rossella_s has joined #openstack-infra | 02:03 | |
| *** asahlin has joined #openstack-infra | 02:04 | |
| *** mtanino has quit IRC | 02:04 | |
| *** Sukhdev_ has joined #openstack-infra | 02:06 | |
| *** browne has quit IRC | 02:07 | |
| *** dchen has quit IRC | 02:07 | |
| *** slogan621 has quit IRC | 02:07 | |
| *** alex_xu has quit IRC | 02:08 | |
| *** Jeffrey4l has quit IRC | 02:08 | |
| *** shashank_hegde has quit IRC | 02:08 | |
| *** ggillies has joined #openstack-infra | 02:09 | |
| *** megm_ has joined #openstack-infra | 02:10 | |
| *** alex_xu has joined #openstack-infra | 02:10 | |
| *** HeOS__ has joined #openstack-infra | 02:11 | |
| *** daemontool has quit IRC | 02:11 | |
| *** Jeffrey4l has joined #openstack-infra | 02:11 | |
| mestery | mordred: Any idea why https://review.openstack.org/#/c/272842/12 would be stuck and not goign into the merge queue? The dependent patch in neutron has merged. | 02:12 |
| *** annegentle_ has joined #openstack-infra | 02:12 | |
| *** thorst_ has joined #openstack-infra | 02:12 | |
| *** dpaterson has quit IRC | 02:14 | |
| *** fbo_ has joined #openstack-infra | 02:14 | |
| *** terryw has joined #openstack-infra | 02:14 | |
| *** Sukhdev has quit IRC | 02:15 | |
| *** bpokorny has quit IRC | 02:15 | |
| *** ctlaugh has quit IRC | 02:15 | |
| *** fbo has quit IRC | 02:15 | |
| *** otherwiseguy has quit IRC | 02:15 | |
| *** hamzy has quit IRC | 02:15 | |
| *** amitgandhinz has quit IRC | 02:15 | |
| *** openstackgerrit has quit IRC | 02:15 | |
| *** HeOS_ has quit IRC | 02:15 | |
| *** vsaienko has quit IRC | 02:15 | |
| *** megm has quit IRC | 02:15 | |
| *** amitgandhinz has joined #openstack-infra | 02:15 | |
| *** chlong has quit IRC | 02:15 | |
| *** annegentle_ has quit IRC | 02:17 | |
| *** baoli has joined #openstack-infra | 02:17 | |
| *** sridhar_ram has joined #openstack-infra | 02:19 | |
| *** thorst_ has quit IRC | 02:19 | |
| jeblair | mestery: it is likely because neutron and neutronclient do not share a change queue | 02:20 |
| jeblair | mestery: that means that there is no job that the two projects have in common | 02:20 |
| jeblair | mestery: that is... surprising to me. | 02:20 |
| tonyb | mestery: It's in both check and gate now. | 02:21 |
| jeblair | mestery: (in other words, there is no job that tests both things together). | 02:21 |
| *** hamzy has joined #openstack-infra | 02:21 | |
| jeblair | mestery, tonyb: adding (or toggling) workflow causes it to be enqueued | 02:21 |
| *** ctlaugh has joined #openstack-infra | 02:23 | |
| *** baoli has quit IRC | 02:23 | |
| *** vsaienko has joined #openstack-infra | 02:23 | |
| *** daemontool has joined #openstack-infra | 02:23 | |
| *** sergmelikyan has joined #openstack-infra | 02:23 | |
| *** openstackgerrit has joined #openstack-infra | 02:24 | |
| sergmelikyan | folks, any ideas why change-request may be stuck in zuul POST pipeline for 13 hours? | 02:24 |
| sergmelikyan | http://status.openstack.org/zuul/ | 02:24 |
| *** baoli has joined #openstack-infra | 02:24 | |
| sergmelikyan | I have several change-requests stuck for openstack/yaql there :( | 02:24 |
| *** slogan621 has joined #openstack-infra | 02:25 | |
| sergmelikyan | on build-tarboll job | 02:25 |
| *** Jeffrey4l has quit IRC | 02:25 | |
| clarkb | sergmelikyan: the post queue is a lower priority than check and gate | 02:25 |
| *** tphummel has joined #openstack-infra | 02:25 | |
| clarkb | as long as there is check and gate demand post is deferred | 02:25 |
| sergmelikyan | yeah, but 13 hours... it's kinda a lot | 02:25 |
| clarkb | yes because there has been 13 hours of check and gate demand | 02:26 |
| sergmelikyan | got it, so we never start processing Post unless we have no more staff in queue for check and gate | 02:26 |
| mestery | thanks for the explanation jeblair | 02:26 |
| sergmelikyan | clarkb: thank you | 02:26 |
| * sergmelikyan is not really worried about building tarball, just was worried about the fact that jobs seemed to be stuck for 13 hours | 02:28 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add Neutron Failure Rate dashboard to Grafana https://review.openstack.org/278832 | 02:29 |
| clarkb | sergmelikyan: yes the way the scheduker works is gatr and check must have all queued jobs running before post will get nodes | 02:29 |
| *** chlong has joined #openstack-infra | 02:29 | |
| *** xyang1 has quit IRC | 02:29 | |
| *** baoli has quit IRC | 02:29 | |
| *** tpsilva has quit IRC | 02:36 | |
| *** roaet- has quit IRC | 02:41 | |
| *** Sukhdev_ has quit IRC | 02:41 | |
| *** roaet_ has joined #openstack-infra | 02:42 | |
| *** armax has quit IRC | 02:43 | |
| *** woodster_ has quit IRC | 02:46 | |
| *** Jeffrey4l has joined #openstack-infra | 02:49 | |
| *** slogan621 has quit IRC | 02:56 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Delete json reports when we delete yaml reports https://review.openstack.org/278841 | 02:56 |
| *** tphummel has quit IRC | 03:03 | |
| *** browne has joined #openstack-infra | 03:06 | |
| *** rguillebert has quit IRC | 03:07 | |
| *** annegentle_ has joined #openstack-infra | 03:08 | |
| *** daemontool has quit IRC | 03:08 | |
| *** links has joined #openstack-infra | 03:10 | |
| *** fawadkhaliq has joined #openstack-infra | 03:16 | |
| *** yamahata has quit IRC | 03:17 | |
| *** thorst_ has joined #openstack-infra | 03:18 | |
| *** lucas-afk has quit IRC | 03:19 | |
| *** lucasagomes has joined #openstack-infra | 03:20 | |
| *** chaitu has joined #openstack-infra | 03:21 | |
| *** annegentle_ has quit IRC | 03:21 | |
| *** sc68cal has quit IRC | 03:23 | |
| nibalizer | infra-root I am takign the hiera lock | 03:24 |
| *** thorst_ has quit IRC | 03:24 | |
| nibalizer | infra-root unlocked | 03:25 |
| *** annegentle_ has joined #openstack-infra | 03:25 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/ansible-puppet: Remove dead code https://review.openstack.org/278850 | 03:27 |
| clarkb | speaking of dead code I saw hpcloud things in oscc | 03:27 |
| prometheanfire | ianw: the diff/patch does not need the dep, but overall, the element does | 03:27 |
| openstackgerrit | Spencer Krum proposed openstack-infra/ansible-puppet: Remove dead code https://review.openstack.org/278850 | 03:27 |
| clarkb | mordred: ^ any idea if anyone could possibly be using that somehow or should we push a cleanup? | 03:27 |
| *** baoli has joined #openstack-infra | 03:27 | |
| *** esker has joined #openstack-infra | 03:32 | |
| lifeless | tonyb: I'll look again first thing tomorrow | 03:34 |
| *** shashank_hegde has joined #openstack-infra | 03:34 | |
| mordred | clarkb: aroo? | 03:36 |
| bkero | Nixon? | 03:37 |
| mordred | clarkb: oh - yeah - I have already pushed an hp cleanup to oscc | 03:37 |
| *** annegentle_ has quit IRC | 03:40 | |
| tonyb | lifeless: Thanks. There is also a thread on os-dev re pip install -U ... You shoudl chime in | 03:41 |
| tonyb | lifeless: Thanks for looking at the mock stuff. | 03:41 |
| *** asahlin has quit IRC | 03:43 | |
| *** asahlin has joined #openstack-infra | 03:44 | |
| *** annegentle_ has joined #openstack-infra | 03:44 | |
| *** harlowja_at_home has joined #openstack-infra | 03:47 | |
| openstackgerrit | Matthew Kassawara proposed openstack-infra/project-config: networking-ovn: Add install job for kuryr https://review.openstack.org/278101 | 03:49 |
| lifeless | tonyb: np | 03:49 |
| *** amotoki has joined #openstack-infra | 03:50 | |
| *** genediazjr has joined #openstack-infra | 03:50 | |
| *** piet has joined #openstack-infra | 03:51 | |
| *** esker has quit IRC | 03:53 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Don't run infracloud puppet twice https://review.openstack.org/278855 | 03:55 |
| *** slogan621 has joined #openstack-infra | 03:56 | |
| *** amotoki has quit IRC | 03:57 | |
| *** esker has joined #openstack-infra | 03:58 | |
| *** hdd has joined #openstack-infra | 04:01 | |
| *** esker has quit IRC | 04:03 | |
| *** Sukhdev has joined #openstack-infra | 04:06 | |
| *** arthus has joined #openstack-infra | 04:08 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Fixed #! linein javascript jobs https://review.openstack.org/278858 | 04:11 |
| *** tphummel has joined #openstack-infra | 04:12 | |
| *** boris-42 has quit IRC | 04:14 | |
| *** dahoo has quit IRC | 04:14 | |
| *** dimtruck is now known as zz_dimtruck | 04:17 | |
| *** tiswanso has quit IRC | 04:18 | |
| *** genediazjr has quit IRC | 04:18 | |
| *** Daisy has joined #openstack-infra | 04:18 | |
| *** sc68cal has joined #openstack-infra | 04:20 | |
| *** thorst_ has joined #openstack-infra | 04:22 | |
| *** woodster_ has joined #openstack-infra | 04:23 | |
| *** Daisy has quit IRC | 04:23 | |
| *** annegentle_ has quit IRC | 04:25 | |
| dougwig | hi infra, i've noticed jobs taking a lot longer than normal. on 268232 e.g., rally looks to be restarting multiple times. is that a known issue? | 04:27 |
| *** browne has left #openstack-infra | 04:28 | |
| clarkb | usually that means the job is crashing in such a way tat the slave is no longer useable, zuul will restart assuming neworking went away | 04:28 |
| *** claudiub has quit IRC | 04:29 | |
| *** amotoki has joined #openstack-infra | 04:29 | |
| *** thorst_ has quit IRC | 04:30 | |
| dougwig | i've been watching this one all day (not all the same jenkins run), and the last job always seems to be the neutron rally job, with those resets. i wonder if there's a problem affecting all neutron runs around that job. | 04:31 |
| prometheanfire | y/win 1 | 04:31 |
| *** jgriffith is now known as jgriffith_away | 04:31 | |
| clarkb | dougwig: have an example console log? | 04:32 |
| dougwig | clarkb: no, because i figured i was nuts thinking the progress bar was jumping backwards all day, until i just now saw it go from halfway to queued to starting. now i've got this one open to see if i can catch it: https://jenkins02.openstack.org/job/gate-rally-dsvm-neutron-neutron/2489/console | 04:33 |
| *** armax has joined #openstack-infra | 04:34 | |
| dougwig | clarkb: or i am just losing my mind. that's always highly possible. :) | 04:34 |
| clarkb | no its a thing, it may be related to using a sparse swapfile on clouds iwthout a swap device | 04:35 |
| clarkb | https://review.openstack.org/#/c/278557/ will in theory make those fail hard when they run out of fsik | 04:35 |
| clarkb | *out of disk | 04:35 |
| clarkb | rather than get into an undebuggable restart loop | 04:36 |
| dougwig | clarkb: is that why the check queue has been so backed up lately? | 04:36 |
| clarkb | I think queues are mostly backed up due to all of the recent breaking regressions | 04:37 |
| clarkb | nova + eventlet, oslo config? + tempest, the mock thing etc | 04:37 |
| clarkb | when that happens are we reset the gate frequently we take resources away from the other queues | 04:37 |
| dims | clarkb : oslo.config => cliff | 04:38 |
| clarkb | dims: thanks, I wasn't sure which thing did it and thought someone said oslo config | 04:38 |
| dims | clarkb : yep, we had a work around which went in last night, i just tracked down the cliff commit and dean has a revert up too | 04:39 |
| *** baoli has quit IRC | 04:39 | |
| clarkb | but some of it certainly is cloud growing pains | 04:39 |
| dougwig | clarkb: ahh, got it. thanks for the info. | 04:40 |
| *** salv-orl_ has joined #openstack-infra | 04:41 | |
| *** amotoki has quit IRC | 04:42 | |
| *** piet has quit IRC | 04:42 | |
| *** dims has quit IRC | 04:43 | |
| *** arthus has quit IRC | 04:43 | |
| *** salv-orlando has quit IRC | 04:44 | |
| clarkb | dougwig: https://jenkins04.openstack.org/job/gate-rally-dsvm-neutron-neutron/2527/console is another I think | 04:45 |
| *** annegentle_ has joined #openstack-infra | 04:46 | |
| dougwig | clarkb: that's a non-voting job. would it help sort things out if we disable it until the issue is sorted? | 04:46 |
| *** fawadkhaliq has quit IRC | 04:46 | |
| *** sergmelikyan has quit IRC | 04:48 | |
| clarkb | dougwig: if the fix is my devstack-gate change then not really, that will get us rolling again | 04:49 |
| clarkb | and it is almost merged | 04:49 |
| dougwig | clarkb: ok | 04:49 |
| openstackgerrit | Merged openstack/diskimage-builder: Correct rhel-common for rhel6 https://review.openstack.org/266016 | 04:49 |
| *** ajmiller has joined #openstack-infra | 04:49 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal: pre-install base packages https://review.openstack.org/278860 | 04:49 |
| *** shashank_hegde has quit IRC | 04:50 | |
| *** ajmiller has quit IRC | 04:50 | |
| *** isaacb has quit IRC | 04:50 | |
| *** annegentle_ has quit IRC | 04:51 | |
| *** armax has quit IRC | 04:51 | |
| dougwig | clarkb: yep, my change just hit that reset again as well. i'll be patient and check after your fix merges. thanks. | 04:51 |
| clarkb | if you look at the log jenkins is losing the connection to the slave | 04:51 |
| clarkb | it is possible that that is an independnet issue to what my d-g change addresses since tended to hve the slaves completely lock up | 04:52 |
| clarkb | but could be different symptoms for the same problem | 04:52 |
| *** arthus has joined #openstack-infra | 04:54 | |
| fungi | one thing we can do, if it persists, is hold a worker (in rackspace if possible so we also have easyish access to an out of band console) and then see if we can spot why the worker is getting completely trashed | 04:56 |
| slogan621 | quick questions: where would I find the list of those who can give +2? Is there an alias or similar that addresses all of them on IRC? I seem to be suffering from having not done any reviews or work for infra according to earlier comments here, and so I'm not getting much support it seems. but I have answered questions in ask.openstack.org in the past and have done code reviews, just not for infra. Does that count for anyt | 04:59 |
| slogan621 | Finally, if I must do work for infra, what kind of work is someone outside of infra capable of doing? | 04:59 |
| slogan621 | My experience is with neutron | 04:59 |
| dougwig | slogan621: in gerrit, top menu 'people' -> groups -> then type in the name of the group you're looking for. but unless something is broken, most of those folks are quite busy. what is it you need reviewed? | 05:00 |
| *** julim has joined #openstack-infra | 05:00 | |
| dougwig | slogan621: and contacting cores directly, if you don't already know them, is a pretty good way to NOT get a review done. :) | 05:00 |
| clarkb | right one of the big issues right now is we are all swamped keeping several boats afloat | 05:00 |
| slogan621 | clarkb: got it :-) | 05:01 |
| fungi | some of us are bailing water out the back while the others are trying to keep rowing ;) | 05:01 |
| *** tzn has joined #openstack-infra | 05:01 | |
| fungi | and some of us are falling asleep. midnight here | 05:01 |
| *** rbrndt has joined #openstack-infra | 05:01 | |
| clarkb | between trying to spin up new clouds to backfill hpcloud going away, nodepol builders that allow us to push images to those clouds, and feeding babies I have no time for my "pet" projects like upgrading the ELK stack | 05:01 |
| dougwig | indeed, infra cores are among the best at getting to reviews unsolicited. if they don't, you can almost bank on their being a fire going. | 05:01 |
| clarkb | and that doesn't even account for the fires :) | 05:02 |
| clarkb | slogan621: but that is why the suggestion is to help out, if you can review a few changes and either ACK or NACK them that can help alleviate the pressure on everyone else | 05:02 |
| clarkb | slogan621: and in return they will ahve mor etime to look at your change(s) | 05:02 |
| slogan621 | dougwig: It's a project create that initially submitted maybe 6 days back. I think we have it, and we have 1 +2 already. https://review.openstack.org/#/c/275873/ | 05:03 |
| slogan621 | so, what can I review, then. Point me at something and glad to look. | 05:03 |
| *** markvoelker has quit IRC | 05:03 | |
| clarkb | slogan621: that project-config repo typically has 10s of changes open for review. If you are familira with how to add projects or jenkins jobs you can review those | 05:04 |
| dougwig | slogan621: i'm not a core, but i'll take a peek. | 05:04 |
| clarkb | dougwig: more info, the two tests we had links for ran in different ovh regions. OVH is one of hte places where we use a sparse swap file so it could still be that issue | 05:04 |
| slogan621 | Like I said, new to neutron but did learn a small amount with this tiny change we made | 05:04 |
| slogan621 | er, new to infra | 05:04 |
| *** rbrndt has quit IRC | 05:04 | |
| clarkb | dougwig: it will be telling if we find a test fial in that way on rax because rax is using a proper swap device not a fiel | 05:04 |
| *** armax has joined #openstack-infra | 05:05 | |
| *** piet has joined #openstack-infra | 05:05 | |
| clarkb | dougwig: and ya when it isn't a few minutes before bed time I/we can work on holding a test node and poke at it more directly | 05:05 |
| slogan621 | let me see if I can handle any of it, but bad reviews are worse than no reviews, so I won't take on anything other than looking for the obvious things like alphabetization issues and so on | 05:05 |
| *** jamielennox is now known as jamielennox|away | 05:06 | |
| dougwig | clarkb: ok, sleep well, and let me know anything i can do to help. | 05:06 |
| *** tzn has quit IRC | 05:06 | |
| clarkb | the other theory I had was that the recent ubuntu trusty kernel may be causing us grief but that was backed by almost no data | 05:07 |
| *** sridhar_ram has quit IRC | 05:08 | |
| *** fawadkhaliq has joined #openstack-infra | 05:11 | |
| *** piet has quit IRC | 05:18 | |
| slogan621 | wow, there is a lot of reviews going on | 05:20 |
| *** jgriffith_away is now known as jgriffith | 05:21 | |
| *** bpokorny has joined #openstack-infra | 05:22 | |
| *** thorst_ has joined #openstack-infra | 05:27 | |
| *** hdd has quit IRC | 05:28 | |
| *** hdd has joined #openstack-infra | 05:31 | |
| *** jamielennox|away is now known as jamielennox | 05:31 | |
| *** hdd has quit IRC | 05:32 | |
| dougwig | clarkb: three fails in a row on ovh nodes. this time it got a rax node, so it'll be interesting to see what happens. | 05:34 |
| *** thorst_ has quit IRC | 05:34 | |
| *** infra-red has joined #openstack-infra | 05:35 | |
| *** yamahata has joined #openstack-infra | 05:36 | |
| *** infra-re_ has joined #openstack-infra | 05:36 | |
| *** piet has joined #openstack-infra | 05:36 | |
| *** armax has quit IRC | 05:37 | |
| *** infra-red has quit IRC | 05:39 | |
| *** hdd has joined #openstack-infra | 05:44 | |
| *** hdd has quit IRC | 05:44 | |
| chaitu | Hi, While trying to add slave agent in the jenkins node i am facing following error http://paste.openstack.org/show/486649/. can someone please help me out with it. | 05:46 |
| *** fedexo has joined #openstack-infra | 05:52 | |
| *** fedexo_ has joined #openstack-infra | 05:53 | |
| *** piet has quit IRC | 05:54 | |
| *** vgridnev has joined #openstack-infra | 05:58 | |
| openstackgerrit | Joseph Robinson proposed openstack-infra/irc-meetings: Add the new US meeting time to the User Guide Team yaml https://review.openstack.org/278866 | 05:59 |
| *** infra-re_ has quit IRC | 06:02 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance https://review.openstack.org/278816 | 06:02 |
| *** rossella_s has quit IRC | 06:03 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance https://review.openstack.org/278816 | 06:03 |
| *** rossella_s has joined #openstack-infra | 06:03 | |
| *** markvoelker has joined #openstack-infra | 06:04 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance https://review.openstack.org/278816 | 06:05 |
| *** nadya has joined #openstack-infra | 06:05 | |
| *** infra-red has joined #openstack-infra | 06:05 | |
| *** nadya is now known as Guest25727 | 06:05 | |
| *** jaosorior has joined #openstack-infra | 06:10 | |
| dougwig | clarkb: success on the rax node. hopefully that means swap is the root issue there as well. | 06:11 |
| *** shashank_hegde has joined #openstack-infra | 06:13 | |
| *** Guest25727 has quit IRC | 06:16 | |
| *** infra-red has quit IRC | 06:17 | |
| jklare | hey, is something wrong with the gerrit-zuul communication? there were no gate jobs started for this https://review.openstack.org/#/c/250854/ (and some other) patches after the workflow +1 | 06:18 |
| *** infra-red has joined #openstack-infra | 06:20 | |
| *** bpokorny has quit IRC | 06:22 | |
| *** cp16net has joined #openstack-infra | 06:23 | |
| cp16net | sounds like the apt mirrors went live and about that time causes some trouble for our trove gate jobs earlier in the day. | 06:24 |
| cp16net | looks like at least one of the mirrors i looked at does not have the ubuntu package python-software-properties | 06:24 |
| cp16net | looked into this mirror and didnt see it. http://mirror.bhs1.ovh.openstack.org/ubuntu/pool/main/p/ | 06:25 |
| *** infra-red has quit IRC | 06:29 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/elastic-recheck: Add query for missing mock in py34 tests bug 1544377 https://review.openstack.org/278871 | 06:29 |
| openstack | bug 1544377 in python-glanceclient "proposed change to test-requirements breaks py34" [Undecided,New] https://launchpad.net/bugs/1544377 | 06:29 |
| *** thorst_ has joined #openstack-infra | 06:32 | |
| *** markvoelker has quit IRC | 06:34 | |
| *** kdas_ has quit IRC | 06:36 | |
| *** woodster_ has quit IRC | 06:36 | |
| *** tphummel has quit IRC | 06:37 | |
| openstackgerrit | Merged openstack/requirements: Revert "Only install mock on Python < 3.3" https://review.openstack.org/278814 | 06:38 |
| *** thorst_ has quit IRC | 06:39 | |
| *** zeih_ has joined #openstack-infra | 06:41 | |
| *** harlowja_at_home has quit IRC | 06:43 | |
| *** vgridnev has quit IRC | 06:43 | |
| *** maishsk has quit IRC | 06:46 | |
| *** maishsk has joined #openstack-infra | 06:47 | |
| *** flepied has joined #openstack-infra | 06:55 | |
| *** vgridnev has joined #openstack-infra | 06:55 | |
| *** sc` has joined #openstack-infra | 06:56 | |
| *** jaosorior has quit IRC | 06:56 | |
| *** rvasilets_ has joined #openstack-infra | 06:58 | |
| *** zeih_ has quit IRC | 06:59 | |
| *** sc68cal has quit IRC | 07:03 | |
| *** flepied1 has joined #openstack-infra | 07:05 | |
| *** infra-red has joined #openstack-infra | 07:06 | |
| *** esikachev has joined #openstack-infra | 07:07 | |
| *** infra-re_ has joined #openstack-infra | 07:08 | |
| slogan621 | question: is the jobs: section in jenkins/jobs/projects.yaml for a project a requirement? | 07:08 |
| *** flepied has quit IRC | 07:09 | |
| slogan621 | if I don't want to run any voting jobs, is there some minimum entry or can it be completely removed? | 07:09 |
| *** vgridnev has quit IRC | 07:10 | |
| slogan621 | the docs suggest it "should" be there but not "must" be there | 07:10 |
| *** nadya has joined #openstack-infra | 07:13 | |
| *** infra-red has quit IRC | 07:13 | |
| *** nadya is now known as Guest13544 | 07:13 | |
| *** fedexo has quit IRC | 07:14 | |
| *** Sukhdev has quit IRC | 07:15 | |
| *** maishsk has quit IRC | 07:15 | |
| *** maishsk has joined #openstack-infra | 07:16 | |
| *** lmiccini|gone is now known as lmiccini | 07:16 | |
| *** vgridnev has joined #openstack-infra | 07:18 | |
| *** jaosorior has joined #openstack-infra | 07:20 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/278886 | 07:21 |
| *** kozhukalov has joined #openstack-infra | 07:23 | |
| *** abregman has joined #openstack-infra | 07:24 | |
| *** abregman|afk has quit IRC | 07:25 | |
| *** ianw has quit IRC | 07:30 | |
| *** ianw has joined #openstack-infra | 07:30 | |
| *** jaosorior has quit IRC | 07:32 | |
| *** jtomasek has joined #openstack-infra | 07:36 | |
| *** mrmartin has joined #openstack-infra | 07:36 | |
| *** scheuran has joined #openstack-infra | 07:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack-dev/hacking: Updated from global requirements https://review.openstack.org/278654 | 07:37 |
| *** thorst_ has joined #openstack-infra | 07:37 | |
| *** amotoki has joined #openstack-infra | 07:38 | |
| *** kozhukalov has quit IRC | 07:38 | |
| *** kozhukalov has joined #openstack-infra | 07:38 | |
| *** kozhukalov has quit IRC | 07:40 | |
| *** kozhukalov has joined #openstack-infra | 07:41 | |
| *** kozhukalov has quit IRC | 07:42 | |
| *** kozhukalov_ has joined #openstack-infra | 07:42 | |
| *** kozhukalov_ has quit IRC | 07:42 | |
| *** kozhukalov_ has joined #openstack-infra | 07:42 | |
| *** kozhukalov has joined #openstack-infra | 07:43 | |
| *** kozhukalov has quit IRC | 07:43 | |
| *** flepied1 has quit IRC | 07:44 | |
| *** kozhukalov has joined #openstack-infra | 07:44 | |
| *** thorst_ has quit IRC | 07:45 | |
| *** zeih_ has joined #openstack-infra | 07:51 | |
| *** electrofelix has joined #openstack-infra | 07:51 | |
| *** max_lobur has quit IRC | 07:54 | |
| yolanda | good morning | 08:00 |
| *** infra-red has joined #openstack-infra | 08:02 | |
| jklare | morning | 08:03 |
| *** abregman_ has joined #openstack-infra | 08:04 | |
| jklare | can somebody help me understand why this patch is not merging https://review.openstack.org/#/c/250854/ ? something wrong with the gates? | 08:04 |
| jklare | other patches from other projects seem to merge fine | 08:04 |
| *** infra-re_ has quit IRC | 08:05 | |
| *** arthus has quit IRC | 08:05 | |
| *** jaosorior has joined #openstack-infra | 08:05 | |
| jklare | but there are multiple cookbook patches (different projects) that should run gate jobs, but nothing happens | 08:06 |
| *** vgridnev has quit IRC | 08:06 | |
| *** achanda has quit IRC | 08:11 | |
| *** arthus has joined #openstack-infra | 08:12 | |
| *** hashar has joined #openstack-infra | 08:13 | |
| *** achanda has joined #openstack-infra | 08:14 | |
| *** flepied1 has joined #openstack-infra | 08:14 | |
| *** kozhukalov has quit IRC | 08:14 | |
| *** kozhukalov has joined #openstack-infra | 08:15 | |
| *** kozhukal` has joined #openstack-infra | 08:15 | |
| *** kozhukal` has quit IRC | 08:16 | |
| *** kozhukalov has quit IRC | 08:16 | |
| *** kozhukalov has joined #openstack-infra | 08:17 | |
| yolanda | jklare, can you try giving another +workflow? someone with +w powers has to send another one | 08:17 |
| jklare | sure | 08:17 |
| jklare | done and seems to work | 08:19 |
| jklare | ok, so i will do this for all patches that failed to trigger the gate jobs? | 08:19 |
| jklare | its no big deal, just asking if that is the right approach to the issue | 08:20 |
| yolanda | jklare, i don't know if there were some problems with zuul yesterday. But some times, seems that the event is not picked properly | 08:20 |
| *** kozhukalov_ has quit IRC | 08:20 | |
| yolanda | and the way to retrigger is to send another +w | 08:20 |
| jklare | yolanda ok, thanks | 08:20 |
| *** esikachev has quit IRC | 08:23 | |
| *** mikelk has joined #openstack-infra | 08:24 | |
| *** jlanoux has joined #openstack-infra | 08:25 | |
| AJaeger | jklare: your job has a depends-on on another project | 08:25 |
| jklare | AJaeger sure, but why is that an issue? | 08:26 |
| AJaeger | If your change gets +W first and these two changes do not share a testing queue, there's no notification that the first one merged and the second one can go in | 08:26 |
| jklare | AJaeger ahhhh, i see | 08:26 |
| *** shashank_hegde has quit IRC | 08:26 | |
| *** jed56 has joined #openstack-infra | 08:26 | |
| AJaeger | jklare: So, what I'm doing: Whenever I see a depends-on: Wait until the dependent job is in and only then approve. | 08:27 |
| jklare | AJaeger yeah, i think that would have been easier | 08:27 |
| AJaeger | Or if there's an approval, just trigger it again | 08:27 |
| jklare | AJaeger next time :D | 08:27 |
| *** hashar has quit IRC | 08:27 | |
| yolanda | oh , that's the other reason.. sorry, i missed the depends-on | 08:27 |
| *** kozhukalov has quit IRC | 08:28 | |
| AJaeger | yolanda: can I push these three on your review queue for later today, please? https://review.openstack.org/278627 https://review.openstack.org/278641 and https://review.openstack.org/278633 Thanks! | 08:29 |
| yolanda | sure, i will look in a while | 08:29 |
| *** nmagnezi has joined #openstack-infra | 08:32 | |
| *** esikachev has joined #openstack-infra | 08:33 | |
| *** kozhukalov has joined #openstack-infra | 08:33 | |
| *** abregman_ has quit IRC | 08:33 | |
| *** abregman has quit IRC | 08:33 | |
| *** rcernin has joined #openstack-infra | 08:35 | |
| *** tzn has joined #openstack-infra | 08:35 | |
| *** kozhukalov has quit IRC | 08:36 | |
| *** isaacb has joined #openstack-infra | 08:36 | |
| *** kozhukalov has joined #openstack-infra | 08:36 | |
| *** maishsk has quit IRC | 08:37 | |
| *** abregman has joined #openstack-infra | 08:38 | |
| *** kozhukalov has quit IRC | 08:39 | |
| *** kozhukalov has joined #openstack-infra | 08:39 | |
| *** maishsk has joined #openstack-infra | 08:39 | |
| *** shardy has joined #openstack-infra | 08:40 | |
| *** kozhukalov has quit IRC | 08:40 | |
| *** kozhukalov has joined #openstack-infra | 08:40 | |
| *** kozhukalov_ has joined #openstack-infra | 08:41 | |
| *** shardy has quit IRC | 08:41 | |
| *** yaume has joined #openstack-infra | 08:41 | |
| *** shardy has joined #openstack-infra | 08:42 | |
| *** tzn has quit IRC | 08:42 | |
| *** e0ne has joined #openstack-infra | 08:42 | |
| *** thorst_ has joined #openstack-infra | 08:42 | |
| *** mrmartin has quit IRC | 08:43 | |
| *** achanda has quit IRC | 08:44 | |
| *** claudiub has joined #openstack-infra | 08:44 | |
| *** amotoki has quit IRC | 08:45 | |
| *** matrohon has joined #openstack-infra | 08:46 | |
| *** openstackgerrit has quit IRC | 08:47 | |
| *** openstackgerrit_ has joined #openstack-infra | 08:47 | |
| *** openstackgerrit_ is now known as openstackgerrit | 08:48 | |
| *** alex_xu_ has joined #openstack-infra | 08:49 | |
| *** abregman has quit IRC | 08:49 | |
| *** thorst_ has quit IRC | 08:50 | |
| *** abregman has joined #openstack-infra | 08:50 | |
| openstackgerrit | Merged openstack-infra/project-config: Force the gate-irc-meetings-tox-ical postbuildscript to always run https://review.openstack.org/278790 | 08:50 |
| yolanda | AJaeger, would you say that https://review.openstack.org/#/c/275873/ fits under networking scope, and needs +1 from neutron liaisons? | 08:50 |
| *** dingyichen has quit IRC | 08:51 | |
| *** ildikov has quit IRC | 08:51 | |
| *** arthus has quit IRC | 08:51 | |
| openstackgerrit | Merged openstack-infra/project-config: Templatize experimental bindep jobs https://review.openstack.org/278784 | 08:51 |
| *** maishsk has quit IRC | 08:52 | |
| *** jtomasek has quit IRC | 08:52 | |
| *** alex_xu has quit IRC | 08:52 | |
| AJaeger | yolanda: no, it's independent. But looking at comments: Wait with your +A another day, there's still some discussion going on whether this is the right approach | 08:56 |
| yolanda | ok we'll wait for that | 08:57 |
| *** jtomasek has joined #openstack-infra | 08:57 | |
| *** fawadkhaliq has quit IRC | 08:59 | |
| *** jordanP has joined #openstack-infra | 09:00 | |
| *** derekh has joined #openstack-infra | 09:00 | |
| *** fhubik has joined #openstack-infra | 09:00 | |
| *** fhubik is now known as fhubik_brb | 09:00 | |
| *** sshnaidm has quit IRC | 09:00 | |
| *** BobBall_AWOL is now known as Bobball | 09:01 | |
| *** Bobball is now known as BobBall | 09:01 | |
| *** zeih_ has quit IRC | 09:01 | |
| *** fhubik_brb is now known as fhubik | 09:05 | |
| *** arthus has joined #openstack-infra | 09:05 | |
| *** mrmartin has joined #openstack-infra | 09:06 | |
| *** placko_ has joined #openstack-infra | 09:06 | |
| *** hashar has joined #openstack-infra | 09:06 | |
| *** ihrachys has joined #openstack-infra | 09:07 | |
| *** jordanP has quit IRC | 09:12 | |
| jhesketh | AJaeger: so I'm confused about the linters stuff sorry... I thought the issue was the pep8 target was poorly named since it might actually be running bashate (for example). So I thought the idea was to rename pep8 to linters effectively? | 09:12 |
| *** fhubik is now known as fhubik_brb | 09:13 | |
| *** ifarkas has joined #openstack-infra | 09:14 | |
| AJaeger | jhesketh: feel free to read backscroll from 14 hours ago where I discussed with jeblair and fungi | 09:16 |
| AJaeger | basically: Yes, that was the idea moving forward. | 09:16 |
| AJaeger | But I forgot about the PTI - and that we have run-pep8 which does some extra tests | 09:16 |
| hashar | iris they started with 'pep8' then switched to use flake8 but to ease transition kept 'pep8' as an entry point | 09:17 |
| hashar | which is really an alias for 'lint stuff' | 09:17 |
| AJaeger | Citing from last night: | 09:17 |
| AJaeger | 1) Updating PTI, 2) fixing linters job to do the same as pep8, 3) adding linters to python-jobs with lots of skips/non-voting, 4) updating official repos to have linters target that does pep8 job on master (not on stable), 5) removing pep8 and skips/non-voting | 09:17 |
| AJaeger | Plan B: 1) Revert changes for bandit, keystone 2) Remove linters job from python-jobs in jenkins . Leave ansible, system-config, project-config as is (using linters) | 09:17 |
| AJaeger | And for plan A, I don't have the time to commit driving this through. | 09:18 |
| * AJaeger would be willing to review but getting this is in properly was more than I can do | 09:18 | |
| *** mrmartin has quit IRC | 09:18 | |
| *** fhubik_brb is now known as fhubik | 09:18 | |
| jhesketh | AJaeger: right, okay, thanks | 09:19 |
| jhesketh | AJaeger: so from plan A.2), what do linters do differently as pep8 presently? | 09:19 |
| jhesketh | (I think this is where my confusion is) | 09:19 |
| AJaeger | jhesketh: So, yes, pep8 is a bad name. | 09:19 |
| AJaeger | jhesketh: Check run-pep8 ;) It adds sdist check as well, to check that tarballs are useable | 09:19 |
| openstackgerrit | Merged openstack-infra/project-config: Fixed #! linein javascript jobs https://review.openstack.org/278858 | 09:20 |
| AJaeger | jhesketh: that'S the easiest part to change ;) | 09:20 |
| AJaeger | jhesketh: If you want to help cleanup, feel free to review: https://review.openstack.org/278627 https://review.openstack.org/278641 and https://review.openstack.org/278633 Thanks! | 09:21 |
| AJaeger | jhesketh: step 4 is a lot of mechanical work- and step 3 and step 5 might be iterative.. | 09:22 |
| jhesketh | AJaeger: right, I agree it's a bit of work to get it entirely there | 09:23 |
| AJaeger | jhesketh: right now we have 380+ repos using pep8 - but not all of them are official projects, so we would only need to update the official ones since those are under PTI | 09:23 |
| jhesketh | we could try tackling it in a backwards compatible way... for example by having the CTI allow use of either linters or pep8 for a while | 09:23 |
| AJaeger | jhesketh: and then you need to figure out what to do with stable branches. | 09:24 |
| AJaeger | Do we want - like keystone had: pep8 in stable but linters in master? | 09:24 |
| AJaeger | jhesketh: yes, transition time would be needed - but if it's too long, we have a mess in zuul/layout.yaml | 09:24 |
| AJaeger | best would be to do it atomic: Prepare changes, mass merge ;) | 09:25 |
| *** vgridnev has joined #openstack-infra | 09:25 | |
| jhesketh | indeed and getting buy in would take time | 09:25 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone https://review.openstack.org/278482 | 09:26 |
| *** ildikov has joined #openstack-infra | 09:26 | |
| AJaeger | from the feedback so far, buy in will take time but it resonated well to rename pep8. So, that's the simplest part | 09:26 |
| openstackgerrit | Ondřej Nový proposed openstack/requirements: Bump PyECLib global-requirement to >= 1.2.0 https://review.openstack.org/278945 | 09:29 |
| *** eliqiao_ has quit IRC | 09:29 | |
| openstackgerrit | Merged openstack-infra/project-config: Add periodic job to autogenerate devstack plugins list https://review.openstack.org/269921 | 09:30 |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Add job gate-swift-tox-func-in-process-fast-post https://review.openstack.org/276823 | 09:30 |
| *** markvoelker has joined #openstack-infra | 09:31 | |
| *** ildikov has quit IRC | 09:33 | |
| *** sheeprine has quit IRC | 09:33 | |
| *** sheeprine has joined #openstack-infra | 09:35 | |
| *** eliqiao_ has joined #openstack-infra | 09:35 | |
| *** markvoelker has quit IRC | 09:36 | |
| *** scheuran has quit IRC | 09:37 | |
| *** gildub has joined #openstack-infra | 09:38 | |
| *** mrmartin has joined #openstack-infra | 09:39 | |
| openstackgerrit | Merged openstack-infra/project-config: Add requirements check for watcher https://review.openstack.org/278326 | 09:40 |
| *** sfinucan has joined #openstack-infra | 09:41 | |
| *** sshnaidm has joined #openstack-infra | 09:41 | |
| *** sshnaidm has quit IRC | 09:42 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove gate-nova-pylint https://review.openstack.org/278094 | 09:43 |
| *** HeOS has quit IRC | 09:44 | |
| *** ociuhandu has quit IRC | 09:44 | |
| *** HeOS has joined #openstack-infra | 09:44 | |
| *** claudiub has quit IRC | 09:45 | |
| *** gildub has quit IRC | 09:45 | |
| derekh | hey, has something changed in the behaviour of ZUUL and how it sets ZUUL_CHANGES | 09:45 |
| *** ildikov has joined #openstack-infra | 09:45 | |
| derekh | e.g. on this patch https://review.openstack.org/#/c/278556/ | 09:46 |
| *** sshnaidm has joined #openstack-infra | 09:46 | |
| derekh | there is a depends on for a patch in openstack/heat | 09:46 |
| derekh | but CI has ZUUL_CHANGES=openstack/tripleo-heat-templates:master:refs/changes/56/278556/1 | 09:46 |
| derekh | no mention of openstack/heat ? | 09:46 |
| derekh | this didn't used to be the case | 09:47 |
| shardy | Same problem with https://review.openstack.org/#/c/275605/ which has a heat Depends-On | 09:47 |
| *** thorst_ has joined #openstack-infra | 09:48 | |
| *** max_lobur has joined #openstack-infra | 09:49 | |
| *** andymaier has joined #openstack-infra | 09:51 | |
| *** ganso has joined #openstack-infra | 09:51 | |
| *** scheuran has joined #openstack-infra | 09:51 | |
| *** kaisers has joined #openstack-infra | 09:54 | |
| *** thorst_ has quit IRC | 09:55 | |
| *** andymaier has quit IRC | 09:56 | |
| derekh | shardy: I've rechecked this patch, it work as expected yesterday (as in ZUUL_CHANGES was set) , lets see what happens https://review.openstack.org/#/c/260466/8 | 09:57 |
| *** tzn has joined #openstack-infra | 09:57 | |
| *** scheuran has quit IRC | 10:00 | |
| *** cgalan has joined #openstack-infra | 10:01 | |
| *** gildub has joined #openstack-infra | 10:02 | |
| *** scheuran has joined #openstack-infra | 10:02 | |
| *** rossella_s has quit IRC | 10:03 | |
| *** reed_ has joined #openstack-infra | 10:03 | |
| *** vgridnev has quit IRC | 10:03 | |
| *** rossella_s has joined #openstack-infra | 10:03 | |
| *** vgridnev has joined #openstack-infra | 10:08 | |
| *** vgridnev has quit IRC | 10:09 | |
| *** yamamoto has quit IRC | 10:10 | |
| *** yamamoto has joined #openstack-infra | 10:10 | |
| *** vgridnev has joined #openstack-infra | 10:11 | |
| *** vgridnev has quit IRC | 10:12 | |
| *** vgridnev has joined #openstack-infra | 10:12 | |
| *** vgridnev has quit IRC | 10:13 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:14 | |
| *** vgridnev has joined #openstack-infra | 10:15 | |
| *** dtantsur|afk is now known as dtantsur | 10:20 | |
| *** mase_x200 has joined #openstack-infra | 10:20 | |
| *** gildub has quit IRC | 10:21 | |
| *** gildub has joined #openstack-infra | 10:22 | |
| *** esikachev has quit IRC | 10:23 | |
| *** fedexo_ has quit IRC | 10:24 | |
| *** matrohon has quit IRC | 10:26 | |
| *** Daisy has joined #openstack-infra | 10:26 | |
| *** esikachev has joined #openstack-infra | 10:26 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Allow zuul-release to push merge commits https://review.openstack.org/278967 | 10:29 |
| *** Daisy has quit IRC | 10:31 | |
| *** esikachev has quit IRC | 10:31 | |
| *** vgridnev has quit IRC | 10:34 | |
| *** esikachev has joined #openstack-infra | 10:36 | |
| *** tpsilva has joined #openstack-infra | 10:36 | |
| *** yaume has quit IRC | 10:38 | |
| *** yamamoto has quit IRC | 10:38 | |
| *** salv-orlando has joined #openstack-infra | 10:42 | |
| *** vgridnev has joined #openstack-infra | 10:42 | |
| *** dizquierdo has joined #openstack-infra | 10:44 | |
| *** achanda has joined #openstack-infra | 10:44 | |
| *** salv-orl_ has quit IRC | 10:45 | |
| *** vgridnev has quit IRC | 10:45 | |
| *** vgridnev has joined #openstack-infra | 10:46 | |
| *** chaitu has quit IRC | 10:47 | |
| *** dims has joined #openstack-infra | 10:48 | |
| *** yamamoto has joined #openstack-infra | 10:49 | |
| *** yamamoto has quit IRC | 10:49 | |
| *** ldnunes has joined #openstack-infra | 10:49 | |
| *** scheuran has quit IRC | 10:50 | |
| *** rfolco has joined #openstack-infra | 10:51 | |
| *** achanda has quit IRC | 10:51 | |
| *** thorst_ has joined #openstack-infra | 10:52 | |
| kzaitsev_mb | Is there a place, where we can find info about 'meta' tags in git tags? | 10:53 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: create openstack/service-registry https://review.openstack.org/278612 | 10:56 |
| *** [1]Thelo has joined #openstack-infra | 10:56 | |
| *** cgalan has quit IRC | 10:57 | |
| *** cgalan has joined #openstack-infra | 10:57 | |
| *** arthus has quit IRC | 10:58 | |
| *** Thelo has quit IRC | 10:58 | |
| *** [1]Thelo is now known as Thelo | 10:58 | |
| *** thorst_ has quit IRC | 11:00 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove large-ops testing https://review.openstack.org/278978 | 11:01 |
| therve | Hum I have an issue with osprofiler in stable/kilo for Heat | 11:01 |
| therve | It seems to depend on a recent oslo.utils version | 11:01 |
| therve | Should I cap osprofiler in kilo? | 11:02 |
| *** arthus has joined #openstack-infra | 11:02 | |
| *** ildikov has quit IRC | 11:03 | |
| sdague | shardy / derekh is HEAT in the project list for those jobs? I know the PROJECTS list was being pruned for the default case to speed up build times | 11:03 |
| *** aysyd has joined #openstack-infra | 11:03 | |
| *** scheuran has joined #openstack-infra | 11:05 | |
| derekh | sdague: not sure, I can check, but would it matter ? heat is getting mentioned nowhere in the jenkins job https://jenkins04.openstack.org/job/gate-tripleo-ci-f22-nonha/635/injectedEnvVars/ | 11:05 |
| shardy | we don't test much in tempest anymore since we moved everything in-tree for our functional tests | 11:06 |
| sdague | derekh: which job is the issue? | 11:06 |
| derekh | sdague: I see the problem (I think), shardy thee patch has merged already | 11:06 |
| *** Guest13544 has quit IRC | 11:07 | |
| shardy | derekh: aha - I was trying to work around the old current-tripleo pin | 11:07 |
| *** kushal has joined #openstack-infra | 11:07 | |
| shardy | I assumed we could override it on a per-package basis via Depends-On | 11:07 |
| derekh | shardy: looks like zuul is smart enough to no let you do that | 11:08 |
| derekh | sdague: thanks for looking | 11:08 |
| *** esikachev has quit IRC | 11:08 | |
| sdague | sure, np | 11:08 |
| derekh | shardy: maybe a patch to add heat to the package list, or add it to zuul changes | 11:08 |
| sdague | you guys sorted now? | 11:08 |
| shardy | sdague: thanks, and sorry I think I misinterpreted your question ;) | 11:08 |
| sdague | shardy: ok, no prob | 11:09 |
| derekh | sdague: yup, we're sorted | 11:09 |
| sdague | cool :) | 11:09 |
| *** claudiub has joined #openstack-infra | 11:12 | |
| openstackgerrit | Alexander Tivelkov proposed openstack/requirements: Bump yaql version to 1.1.0 https://review.openstack.org/278983 | 11:12 |
| *** smanjara has joined #openstack-infra | 11:13 | |
| openstackgerrit | Thomas Herve proposed openstack/requirements: Cap osprofiler https://review.openstack.org/278984 | 11:13 |
| *** reed_ has quit IRC | 11:15 | |
| *** fawadkhaliq has joined #openstack-infra | 11:16 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: accommodate for double tunneling https://review.openstack.org/273157 | 11:17 |
| *** dizquierdo has quit IRC | 11:17 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Make header 'quick navigate' and 'search' clearer https://review.openstack.org/278987 | 11:18 |
| *** beisner has quit IRC | 11:19 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/infra-specs: Add spec to provide an Infra Root CA https://review.openstack.org/278989 | 11:20 |
| *** beisner has joined #openstack-infra | 11:20 | |
| *** flepied1 has quit IRC | 11:20 | |
| *** nadya has joined #openstack-infra | 11:21 | |
| *** daemontool has joined #openstack-infra | 11:21 | |
| *** nadya is now known as Guest74796 | 11:22 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin https://review.openstack.org/265311 | 11:23 |
| *** cgalan has quit IRC | 11:24 | |
| *** cgalan has joined #openstack-infra | 11:25 | |
| *** shardy has quit IRC | 11:26 | |
| *** shardy has joined #openstack-infra | 11:28 | |
| *** mase_x200 has quit IRC | 11:29 | |
| *** dizquierdo has joined #openstack-infra | 11:32 | |
| *** markvoelker has joined #openstack-infra | 11:32 | |
| *** fhubik is now known as fhubik_brb | 11:33 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: accommodate for double tunneling https://review.openstack.org/273157 | 11:34 |
| openstackgerrit | Merged openstack/os-testr: Updated from global requirements https://review.openstack.org/278703 | 11:34 |
| *** fhubik_brb is now known as fhubik | 11:34 | |
| *** markvoelker has quit IRC | 11:37 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: accommodate for double tunneling https://review.openstack.org/273157 | 11:37 |
| *** esikachev has joined #openstack-infra | 11:37 | |
| openstackgerrit | Alexander Charykov proposed openstack-infra/gerritlib: Add limit for bulk_query request https://review.openstack.org/277773 | 11:38 |
| *** amotoki has joined #openstack-infra | 11:41 | |
| *** fhubik is now known as fhubik_brb | 11:41 | |
| *** shardy has quit IRC | 11:46 | |
| *** achanda has joined #openstack-infra | 11:48 | |
| *** yamamoto has joined #openstack-infra | 11:50 | |
| *** Nikolay_St has joined #openstack-infra | 11:50 | |
| Nikolay_St | hi all | 11:50 |
| Nikolay_St | we have a strange update from OpenstackBot in python-muranoclient: https://review.openstack.org/#/c/278731/1/test-requirements.txt | 11:51 |
| Nikolay_St | is it expected that we decrese python version to <3.3? | 11:51 |
| *** achanda has quit IRC | 11:52 | |
| sdague | Nikolay_St: see mailiing list, that was a mistake | 11:52 |
| *** dprince has joined #openstack-infra | 11:55 | |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: remove large-ops testing https://review.openstack.org/278978 | 11:55 |
| *** vgridnev has quit IRC | 11:56 | |
| *** vgridnev has joined #openstack-infra | 11:56 | |
| *** thorst_ has joined #openstack-infra | 11:58 | |
| *** yamamoto has quit IRC | 11:58 | |
| *** Guest74796 has quit IRC | 11:59 | |
| *** amotoki has quit IRC | 12:00 | |
| Nikolay_St | sdague: ok, thx | 12:02 |
| *** thorst_ has quit IRC | 12:02 | |
| *** amotoki has joined #openstack-infra | 12:03 | |
| *** pblaho has quit IRC | 12:06 | |
| *** kzaitsev_mb has quit IRC | 12:06 | |
| Nikolay_St | sdague: what is the title of the letter in ML? | 12:08 |
| *** kushal has quit IRC | 12:10 | |
| *** rguillebert has joined #openstack-infra | 12:11 | |
| openstackgerrit | Martin Hickey proposed openstack/requirements: Update Oslo VersionedObjects to version 1.5.0 https://review.openstack.org/279003 | 12:11 |
| *** ig0r_ has joined #openstack-infra | 12:12 | |
| *** gildub has quit IRC | 12:14 | |
| *** pradk_ has quit IRC | 12:24 | |
| *** |-paul-| has joined #openstack-infra | 12:25 | |
| *** pc_m has quit IRC | 12:26 | |
| *** cgalan has quit IRC | 12:26 | |
| *** nadya has joined #openstack-infra | 12:27 | |
| *** nadya is now known as Guest25526 | 12:27 | |
| *** Guest25526 has quit IRC | 12:27 | |
| *** nmagnezi has quit IRC | 12:28 | |
| openstackgerrit | Martin Hickey proposed openstack/requirements: Update Oslo VersionedObjects to version 1.5.0 https://review.openstack.org/279003 | 12:28 |
| *** weshay has joined #openstack-infra | 12:28 | |
| *** _amrith_ is now known as amrith | 12:28 | |
| *** doug-fish has joined #openstack-infra | 12:29 | |
| *** pc_m has joined #openstack-infra | 12:30 | |
| *** rvasilets_ has left #openstack-infra | 12:32 | |
| therve | My requirement patch fails with "You must put some 'source' URIs in your sources.list" | 12:32 |
| therve | Is that something intermittent? | 12:32 |
| *** smanjara has quit IRC | 12:33 | |
| *** markvoelker has joined #openstack-infra | 12:33 | |
| *** aspiers_ is now known as aspiers | 12:35 | |
| *** mase_x200 has joined #openstack-infra | 12:35 | |
| *** flepied1 has joined #openstack-infra | 12:36 | |
| *** markvoelker has quit IRC | 12:37 | |
| *** pradk_ has joined #openstack-infra | 12:38 | |
| *** thorst_ has joined #openstack-infra | 12:38 | |
| *** daemontool has quit IRC | 12:39 | |
| *** fhubik_brb is now known as fhubik | 12:39 | |
| *** thorst_ has quit IRC | 12:39 | |
| *** daemontool has joined #openstack-infra | 12:39 | |
| *** dims has quit IRC | 12:40 | |
| *** nmagnezi has joined #openstack-infra | 12:40 | |
| *** thorst_ has joined #openstack-infra | 12:41 | |
| *** amotoki has quit IRC | 12:41 | |
| *** arthus has quit IRC | 12:42 | |
| *** Nikolay_St has quit IRC | 12:42 | |
| *** flepied1 is now known as flepied | 12:44 | |
| *** baoli has joined #openstack-infra | 12:47 | |
| *** arthus has joined #openstack-infra | 12:47 | |
| *** placko_ has quit IRC | 12:47 | |
| *** baoli_ has joined #openstack-infra | 12:48 | |
| *** pblaho has joined #openstack-infra | 12:48 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/diskimage-builder: Do not remove python-dev from ironic-agent image https://review.openstack.org/279031 | 12:48 |
| *** daemontool_ has joined #openstack-infra | 12:49 | |
| *** daemontool__ has joined #openstack-infra | 12:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 12:49 | |
| *** daemontool has quit IRC | 12:51 | |
| *** baoli has quit IRC | 12:51 | |
| *** |-paul-| has quit IRC | 12:51 | |
| *** daemontool_ has quit IRC | 12:54 | |
| *** zeih_ has joined #openstack-infra | 12:55 | |
| *** jaypipes has joined #openstack-infra | 12:56 | |
| *** nadya has joined #openstack-infra | 12:56 | |
| *** nadya is now known as Guest65333 | 12:57 | |
| *** rlandy has joined #openstack-infra | 13:01 | |
| *** annegentle_ has joined #openstack-infra | 13:01 | |
| *** mase_x200 has quit IRC | 13:02 | |
| *** jcoufal has joined #openstack-infra | 13:06 | |
| *** annegentle_ has quit IRC | 13:08 | |
| *** amotoki has joined #openstack-infra | 13:09 | |
| *** zeih_ has quit IRC | 13:09 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/devstack-gate: WIP: neutron: try hugely reduced MTU on br-ex https://review.openstack.org/273157 | 13:10 |
| *** andreykurilin__ has joined #openstack-infra | 13:10 | |
| *** arthus has quit IRC | 13:10 | |
| *** jsavak has joined #openstack-infra | 13:12 | |
| *** shardy has joined #openstack-infra | 13:14 | |
| *** esp has joined #openstack-infra | 13:17 | |
| *** eharney has joined #openstack-infra | 13:18 | |
| *** tcammann has joined #openstack-infra | 13:18 | |
| *** NobodyCa1 has joined #openstack-infra | 13:18 | |
| *** esker has joined #openstack-infra | 13:20 | |
| *** dims has joined #openstack-infra | 13:20 | |
| *** jtomasek_ has joined #openstack-infra | 13:20 | |
| *** Guest57497 is now known as zeus | 13:22 | |
| *** zeus has joined #openstack-infra | 13:22 | |
| *** arthus has joined #openstack-infra | 13:22 | |
| *** daemontool__ has quit IRC | 13:23 | |
| *** markvoelker has joined #openstack-infra | 13:23 | |
| *** esp has quit IRC | 13:23 | |
| *** kushal has joined #openstack-infra | 13:23 | |
| *** tcammann has quit IRC | 13:23 | |
| *** kzaitsev_mb has quit IRC | 13:24 | |
| *** Daisy has joined #openstack-infra | 13:24 | |
| *** xyang1 has joined #openstack-infra | 13:24 | |
| *** NobodyCa1 has quit IRC | 13:24 | |
| *** moravec has quit IRC | 13:28 | |
| *** asahlin has quit IRC | 13:29 | |
| *** asahlin has joined #openstack-infra | 13:30 | |
| *** dpaterson has joined #openstack-infra | 13:30 | |
| *** woodster_ has joined #openstack-infra | 13:33 | |
| *** annegentle_ has joined #openstack-infra | 13:34 | |
| *** daemontool__ has joined #openstack-infra | 13:35 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Use release version of Ansible for testing https://review.openstack.org/279057 | 13:36 |
| *** jtomasek_ has quit IRC | 13:37 | |
| *** kgiusti has joined #openstack-infra | 13:37 | |
| *** burgerk has joined #openstack-infra | 13:37 | |
| *** edmondsw has joined #openstack-infra | 13:38 | |
| *** rubasov has quit IRC | 13:42 | |
| *** daemontool_ has joined #openstack-infra | 13:42 | |
| *** amitgandhinz has quit IRC | 13:45 | |
| *** jpeeler has joined #openstack-infra | 13:46 | |
| *** daemontool__ has quit IRC | 13:46 | |
| *** pradk_ is now known as pradk | 13:46 | |
| *** links has quit IRC | 13:48 | |
| *** amrith is now known as _amrith_ | 13:49 | |
| Shrews | mordred: hopefully that ^^^ unbreaks us, yet again | 13:50 |
| *** cgalan has joined #openstack-infra | 13:50 | |
| mordred | therve: link? | 13:50 |
| *** achanda has joined #openstack-infra | 13:50 | |
| *** rhallisey has joined #openstack-infra | 13:51 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Add fuel-virtualbox project https://review.openstack.org/279074 | 13:51 |
| *** cbader has quit IRC | 13:52 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone https://review.openstack.org/278482 | 13:52 |
| *** cgalan has quit IRC | 13:53 | |
| openstackgerrit | Vladimir Kozhukalov proposed openstack-infra/project-config: Add fuel-virtualbox project https://review.openstack.org/279074 | 13:53 |
| *** dkranz has joined #openstack-infra | 13:55 | |
| *** achanda has quit IRC | 13:55 | |
| *** daemontool_ is now known as daemontool | 13:56 | |
| *** rhallisey has quit IRC | 13:56 | |
| *** kzaitsev_mb has joined #openstack-infra | 13:57 | |
| *** arthus has quit IRC | 13:57 | |
| *** fhubik is now known as fhubik_brb | 13:57 | |
| *** ddieterly has joined #openstack-infra | 13:59 | |
| *** edmondsw has quit IRC | 13:59 | |
| *** ildikov has joined #openstack-infra | 14:02 | |
| *** rossella_s has quit IRC | 14:03 | |
| *** annegentle_ has quit IRC | 14:03 | |
| *** rossella_s has joined #openstack-infra | 14:03 | |
| *** jaosorior has quit IRC | 14:05 | |
| *** julim has quit IRC | 14:05 | |
| *** notnownikki has joined #openstack-infra | 14:05 | |
| *** jaosorior has joined #openstack-infra | 14:05 | |
| *** sheeprine_ has joined #openstack-infra | 14:07 | |
| therve | mordred, https://review.openstack.org/#/c/278984/ | 14:07 |
| *** esker has quit IRC | 14:08 | |
| *** sheeprine has quit IRC | 14:08 | |
| therve | http://logs.openstack.org/84/278984/1/check/gate-requirements-integration-dsvm/5d79770/ more specifically | 14:09 |
| *** sheeprine_ is now known as sheeprine | 14:09 | |
| *** sheeprine has joined #openstack-infra | 14:09 | |
| *** arthus has joined #openstack-infra | 14:10 | |
| *** Daisy has quit IRC | 14:11 | |
| *** zz_dimtruck is now known as dimtruck | 14:11 | |
| *** smanjara has joined #openstack-infra | 14:12 | |
| *** amitgandhinz has joined #openstack-infra | 14:15 | |
| *** krotscheck_dcm is now known as krotscheck | 14:15 | |
| *** mrmartin has quit IRC | 14:16 | |
| *** mriedem has joined #openstack-infra | 14:17 | |
| sdague | dstufft: random pypi wishlist | 14:17 |
| sdague | have a way to let CI systems acknowledge themselves to it | 14:17 |
| *** rhallisey has joined #openstack-infra | 14:18 | |
| sdague | to be able to understand downloads by people and by ci systems | 14:18 |
| *** _amrith_ is now known as amrith | 14:18 | |
| *** esker has joined #openstack-infra | 14:18 | |
| *** rhallisey has quit IRC | 14:22 | |
| *** Daisy has joined #openstack-infra | 14:23 | |
| *** esker has quit IRC | 14:23 | |
| *** ajmiller has joined #openstack-infra | 14:23 | |
| *** tiswanso has joined #openstack-infra | 14:24 | |
| mordred | therve: thanks - looking | 14:24 |
| mordred | clarkb: https://review.openstack.org/278797 updated - have tested locally that it works with rackspace | 14:25 |
| mordred | clarkb: at least, I have tested that it constructs a swift client that can connect to rackspace with the new parameter choices | 14:25 |
| mordred | clarkb: I have not tested if it helps the token re-get problem | 14:25 |
| *** abregman is now known as abregman|mtg | 14:25 | |
| *** Daisy has quit IRC | 14:26 | |
| *** rhallisey has joined #openstack-infra | 14:26 | |
| *** thiagop has joined #openstack-infra | 14:28 | |
| *** links has joined #openstack-infra | 14:28 | |
| *** ajmiller has quit IRC | 14:30 | |
| *** rhallisey has quit IRC | 14:30 | |
| *** rhallisey has joined #openstack-infra | 14:30 | |
| openstackgerrit | Monty Taylor proposed openstack/requirements: Stop using build-dep for numpy/pyyaml depends https://review.openstack.org/279099 | 14:33 |
| mordred | therve: that shold fix it ^^ | 14:33 |
| *** annegentle_ has joined #openstack-infra | 14:33 | |
| therve | mordred, Awesome, thanks | 14:34 |
| *** Nikolay_St has joined #openstack-infra | 14:34 | |
| *** annegentle_ has quit IRC | 14:34 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Import ansible-role-nginx https://review.openstack.org/279101 | 14:34 |
| *** rhallisey has quit IRC | 14:35 | |
| *** scheuran has quit IRC | 14:35 | |
| yolanda | hi fungi, about the spec.. what we've done now is to create the self signed cert manually. We ended with a CA in west and another in east, because i didn't have the details, config files, etc... to recreate certs on east. My initial idea is to generate a OpenStack Infra CA, self-signed, in my computer. Then create a cert with alt names for east and west. But that's manual, config details are missing, and cannot be reproduced, as they are on my l | 14:35 |
| yolanda | aptop | 14:35 |
| *** rhallisey has joined #openstack-infra | 14:36 | |
| yolanda | so when we need to do it again, how will we reproduce? create a ca again on laptop, config file, generate more certs? | 14:36 |
| asselin__ | why do we install pip packages in puppet instead of using a project's requirements.txt? | 14:36 |
| openstackgerrit | Bob Ball proposed openstack-infra/puppet-openstackci: Don't require a simple log_server https://review.openstack.org/279102 | 14:36 |
| *** mrmartin has joined #openstack-infra | 14:36 | |
| yolanda | fungi, why don't we have these config files, ca, etc... on a shared host like puppetmaster, so we can reuse when we need? not generally for infra, but we may need for infra cloud now, or for other future uses | 14:37 |
| *** sabeen has joined #openstack-infra | 14:37 | |
| *** moravec has joined #openstack-infra | 14:38 | |
| pabelanger | asselin__: install a new version not hosted on pypi? | 14:38 |
| *** akshai has joined #openstack-infra | 14:38 | |
| *** mrmartin has quit IRC | 14:38 | |
| asselin__ | pabelanger, you cannot specify a repo in requirements.txt? | 14:39 |
| *** nmagnezi has quit IRC | 14:39 | |
| *** ociuhandu has joined #openstack-infra | 14:39 | |
| *** annegentle_ has joined #openstack-infra | 14:39 | |
| rcarrillocruz | asselin__: you mean by repo a git repo in requirements? | 14:39 |
| pabelanger | asselin__: right, maybe we 'infra' just want to dogfood from master over the release on pypi | 14:40 |
| asselin__ | rcarrillocruz, yes | 14:40 |
| rcarrillocruz | you can, it's called 'editable packages' | 14:40 |
| pabelanger | asselin__: I do that with diskimage-builder. I install it from git, even though nodepool has it in requirements | 14:40 |
| rcarrillocruz | you can for example install from let's say github/lolrepo , commit foo | 14:40 |
| rcarrillocruz | pabelanger: i'm intrigued by letsencrypt, looks pretty straightforward and easy to automate | 14:41 |
| asselin__ | pabelanger, so how does puppet manage that? You have to put all of nodepools requirmeents in puppet? | 14:41 |
| *** sabeen3 has joined #openstack-infra | 14:41 | |
| rcarrillocruz | not sure what infra folks think about relying on that service tho | 14:41 |
| *** rhallisey has quit IRC | 14:41 | |
| pabelanger | rcarrillocruz: Ya, we've talked about it before. I think somebody just needs to do the work to try it out | 14:41 |
| *** sabeen has quit IRC | 14:41 | |
| pabelanger | asselin__: No, I only care about diskimage-builder and shade. I'm mostly dogfooding from master on them, while using nodepool from pip. | 14:42 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/278886 | 14:43 |
| yolanda | letsencrypt was raised on some meeting, i think cody-somerville proposed that, but this was discarded | 14:43 |
| *** alivigni has joined #openstack-infra | 14:44 | |
| pabelanger | yolanda: happen to have an link do that discussion? | 14:44 |
| yolanda | i can check | 14:45 |
| asselin__ | pabelanger, https://review.openstack.org/#/c/275535/ is raising the issue. How do we manage what to get from pip and what to get from repos? | 14:45 |
| fungi | yolanda: we can't just use self-signed certs and put those certs/keys in hiera? | 14:45 |
| yolanda | i think it was on prev 2 meetings | 14:45 |
| *** nmagnezi has joined #openstack-infra | 14:45 | |
| yolanda | fungi yes, but the thing is, what's the way to generate them? manually from our laptops? | 14:45 |
| yolanda | i think it will be better to have something in infra on place | 14:45 |
| yolanda | so we can share same CA, same key, config | 14:46 |
| fungi | the "let's use self-signed certs" decision was really meant to imply "let's not rely on a certificate authority for this" and not "let's run our own certificate authority for this!" | 14:46 |
| yolanda | ok if you want to change the intention of the spec | 14:46 |
| fungi | yolanda: usually i just generate them on the puppetmaster | 14:46 |
| fungi | and then copy the results into hiera | 14:46 |
| yolanda | so yes, generate that on pupetmaster | 14:46 |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34 check https://review.openstack.org/279108 | 14:46 |
| fungi | and let puppet put them onto the machines that need them | 14:46 |
| yolanda | but have the config on place, automated | 14:46 |
| *** annegentle_ has quit IRC | 14:46 | |
| yolanda | to generate a CA you need to have openssl.cnf, same for intermediate | 14:47 |
| *** cbader has joined #openstack-infra | 14:47 | |
| yolanda | a CA key, etc. | 14:47 |
| *** fhubik_brb is now known as fhubik | 14:47 | |
| *** nwkarsten has joined #openstack-infra | 14:47 | |
| BobBall | asselin__: Is there a reason why we have a project_config_repo but no project_config_revision at the top level? I see that project_config_revision is defaulted to 'master' in zuul.pp but the other classes that use project_config_repo don't seem to have a revision? | 14:47 |
| yolanda | so i propose at least have this setup on puppetmaster | 14:47 |
| fungi | yolanda: why do you need an intermediate/chain cert in this case? there's nothing to chain | 14:47 |
| yolanda | the same thing we do unofficially, have it on our repos, officially seutp | 14:47 |
| fungi | a self-signed cert is exactly that. the private key is used to sign its own public component | 14:48 |
| asselin__ | BobBall, I think that's something that started being added later. there's also the 'base' parameter | 14:48 |
| fungi | there's nothing to trust beyond the key associated with the cert | 14:48 |
| yolanda | it was setup like that on prev versions with crinkle/nibalizer so i generated and intermediate as well on east. And our infra cloud uses chain files | 14:48 |
| pabelanger | asselin__: For me, it is at the discretion of the user using puppet IMO. If they want to rev up dependencies at their own pace, why stop them. There is nothing stopping a user from doing pip install diskimage-builder even before running puppet. By the time puppet installs nodepool, it won't need to worry about diskimage-builder download, because it already exists on the system. Assuming both upper | 14:48 |
| pabelanger | and lower requirements are met | 14:48 |
| fungi | yolanda: that's pure cargo-culting | 14:49 |
| fungi | on everyone's part apparently | 14:49 |
| rcarrillocruz | hmm, so from now on, we shouldn't really put intermediate chain files | 14:49 |
| rcarrillocruz | ? | 14:49 |
| pabelanger | asselin__: especially because we are dogfooding a lot of projects from master, I really think both shade and diskimage-builder should be moved out of the nodepool class into their own. Then allow the user to choose how to install them | 14:49 |
| fungi | rcarrillocruz: you should use them when necessary | 14:49 |
| yolanda | fungi, that's how puppet is setup on infra cloud right now | 14:50 |
| rcarrillocruz | cos spinning up openstackci in test environments i found them just a pita | 14:50 |
| fungi | rcarrillocruz: not all certificate authorities need a chain cert, and self-signed certs _never_ need a chain cert because there's nothing to chain back to | 14:50 |
| yolanda | anyway, even if we just use our own CA, it should be nice to have a shared place for it | 14:50 |
| yolanda | so every self-signed cert shares the same | 14:50 |
| fungi | rcarrillocruz: some certificate authorities need chain certs because their trust is delegated from some other certificate authority | 14:50 |
| openstackgerrit | Merged openstack/requirements: bump openstackclient in u-c to 2.1.0 https://review.openstack.org/276373 | 14:50 |
| openstackgerrit | Merged openstack/requirements: Bump futurist to 0.11.0 https://review.openstack.org/278272 | 14:50 |
| asselin__ | pabelanger, sound like you agree with my suggestion in that patch then right? | 14:50 |
| *** zeih_ has joined #openstack-infra | 14:50 | |
| fungi | yolanda: self-signed certs don't share an anything | 14:51 |
| fungi | they're self-signed | 14:51 |
| *** abregman|mtg is now known as abregman | 14:51 | |
| fungi | not signed by a ca | 14:51 |
| yolanda | fungi ok, then we need to change several things on our infra cloud setup. Can we make sure we are all on the same page? | 14:52 |
| pabelanger | asselin__: Right, I think moving them out is a good idea. I already do that with ansible (ansible-role-diskimage-builder). Then simply have nodepool include that role. The same concept could apply to our puppet modules | 14:52 |
| pabelanger | eg: puppet-diskimage-builder | 14:52 |
| yolanda | fungi, what do you think about a self-signed cert with alt names? for *.hpuseast.ic.openstack.org and *.hpuswest.ic.openstack.org | 14:52 |
| fungi | yolanda: what else needs to change beyond what was decided in the meeting when we said we were going with self-signed certificates? | 14:52 |
| fungi | yolanda: subject-altnames is fine, sure | 14:52 |
| *** julim has joined #openstack-infra | 14:52 | |
| yolanda | fungi, well , puppet code needs to change, hiera values need to change | 14:53 |
| fungi | or just *.ic.openstack.org | 14:53 |
| yolanda | fungi, tested that one and was buggy, because we are on two levels | 14:53 |
| yolanda | *.ic.openstack.org doesn't match controller.hpuseast.ic.openstack.org | 14:53 |
| yolanda | alt names worked fine | 14:53 |
| fungi | ahh, then sure, subject-altnames is the way to go | 14:53 |
| *** dizquierdo has quit IRC | 14:53 | |
| yolanda | so this needs agreement with crinkle and nibalizer, because it needs changes | 14:54 |
| fungi | i forgot some libraries don't consider wildcards to validate below one subdomain level | 14:54 |
| yolanda | part of the spec comes from the conversations we've had these days about how to handle certs | 14:54 |
| fungi | i thought they agreed in the infra meeting? | 14:54 |
| pabelanger | yolanda: if you are using ubuntu, can you use the ssl-cert package and just setup snakeoil certs? | 14:54 |
| pabelanger | eg: make-ssl-cert generate-default-snakeoil --force-overwrite | 14:54 |
| fungi | or did they not understand what self-signed certs actually are? | 14:54 |
| *** erlon has joined #openstack-infra | 14:55 | |
| yolanda | fungi, looks as we missed some bits there | 14:55 |
| fungi | i wonder if everyone fixated on the not-buying-certs part of that discussion rather than the actual proposal which was to explicitly use self-signed certs (i don't mind buying certs when they're needed, but this use case didn't seem to need them) | 14:56 |
| *** ccrouch has joined #openstack-infra | 14:56 | |
| *** rbrndt has joined #openstack-infra | 14:56 | |
| *** fhubik is now known as fhubik_brb | 14:57 | |
| yolanda | fungi the idea i ended having, from the conversation + the current puppet code, was have a CA for infra, and i think the other people working in infra cloud had the same idea | 14:57 |
| fungi | regardless, i'm probably missing something that makes the infra-cloud case not a matter of just having puppet apply blobs from hiera to create the usual certificate, key (and optionally chain cert when needed) files on the systems which need to serve them | 14:57 |
| fungi | oh, and the tricky bit, which is extending the trust to include the self-signed certs for things it needs to connect to | 14:58 |
| fungi | but that same need exists even if you set up your own ca | 14:58 |
| yolanda | for extending the trust i created a patch https://review.openstack.org/278368 | 14:58 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Removes posix-ipc from global-requirements https://review.openstack.org/217824 | 14:58 |
| yolanda | and we've found today a bug in neutron with self signed certs, people in puppet-openstack are fixing now | 14:58 |
| fungi | awesome. i love it when the things we build are able to surface bugs in other projects they didn't know they had | 14:59 |
| yolanda | we are amazing beta-testers | 15:00 |
| rcarrillocruz | well | 15:01 |
| rcarrillocruz | yeah | 15:01 |
| rcarrillocruz | i think this is the great thing about the infracloud | 15:01 |
| rcarrillocruz | is that we dogfood a lot of stuff | 15:01 |
| AJaeger | fungi, sdague : Could you review a pep8/linters change for me, please? https://review.openstack.org/278633 | 15:01 |
| rcarrillocruz | so it's a win-win for the whole project | 15:01 |
| *** burgerk has quit IRC | 15:01 | |
| *** zeih_ has quit IRC | 15:02 | |
| fungi | everyone working on infra-cloud is doing an amazing job of finding rough edges in openstack. it's one of the most exciting things about the opportunity to build this | 15:02 |
| *** tiswanso has quit IRC | 15:02 | |
| *** eharney has quit IRC | 15:02 | |
| *** Guest65333 is now known as nadya | 15:02 | |
| *** tiswanso has joined #openstack-infra | 15:03 | |
| pabelanger | rcarrillocruz: hopefully we'll be able to dogfood other OSes at some point in the future :) | 15:03 |
| rcarrillocruz | heh | 15:03 |
| rcarrillocruz | i heard you on the meeting, yup | 15:03 |
| rcarrillocruz | ;-) | 15:03 |
| openstackgerrit | Max Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository https://review.openstack.org/279120 | 15:03 |
| *** nadya is now known as Guest34507 | 15:03 | |
| *** Guest34507 is now known as _nadya_ | 15:03 | |
| pabelanger | rcarrillocruz: I actually download puppet-infracloud the other day and started adding rhel support. But haven't pushed up any patches yet. Holding off until sprint to see if there is interest | 15:04 |
| rcarrillocruz | i believe we should probably focus on making it work and really automate the whole thing. We have way too many pieces and cherry picked changes wandering around on the jumphosts | 15:06 |
| rcarrillocruz | then discuss HA | 15:06 |
| rcarrillocruz | then discuss dogfooding other stuff | 15:06 |
| rcarrillocruz | centos/rh | 15:06 |
| rcarrillocruz | monasca/ceilometer | 15:06 |
| rcarrillocruz | that's just me | 15:06 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Adds PyMI as a global-requirement https://review.openstack.org/279122 | 15:06 |
| *** jaosorior has quit IRC | 15:06 | |
| *** piet has joined #openstack-infra | 15:06 | |
| yolanda | so yes, right now manifest cannot work without patches. Several fixes going on | 15:06 |
| *** sc68cal has joined #openstack-infra | 15:07 | |
| mriedem | is the elastic-recheck bot dead? | 15:09 |
| mriedem | i'm not seeing it commenting on known tracked failures | 15:09 |
| mriedem | clarkb: ^? | 15:09 |
| *** sshnaidm has quit IRC | 15:10 | |
| fungi | mriedem: i can look in a sec and see if it's throwing errors or something | 15:10 |
| mriedem | thanks | 15:10 |
| fungi | 2016-02-11 14:50:30,168 INFO recheckwatchbot: Looking for failures in 244328,20 on gate-tempest-dsvm-neutron-full | 15:11 |
| fungi | seems to be active at least | 15:11 |
| *** annegentle_ has joined #openstack-infra | 15:11 | |
| *** sdake_ has joined #openstack-infra | 15:11 | |
| fungi | the bot seems to be running since yesterday. looks like it got restarted, maybe by a new patch? | 15:12 |
| *** piet has quit IRC | 15:12 | |
| *** fhubik_brb is now known as fhubik | 15:12 | |
| mriedem | i guess it did show up this morning on something https://review.openstack.org/#/c/266425/ | 15:12 |
| *** piet has joined #openstack-infra | 15:12 | |
| mriedem | Elastic Recheck6:49 AM↩Patch Set 25:I noticed jenkins failed, I think you hit bug(s): | 15:12 |
| fungi | yeah, there was a pattern removal change which landed yesterday, so puppet restarted it for that i guess | 15:13 |
| anteaya | anyone know why patchbot it in -meeting? | 15:15 |
| *** Daisy has joined #openstack-infra | 15:16 | |
| *** sdake_ is now known as sdake | 15:16 | |
| jeblair | if someone wants to run a bot in meeting, they can add it to puppet | 15:17 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:17 | |
| openstackgerrit | Merged openstack-infra/nodepool: Document clouds.yaml as the preferred configuration https://review.openstack.org/278529 | 15:17 |
| asselin__ | BobBall, does this work? https://review.openstack.org/#/c/279102/1/contrib/single_node_ci_site.pp | 15:17 |
| *** Daisy has quit IRC | 15:17 | |
| jeblair | notmyname: is patchbot you? | 15:17 |
| *** rhallisey has joined #openstack-infra | 15:18 | |
| *** arthus has quit IRC | 15:19 | |
| *** ig0r_ has quit IRC | 15:21 | |
| openstackgerrit | Max Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository https://review.openstack.org/279120 | 15:21 |
| *** eharney has joined #openstack-infra | 15:21 | |
| jeblair | notmyname: if you want to run bots in common openstack channels, would you mind please adding them to the openstack-infra configuration? you can find information on the relevant files here: http://docs.openstack.org/infra/system-config/irc.html | 15:22 |
| *** rhallisey has quit IRC | 15:23 | |
| jeblair | notmyname: in the mean time, i have removed patchbot from the meeting channel | 15:23 |
| anteaya | jeblair: thank you | 15:23 |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Rename fuel-plugin-astra to fuel-plugin-astara https://review.openstack.org/279138 | 15:24 |
| *** ddieterly has quit IRC | 15:24 | |
| *** sshnaidm has joined #openstack-infra | 15:25 | |
| *** burgerk has joined #openstack-infra | 15:26 | |
| anteaya | your kidding | 15:27 |
| anteaya | another rename patch | 15:27 |
| BobBall | asselin__: I assume so - but TBH I uploaded too early; currently having difficulties with the deploy elsewhere. | 15:27 |
| reed | more spam on the wiki | 15:27 |
| reed | https://wiki.openstack.org/wiki/Category_talk:Home | 15:27 |
| jeblair | someone should add a spellchecker lint that detects astara | 15:27 |
| asselin__ | BobBall, my intuition says it won't work as-is. | 15:28 |
| anteaya | reed: can you get rid of it? I recall last time you did | 15:28 |
| anteaya | jeblair: agreed | 15:28 |
| reed | anteaya, I sure can :) | 15:28 |
| anteaya | markmcclain: folks are having difficulty with the proper spelling of your repo name | 15:28 |
| anteaya | markmcclain: just as a heads up | 15:29 |
| mordred | anteaya, markmcclain: although to be fair it's still a better name than jeepyb | 15:29 |
| *** angdraug has joined #openstack-infra | 15:29 | |
| anteaya | it is, jeepyb is a terrible name, but it doesn't get mis-spelled | 15:30 |
| anteaya | reed: thank you | 15:30 |
| clarkb | or renamed every other week | 15:30 |
| anteaya | that too | 15:30 |
| *** daemontool_ has joined #openstack-infra | 15:32 | |
| *** jsavak has quit IRC | 15:32 | |
| *** dpaterson has quit IRC | 15:32 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check for astra misspelling https://review.openstack.org/279143 | 15:33 |
| AJaeger | anteaya, jeblair: Not sure whether the above works ^ | 15:33 |
| *** lezbar has quit IRC | 15:33 | |
| anteaya | ha ha ha | 15:33 |
| *** ajmiller has joined #openstack-infra | 15:33 | |
| anteaya | always at the ready with a linter change | 15:34 |
| AJaeger | anteaya, we have simple rules for Openstack etc ;) | 15:34 |
| anteaya | wish I knew what they were | 15:34 |
| AJaeger | anteaya: check the change, the two lines above mine | 15:34 |
| *** daemontool has quit IRC | 15:34 | |
| rbradfor | hi infra, I just wanted to know following on from the infra-cloud spec, puppet-infracloud cloud and infra-cloud topic details I've been reading what is a good next point to become more aware of work progress. Is reviewing the -infra channel history the best place to look next? | 15:34 |
| *** achanda has joined #openstack-infra | 15:34 | |
| *** kozhukal` has joined #openstack-infra | 15:35 | |
| anteaya | AJaeger: nice | 15:35 |
| *** rhallisey has joined #openstack-infra | 15:35 | |
| *** sshnaidm has quit IRC | 15:35 | |
| clarkb | rbradfor: our meeting logs typically have updates (and for infracloud links to various changes and docs too) | 15:35 |
| AJaeger | I fear the check is only for the description... | 15:35 |
| *** zul_ has joined #openstack-infra | 15:36 | |
| AJaeger | indeed ;( | 15:36 |
| *** kozhukalov has quit IRC | 15:36 | |
| *** zul has quit IRC | 15:36 | |
| anteaya | AJaeger: :( | 15:36 |
| rbradfor | clarkb, great thanks, meeting logs would be a much smaller volume place to start. | 15:36 |
| AJaeger | description is misspelled as well, so might have helped... | 15:36 |
| *** woodster_ has quit IRC | 15:36 | |
| *** kozhukalov_ has quit IRC | 15:36 | |
| anteaya | AJaeger: might have | 15:37 |
| AJaeger | exactly: *might* ;) | 15:37 |
| anteaya | rbradfor: have you spent much time talking with crinkle? | 15:37 |
| *** Daisy has joined #openstack-infra | 15:37 | |
| anteaya | or nibalizer? | 15:37 |
| *** piet has quit IRC | 15:37 | |
| anteaya | here is the current etherpad for the infra cloud sprint: https://etherpad.openstack.org/p/mitaka-infra-midcycle | 15:38 |
| rbradfor | anteaya, no, I've met crinkle and nibalizer in Vancouver and I've seen the puppet work via repos only. | 15:38 |
| *** FallenPegasus has joined #openstack-infra | 15:38 | |
| *** placko has joined #openstack-infra | 15:39 | |
| rbradfor | anteaya, thanks, another resource I'd not looked at to review, Tokyo or midcycle etherpads. | 15:39 |
| * rbradfor adds these to my reading list | 15:39 | |
| anteaya | awesome | 15:39 |
| *** rhallisey has quit IRC | 15:39 | |
| anteaya | and here is the storyboard story for infra-cloud: https://storyboard.openstack.org/#!/story/2000175 | 15:39 |
| *** mtanino has joined #openstack-infra | 15:40 | |
| *** jsavak has joined #openstack-infra | 15:40 | |
| rbradfor | anteaya, thanks had read through that | 15:40 |
| openstackgerrit | Victor Stinner proposed openstack-infra/project-config: Add non-voting gate-trove-python34 check https://review.openstack.org/279108 | 15:41 |
| fungi | reed: i think i reverted the last spam on that page | 15:41 |
| fungi | it seems to be popular | 15:41 |
| *** Daisy has quit IRC | 15:41 | |
| anteaya | rbradfor: wonderful | 15:41 |
| rbradfor | anteaya, put perhaps I cannot drive storyboard that well, other then story tasks, I didn't find any details? | 15:41 |
| fungi | was just last week if i'm remembering correctly | 15:41 |
| anteaya | rbradfor: well storyboard is also under active development | 15:42 |
| *** achanda has quit IRC | 15:42 | |
| *** fawadkhaliq has quit IRC | 15:42 | |
| yolanda | rbradfor, https://storyboard.openstack.org/#!/story/2000175 | 15:42 |
| anteaya | rbradfor: so there are some gaps in its functionality, one of them being the ablity to split out tasks within a story and add details to a specific task rather than the story as a whole | 15:42 |
| anteaya | Zara: ^^ | 15:43 |
| *** fawadkhaliq has joined #openstack-infra | 15:43 | |
| *** edmondsw has joined #openstack-infra | 15:43 | |
| fungi | worth noting, launchpad doesn't let you add task-specific details either | 15:43 |
| anteaya | rbradfor: so I think you have done a lot of reading and you might be at the point of participating in discussion | 15:43 |
| fungi | comments in lp are on the entire bug, regardless of how many bugtasks it has | 15:44 |
| * anteaya notes fungi's point about launchpad functionality | 15:44 | |
| anteaya | so parity | 15:44 |
| rbradfor | anteaya, I'll read the meeting notes from clarkb recommendation and I'll join the weekly -infra meeting to keep abreast more | 15:44 |
| markmcclain | anteaya: haha yeah I think lots of autocorrect change it astor | 15:44 |
| rbradfor | anteaya, thank you for point me in the right direction | 15:44 |
| anteaya | markmcclain: good to know | 15:44 |
| fungi | asteroids! | 15:44 |
| AJaeger | ;) | 15:45 |
| fungi | name idea for my next project | 15:45 |
| anteaya | markmcclain: mind adding that to https://review.openstack.org/#/c/279143/1/tools/check_valid_gerrit_projects.py | 15:45 |
| anteaya | I like asteroids | 15:45 |
| *** tiswanso has quit IRC | 15:45 | |
| yolanda | rbradfor, so one part of the work of infra cloud is work with bifrost, ironic, glean...for the deploy of servers. Then another work on top of that is ansible + puppet to install manifests for controller and compute, reusing puppet-openstack | 15:45 |
| AJaeger | anteaya: let me update my change, just tell me what to add. | 15:45 |
| AJaeger | anteaya: I'm sure it won't pass the testsuite as is... | 15:45 |
| anteaya | rbradfor: wonderful, thanks for wanting to get involved, we need more folks like you helping out | 15:45 |
| anteaya | rbradfor: thank you | 15:46 |
| *** tiswanso has joined #openstack-infra | 15:46 | |
| rbradfor | yolanda, thanks. other than knowing the names of bifrost, ironic and glean I've never read in detail or tried to use. Seems a next starting point after I learn more about them. | 15:46 |
| anteaya | AJaeger: well markmcclain pointed out astor is a popular choice of autocorrect | 15:46 |
| *** rhallisey has joined #openstack-infra | 15:46 | |
| AJaeger | anteaya: yes - just added to my patch. And now I test locally ;) | 15:47 |
| Zara | rbradfor: yolanda: there are also storyboard patches in review for linking resources to tasks: https://review.openstack.org/#/c/277481/ https://review.openstack.org/#/c/277831/ https://review.openstack.org/#/c/277882/ | 15:47 |
| anteaya | AJaeger: awesome | 15:47 |
| anteaya | Zara: knew you were working on something, yay! | 15:47 |
| anteaya | Zara: you are awesome \o/ | 15:47 |
| * anteaya is looking forward to cake next week | 15:47 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Add a method to download an image from glance https://review.openstack.org/278816 | 15:47 |
| *** placko has quit IRC | 15:48 | |
| Zara | rbradfor: yolanda: so those don't allow task descriptions within storyboard itself, but where there are more details than would fit sensibly in the story description, a user can link to those. | 15:48 |
| *** ddieterly has joined #openstack-infra | 15:48 | |
| Zara | anteaya: aw, thank you! :) | 15:48 |
| Zara | and me too! :) | 15:48 |
| yolanda | Zara that's great, it's something i miss sometimes, the ability to explain the tasks deeper | 15:49 |
| * fungi is missing out on teacakes | 15:49 | |
| *** jlanoux has quit IRC | 15:49 | |
| mordred | fungi: NEVER skip teacakes | 15:49 |
| yolanda | btw, fungi, testing a deploy of a controller with self-signed certs, going to see what breaks | 15:49 |
| fungi | but best wishes for the storyboard midcycle | 15:49 |
| * anteaya reviews Zara's posted patches | 15:49 | |
| clarkb | pabelanger: +1 to snakeoio if you can get away with them | 15:50 |
| clarkb | so simple | 15:50 |
| fungi | yolanda: no doubt there's likely something. it sounds like there are multiple places the cert needs to be added to the trust set | 15:50 |
| *** dangers_away is now known as dangers | 15:51 | |
| *** rhallisey has quit IRC | 15:51 | |
| AJaeger | anteaya: added and fixed | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Check for astra misspelling https://review.openstack.org/279143 | 15:52 |
| *** kozhukalov_ has joined #openstack-infra | 15:52 | |
| *** amotoki has quit IRC | 15:52 | |
| fungi | but if this gets worked out, it's a big win for anyone who wants to puppet a cloud test/poc deployment without bothering to get certs from an outside ca and without having to set up a ca of their own | 15:52 |
| *** rubasov has joined #openstack-infra | 15:53 | |
| Zara | yolanda: yes, it was really annoying me when I was updating stories! I mainly planned it as a way for people to link to things in gerrit and say 'patch here' (I was sick of putting that in descriptions/comments), but it should also work for docs. | 15:53 |
| *** GonZo2000 has quit IRC | 15:53 | |
| anteaya | AJaeger: wonderful thank you | 15:53 |
| *** EmilienM has quit IRC | 15:53 | |
| *** dpaterson has joined #openstack-infra | 15:54 | |
| anteaya | AJaeger: I think astara lowercase is acceptable, would you agree markmcclain? | 15:55 |
| *** HenryG has quit IRC | 15:55 | |
| *** niska has quit IRC | 15:55 | |
| anteaya | https://review.openstack.org/#/c/279143/2/tools/check_valid_gerrit_projects.py | 15:55 |
| *** sshnaidm has joined #openstack-infra | 15:55 | |
| *** rhallisey has joined #openstack-infra | 15:55 | |
| AJaeger | anteaya: it's just for that single file gerrit/projects.yaml... | 15:56 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Setup rabbit user/pass for keystone https://review.openstack.org/278482 | 15:56 |
| *** jcoufal has quit IRC | 15:57 | |
| anteaya | AJaeger: oh and just for the description | 15:57 |
| fungi | i'm convinced we should be maintaining a global project name spell-checker, but i'll let someone else fight that battle | 15:57 |
| anteaya | AJaeger: okay | 15:57 |
| fungi | er, not convinced | 15:57 |
| *** infra-red has quit IRC | 15:58 | |
| clarkb | TIL keystone uses rabbit | 15:58 |
| anteaya | fungi: well this is the third time we are renaming astara things | 15:58 |
| anteaya | I'm open to options | 15:58 |
| anteaya | folks can't seem to spell it correctly, including me | 15:58 |
| fungi | at least we already had plans to rename some stuff tomorrow | 15:58 |
| anteaya | yes | 15:59 |
| fungi | which reminds me, i forgot i have an appointment with my accountant tomorrow afternoon, but i should still be back in time for the maintenance. i just won't be around for a couple hours prior | 15:59 |
| rcarrillocruz | that's probably because of opel astra :D | 15:59 |
| rcarrillocruz | https://en.wikipedia.org/wiki/Opel_Astra | 15:59 |
| rcarrillocruz | just guessing | 15:59 |
| *** rhallisey has quit IRC | 15:59 | |
| *** achanda has joined #openstack-infra | 15:59 | |
| anteaya | and glad I saw the fuel rename patch as the person proposing it didn't add it to the wikipage: https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting | 15:59 |
| anteaya | fungi: happy taxes | 16:00 |
| *** mriedem is now known as mriedem_meeting | 16:00 | |
| *** rcarrillocruz has left #openstack-infra | 16:00 | |
| *** jordanP has joined #openstack-infra | 16:00 | |
| anteaya | zaro: did you get anywhere testing renaming on new gerrit? | 16:00 |
| *** tiswanso has quit IRC | 16:00 | |
| *** GonZo2000 has joined #openstack-infra | 16:00 | |
| *** GonZo2000 has quit IRC | 16:00 | |
| *** GonZo2000 has joined #openstack-infra | 16:00 | |
| anteaya | zaro: any new steps or changed steps we should be aware of? | 16:01 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: rename ceilometer-specs to telemetry-specs https://review.openstack.org/248897 | 16:01 |
| *** moravec has quit IRC | 16:01 | |
| * AJaeger just rebased the change above ^ | 16:01 | |
| *** tiswanso has joined #openstack-infra | 16:01 | |
| *** HenryG has joined #openstack-infra | 16:02 | |
| *** achanda has quit IRC | 16:02 | |
| anteaya | I see that | 16:02 |
| anteaya | thank you | 16:02 |
| anteaya | thanks | 16:03 |
| *** kozhukal` has quit IRC | 16:04 | |
| *** links has quit IRC | 16:04 | |
| AJaeger | anteaya: do you have time to review 278633, 278641, and 278627, please? | 16:04 |
| *** kozhukalov has joined #openstack-infra | 16:05 | |
| *** kzaitsev_mb has quit IRC | 16:05 | |
| sdague | AJaeger: before I +A https://review.openstack.org/#/c/278633/ - whats the jjb job count on that? | 16:06 |
| sdague | oh, I guess it's only a couple of instances | 16:06 |
| *** EmilienM has joined #openstack-infra | 16:06 | |
| anteaya | AJaeger: only if they are rename patches | 16:06 |
| anteaya | I want to get the rename things done without interruption if i can, so I can actually finish it | 16:07 |
| *** moravec has joined #openstack-infra | 16:07 | |
| *** piet has joined #openstack-infra | 16:07 | |
| *** niska has joined #openstack-infra | 16:07 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:07 | |
| *** jpr has joined #openstack-infra | 16:09 | |
| *** mattt has left #openstack-infra | 16:09 | |
| *** shardy has quit IRC | 16:10 | |
| *** kushal has quit IRC | 16:10 | |
| AJaeger | sdague: only touches zuul/layout | 16:10 |
| AJaeger | thanks, sdague. | 16:10 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Extend experimental-bindep-jobs to 14 projects https://review.openstack.org/278798 | 16:10 |
| AJaeger | anteaya: they're not. | 16:11 |
| *** kushal has joined #openstack-infra | 16:11 | |
| AJaeger | then let me ask sdague for 278641, and 278627 as well, please ;) | 16:11 |
| *** woodster_ has joined #openstack-infra | 16:11 | |
| AJaeger | fungi: do you have a spec or larger explanation of what you do with the bindep changes? | 16:12 |
| *** smanjara has quit IRC | 16:12 | |
| fungi | anteaya: AJaeger: looks like the author of that new rename patch replied to the maintenance announcement letting us know at least | 16:12 |
| fungi | AJaeger: this is part of the diskimage-builder rollout for nodepool images | 16:12 |
| fungi | AJaeger: need to make sure we can run jobs on dib-built ubuntu-trusty so we can get rid of bare-trusty | 16:12 |
| anteaya | AJaeger: okay thanks | 16:13 |
| anteaya | fungi: well that is something, thanks | 16:13 |
| clarkb | fungi: sdague http://logs.openstack.org/57/278557/1/gate/gate-tempest-dsvm-postgres-full/7381fc6/logs/screen-c-vol.txt.gz?level=WARNING appears to be what caused the failure of the preallocated swapfile gate run last night | 16:13 |
| fungi | AJaeger: but confirming it with a bunch of experimental equivalents of jobs on major projects before we start switching over the main job templates to use these builder macros | 16:13 |
| crinkle | fungi: yolanda the reason we've been using a CA and not just self-signed certs is that we were adding the CA to the system-trusted certificates, which was necessary because the puppet-openstack modules to not just roll with insecure certs | 16:14 |
| clarkb | fungi: sdague does that look like lack of disk space? the worlddump at the end of the run shows plenty of disk but that may be because tempest has stopped running | 16:14 |
| yolanda | mordred, greghaynes, SpamapS, can you review https://review.openstack.org/276292 ? it is the element for infra-cloud bridge | 16:14 |
| fungi | crinkle: you can't put the self-signed certs in as a "ca"? | 16:14 |
| *** cbader has quit IRC | 16:14 | |
| *** rhallisey has joined #openstack-infra | 16:14 | |
| yolanda | crinkle, i'm trying adding the cert on the ca.crt | 16:14 |
| yolanda | let's see if that works fine | 16:14 |
| fungi | because they are, they're just a ca that only ever signed one cert (which happens to be themselves) | 16:14 |
| *** vgridnev has quit IRC | 16:15 | |
| *** cbader has joined #openstack-infra | 16:15 | |
| AJaeger | fungi, so part of http://specs.openstack.org/openstack-infra/infra-specs/specs/dib-nodepool.html - a spec that has far too many holes ;) | 16:15 |
| crinkle | fungi: no I hadn't tried, didn't know that was a possibility | 16:15 |
| fungi | crinkle: there's really nothing special about a self-signed cert | 16:15 |
| yolanda | fungi crinkle, so deploying with cert in chain file field now, so it adds it to the global trsut | 16:16 |
| yolanda | trust | 16:16 |
| fungi | it's still a "certificate authority" in the strictest sense | 16:16 |
| *** dizquierdo has joined #openstack-infra | 16:16 | |
| fungi | just happens to serve as its own certificate authority | 16:16 |
| yolanda | btw crinkle, i needed to add this to manifest to work https://review.openstack.org/278362 | 16:16 |
| yolanda | and https://review.openstack.org/278368 | 16:16 |
| openstackgerrit | Merged openstack-infra/project-config: Use pep8 for doc projects https://review.openstack.org/278633 | 16:16 |
| mordred | fungi: ooh! I didn't realize we were that close to being able to test dib-ubuntu-trusty for everyting | 16:16 |
| mordred | fungi: that's exciting | 16:17 |
| fungi | the only place i would expect that to not work as intended is if something is explicitly disallowing self-signed certs (a certificate which is signed by a certificate authority that happens to be its own key) | 16:17 |
| mordred | these certs are for? | 16:17 |
| crinkle | fungi: yolanda in any case we've found that on liberty self-signed or signed-by-self-signed-ca certs don't work anyways | 16:17 |
| clarkb | mordred: the openstack services in infra cloud | 16:17 |
| yolanda | crinkle, i found a bug in neutron today, they are fixing it now | 16:17 |
| mordred | right - for the API endpoints? | 16:17 |
| yolanda | degorenko and EmilienM | 16:17 |
| fungi | mordred: we've been that close for a while, i just hadn't gotten around to broadening the validation to more jobs, and nobody else has had time to pick it up and run with it either, so i finally found a few minutes | 16:18 |
| clarkb | mordred: yes | 16:18 |
| mordred | fungi: neat! | 16:18 |
| mordred | clarkb: nod | 16:18 |
| crinkle | yolanda: weren't you having issues with openstackclient too? | 16:18 |
| EmilienM | yolanda: nice spot! | 16:18 |
| yolanda | crinkle, i fixed those. I didn't notice that my key had a pass | 16:18 |
| mordred | dare I ask why we don't just get real certs? I'm guessing it's in the scrollback | 16:18 |
| yolanda | so openstackclient was failing for that | 16:18 |
| mordred | I mean, we're talking like 5 certs per cloud at most | 16:18 |
| crinkle | yolanda: ah ok | 16:18 |
| yolanda | when i created that passwordless, opensatck client started working, the failure is on neutron | 16:18 |
| fungi | crinkle: can you elaborate? i believe that's just a matter of dealing with the fact that requests provides its own trust set rather than looking at the system trusts by default | 16:19 |
| yolanda | fungi, crinkle, i found that exporting REQUESTS_CA_BUNDLE can help | 16:19 |
| mordred | fungi: for intra-cloud interactions, there is no way to configure things to use specific certs | 16:19 |
| mordred | fungi: this is a known problem with openstack services talking to each other | 16:19 |
| crinkle | fungi: it sounds like yolanda figured it out | 16:19 |
| mordred | and with the way requests has decided to behave | 16:19 |
| *** bdemers has joined #openstack-infra | 16:19 | |
| clarkb | mordred: you can do the requests override | 16:20 |
| fungi | mordred: right, you'd have to add them to the trusted certificate authorities | 16:20 |
| clarkb | and do ^ | 16:20 |
| mordred | yah - but without exporting the env var wrapped around the API service start | 16:20 |
| fungi | but my point was that's going to be the same problem whether you use self-signed certs or decide to run your own certificate authority | 16:20 |
| mordred | there is no config file value to pass in the requests ca bundle args | 16:20 |
| yolanda | export REQUESTS_CA_BUNDLE=/etc/ssl/ca-certificates.crt helped with requests | 16:20 |
| mordred | fungi: yes | 16:20 |
| mordred | fungi: which is why I was wondering why we don't just buy cheap certs | 16:21 |
| bdemers | Can I get a new channel created? "#openstack-networking-cisco" Is there anything I should do, other then just request it here ? | 16:21 |
| *** bnemec has quit IRC | 16:21 | |
| jeblair | mordred: well, honestly, this ought to work | 16:21 |
| crinkle | yolanda: in the puppet run? | 16:21 |
| jeblair | mordred: i think it's worth exploring to see if we can do it or if not, what's so hard about it | 16:21 |
| mordred | jeblair: it does not- the support for this in openstack is broadly broken | 16:21 |
| fungi | it seems like punting on something we could make openstack do better | 16:21 |
| mordred | because of reuqests | 16:21 |
| yolanda | crinkle or global var, just having it on the env | 16:21 |
| mordred | this is a known problem | 16:21 |
| jeblair | mordred: yeah, but there's a workaround in requests that maybe could be applied here | 16:21 |
| crinkle | yolanda: okay awesome | 16:22 |
| *** vgridnev has joined #openstack-infra | 16:22 | |
| mordred | so - honestly fixing requests is the right thing -because requests' behavior is broken | 16:22 |
| yolanda | mordred, i could make it work properly unti i hit the neutron bug | 16:22 |
| mordred | and the attempts to fix it openstack side involved heaping more workaround on top of more workarounds | 16:22 |
| jeblair | mordred: i understand that it's not a lot of money, but it also doesn't get us any benefit, so that's weird. | 16:22 |
| clarkb | mordred: but requests has explicitly said they will not accept those changes... | 16:22 |
| mordred | for the fact that requests is categorically broken | 16:22 |
| *** vgridnev has quit IRC | 16:22 | |
| mordred | clarkb: I know | 16:22 |
| clarkb | bdemers: we have it documented somewhere, let me find that link | 16:22 |
| yolanda | it was a funny bug, creating a network called InsecureRequestWarning :) | 16:22 |
| bdemers | thanks | 16:23 |
| *** [HeOS] has joined #openstack-infra | 16:23 | |
| anteaya | bdemers: http://docs.openstack.org/infra/system-config/irc.html | 16:23 |
| mordred | the 'right' way to do it in creating the requests Session objects involves passing cacert options to the constructor | 16:23 |
| anteaya | clarkb: ^^ | 16:23 |
| clarkb | anteaya: thanks | 16:23 |
| *** HeOS has quit IRC | 16:23 | |
| mordred | but the various openstack services do not all provide config values in their files for specifying those in the areas where one configures intra-service communication | 16:24 |
| anteaya | clarkb: welcome | 16:24 |
| anteaya | clarkb: everybody is in the cert dustup | 16:24 |
| mordred | in any case - I actually do not think it is valuable to fix this in openstack | 16:24 |
| jeblair | i think it would be useful if they did, and i think it would be a good thing for someone to work on fixing that, even if it takes a little while. | 16:24 |
| mordred | since it further validates requests ridiculous position of breaking SSL everywhere | 16:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Move bindep-fallback to non-voting for trusty https://review.openstack.org/279161 | 16:25 |
| pabelanger | fungi: mordred: I thought I had pushed that up already ^ | 16:25 |
| mordred | there is a system on linux for dealing with this | 16:25 |
| Clint | is it called letsencrypt | 16:25 |
| mordred | and it works well | 16:25 |
| mordred | requests has chosen to break that for everyone | 16:25 |
| jeblair | mordred: i understand the stand on principal, but i don't think it's going to make requests change their mind, so in the mean time, we hurt folks who can and want to operate their own ca | 16:25 |
| *** jlanoux has joined #openstack-infra | 16:25 | |
| mordred | and people doing work to support requsts position is damaging to the entire ecosystem | 16:25 |
| jeblair | mordred: which is potentially a lot of private cloud operators | 16:26 |
| jeblair | (lots of big shops operate their own cas) | 16:26 |
| jeblair | (actually, like *every* big shop operates their own ca) | 16:26 |
| mordred | jeblair: indeed. however- fixing that is a non-small task and one I don't understand why we think we should take on as infra | 16:26 |
| mordred | it involves adding a lot of config options to a lot of openstack services | 16:26 |
| mordred | and piping them through to a lot of constructor calls | 16:26 |
| jeblair | mordred: i'm glad no one gave you that argument for any of the many quixotic tasks you have started ;) | 16:27 |
| mordred | :) | 16:27 |
| mordred | jeblair: fair enough | 16:27 |
| *** jordanP has quit IRC | 16:27 | |
| clarkb | Clint: only if letsencrypt is trusted by requests :/ | 16:27 |
| mordred | I just don't understand why of all of the tasks we might want to undertake, we'd choose "fix the openstack workarounds to requests breakage" over "just get 5 real certs" | 16:27 |
| clarkb | Clint: which it probably is, but you ahve to explicitly check because of how they manage their cert lists | 16:28 |
| mordred | _especially_ considering that we're running kilo | 16:28 |
| bdemers | anteaya, thanks! | 16:28 |
| jeblair | mordred: i don't fully understand the problem yet | 16:28 |
| jeblair | mordred: this is the first time i've heard about intra-cloud tasks being a problem -- how many points of interaction are there that we would run into? | 16:28 |
| mgagne | mordred: was about to say: so it can benefit to other people but then you said: "but we run kilo" :-/ =) | 16:28 |
| *** nwkarsten has quit IRC | 16:29 | |
| *** harlowja_at_home has joined #openstack-infra | 16:29 | |
| clarkb | jeblair: the big one are goint to be nova + glance, nova + neutron, and nova + cinder | 16:29 |
| clarkb | and everyone talking to keystone | 16:29 |
| *** nwkarsten has joined #openstack-infra | 16:29 | |
| fungi | mgagne: well, we have liberty in one region and kilo in the other i think | 16:29 |
| yolanda | fungi, mgagne, i'm testing on liberty | 16:30 |
| yolanda | as it's not yet on production, i prefered to test the upgrade | 16:30 |
| pabelanger | clarkb: do you have a link to the requests issue about rejecting the changes? Curious to see that | 16:30 |
| clarkb | pabelanger: http://docs.python-requests.org/en/master/user/advanced/#ca-certificates | 16:31 |
| pabelanger | clarkb: thanks | 16:31 |
| *** bpokorny has joined #openstack-infra | 16:31 | |
| fungi | anyway, i'm still not ruling out the possibility that we may just end up obtaining (and continually keeping on top of renewing) certs for infra-cloud ssl/tls interactions, but at the very least we should be able to pin down where using your own ca (which is what self-signed certs ultimately are, the trust challenges will be the same) breaks for an openstack deployment so that someone else has more | 16:31 |
| fungi | information on what needs fixing even if we don't have the bandwidth to solve all the issues ourselves | 16:31 |
| *** bpokorny has quit IRC | 16:31 | |
| anteaya | bdemers: welcome | 16:32 |
| *** bpokorny has joined #openstack-infra | 16:32 | |
| jeblair | fungi: ++ this has all been a surprise to me -- it certainly breaks my expectations of how the software should operate. getting it documented and raising awareness of the problem is very useful. | 16:32 |
| *** Swami has joined #openstack-infra | 16:32 | |
| fungi | also, i suspect this is actually _not_ a problem for people deploying distro packages of python-requests, as i believe the distros rip out the bundled trust set and point requests at the system-provided one | 16:32 |
| mgagne | fungi: could it be a cross-project initiative? I mean, I don't believe infra should be the only people working on it. | 16:33 |
| yolanda | so i'm testing with self signed now (not with infra ca), and now i'm getting problems | 16:33 |
| jeblair | fungi: oh! | 16:33 |
| mordred | fungi: I recommend reading the openstack-dev thread "Use of self signed certs in endpoints" | 16:33 |
| yolanda | fungi, with infra ca i managed to get it work, but with self-signed only, it complains about CA, even if the cert is added to trust | 16:33 |
| mordred | lemme find web link | 16:33 |
| pabelanger | fungi: Ya, believe that is the case. Going to check however | 16:33 |
| crinkle | (https://review.openstack.org/#/c/267817 and https://review.openstack.org/#/c/267863/ are patches to upgrade btw) | 16:33 |
| mordred | http://lists.openstack.org/pipermail/openstack-dev/2015-November/079468.html | 16:33 |
| openstackgerrit | Russell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl. https://review.openstack.org/269174 | 16:34 |
| crinkle | yolanda: does that have to do with the config files pointing at a path for a ca? | 16:34 |
| openstackgerrit | Gábor Antal proposed openstack-infra/gerritbot: Reduced code duplication https://review.openstack.org/279169 | 16:34 |
| pabelanger | fungi: Yup: simply patch: http://pkgs.fedoraproject.org/cgit/rpms/python-requests.git/tree/python-requests-system-cert-bundle.patch | 16:34 |
| mordred | fungi: ^^ that | 16:34 |
| fungi | mgagne: infra _is_ a cross-project initiative | 16:34 |
| mordred | yah | 16:34 |
| *** bpokorny_ has joined #openstack-infra | 16:34 | |
| yolanda | crinkle i removed the chainfile of apache | 16:34 |
| yolanda | and testing keystone now, that fails | 16:34 |
| yolanda | i will check keystone conf as well | 16:34 |
| mordred | using distro requests appropriate patches requests to behave like people expect it to behave on linux | 16:35 |
| *** nwkarsten has quit IRC | 16:35 | |
| *** lmiccini is now known as lmiccini|away | 16:35 | |
| *** sshnaidm has quit IRC | 16:35 | |
| mgagne | fungi: didn't meant that you weren't but that people outside the infra team can also contribute so you don't have to do all by yourself | 16:35 |
| mordred | but pip installed requests does not honor system ssl bundles or additions made to them by admins | 16:35 |
| clarkb | one terrible option would be a pip package that monkey patches requests | 16:35 |
| jeblair | or a simple wrapper around it | 16:36 |
| mordred | clarkb: notmorgan and I actually poked around at that a little while ago | 16:36 |
| crinkle | yolanda: can you paste the errors? | 16:36 |
| pabelanger | what is the issue what using python-requests from the OS? Just the version that ships right? | 16:36 |
| yolanda | just with curl curl: (60) SSL certificate problem: unable to get local issuer certificate | 16:36 |
| yolanda | curl --cacert /etc/ssl/certs/ca-certificates.crt https://controller00.hpuseast.ic.openstack.org:35357/v3/services | 16:36 |
| yolanda | curl: (60) SSL certificate problem: unable to get local issuer certificate | 16:36 |
| jeblair | mordred: i'm reading your thread now -- as you point out, we already "wrap" requests :) | 16:36 |
| mordred | pabelanger: mixing distro installed python packages and pip installed python packages leads to pain | 16:37 |
| *** skath_ has quit IRC | 16:37 | |
| *** mkwiek has quit IRC | 16:37 | |
| mordred | jeblair: we do - except we don't use that wrapper everywhere yet | 16:37 |
| *** alkhodos has joined #openstack-infra | 16:37 | |
| yolanda | crinkle, i pasted the full crt as trusted cert | 16:37 |
| *** rguillebert has quit IRC | 16:37 | |
| *** nmagnezi has quit IRC | 16:37 | |
| mordred | this gets easier once we're done getting all of the projects to use keystoneauth1 | 16:37 |
| mordred | but they don't yet (and certainly don't in kilo) | 16:37 |
| *** samyewell has quit IRC | 16:37 | |
| *** tphummel has joined #openstack-infra | 16:37 | |
| pabelanger | mordred: true. But I thought we were using packages for infracloud? Or are we pip installing deps? | 16:37 |
| *** ams_ has quit IRC | 16:37 | |
| *** bpokorny has quit IRC | 16:37 | |
| yolanda | pabelanger packages | 16:37 |
| *** nikhil_k has joined #openstack-infra | 16:38 | |
| jeblair | mordred: would we still need them each to pass in parameters? | 16:38 |
| *** jbryce has quit IRC | 16:38 | |
| *** ganso has quit IRC | 16:38 | |
| *** ashtokolov has quit IRC | 16:38 | |
| *** tpsilva has quit IRC | 16:38 | |
| mordred | jeblair: once we get keystoneauth to be the way services talk to each other, we could add a monkeypatch patch to ksa | 16:38 |
| *** igorbelikov has quit IRC | 16:38 | |
| mordred | jeblair: and then the need for parameters goes away | 16:38 |
| *** wendar_ has joined #openstack-infra | 16:38 | |
| clarkb | to clarify infra cloud should usedistro requests | 16:38 |
| *** zhenguo_ has quit IRC | 16:38 | |
| mordred | because as an admin you can just add ca bundles to the system | 16:38 |
| *** Ng has quit IRC | 16:38 | |
| *** tpeoples has quit IRC | 16:38 | |
| *** comstud has quit IRC | 16:38 | |
| mordred | if we're using distro requests | 16:38 |
| jeblair | yolanda: so that means the intra-cloud stuff can be taken care of by system ca-certs, and it's just when we use something external to that (eg nodepool) where we would have a problem? | 16:38 |
| pabelanger | clarkb: right, that's what I would expect | 16:39 |
| clarkb | because that is how the puppet stuff is installed | 16:39 |
| *** mwhahaha has quit IRC | 16:39 | |
| mordred | then we should be able to add ca bundles to the system CA | 16:39 |
| jeblair | (intra-infra-cloud :) | 16:39 |
| *** betherly has quit IRC | 16:39 | |
| *** jmccrory has quit IRC | 16:39 | |
| *** wendar has quit IRC | 16:39 | |
| *** piet has quit IRC | 16:39 | |
| *** mrmartin has joined #openstack-infra | 16:39 | |
| yolanda | jeblair i could make it work with and Infra CA | 16:39 |
| *** asahlin has quit IRC | 16:39 | |
| *** ayoung has quit IRC | 16:39 | |
| *** nikhil has quit IRC | 16:39 | |
| mordred | jeblair: yes. and telling nodepool to deal with the self-signed cert is easy | 16:39 |
| anteaya | no spelling mistake possibilies there | 16:39 |
| yolanda | but fails for me with a simple self signed | 16:39 |
| jeblair | mordred: yep | 16:39 |
| mordred | yolanda: yah - because you need the CA to be able to add it ot the system CA bundle | 16:39 |
| pabelanger | Ya, I think update-ca-certificates should work once the cert is in the proper location | 16:40 |
| mordred | pabelanger: ++ | 16:40 |
| jeblair | yolanda: if you add the self-signed cert to the ca-bundle it doesn't work? | 16:40 |
| *** rguillebert has joined #openstack-infra | 16:40 | |
| yolanda | mordred, but if i go with a simple self signed, i don't have the CA | 16:40 |
| *** ganso has joined #openstack-infra | 16:40 | |
| *** skath_ has joined #openstack-infra | 16:40 | |
| yolanda | jeblair, no, i added | 16:40 |
| yolanda | refreshed | 16:40 |
| *** krtaylor has quit IRC | 16:40 | |
| *** mkwiek has joined #openstack-infra | 16:40 | |
| yolanda | but fails | 16:40 |
| jeblair | fungi: have i just moved my brain to where you were like 45 minutes ago? | 16:40 |
| *** ashtokolov has joined #openstack-infra | 16:40 | |
| Clint | clarkb: it does as of v2.8.1 | 16:40 |
| yolanda | in nodepool we don't have the problem, because we can pass the cert, or the insecure flag | 16:41 |
| *** armax has joined #openstack-infra | 16:41 | |
| mordred | yolanda: ++ | 16:41 |
| *** ashleighfarnham has joined #openstack-infra | 16:41 | |
| yolanda | pain is to make it work with self signed, i thoguht that adding that to CA will do the trick | 16:41 |
| yolanda | at least is documented like that | 16:41 |
| *** salv-orl_ has joined #openstack-infra | 16:41 | |
| *** asahlin has joined #openstack-infra | 16:41 | |
| *** betherly has joined #openstack-infra | 16:41 | |
| *** samyewell has joined #openstack-infra | 16:41 | |
| *** igorbelikov has joined #openstack-infra | 16:42 | |
| jeblair | yeah, this seems like a fairly constrained problem -- we know adding *something* to the bundle should work, and it's a matter of working out if there's something unusual about adding a self-signed cert vs a ca-cert | 16:42 |
| *** esker has joined #openstack-infra | 16:42 | |
| fungi | yolanda: does /etc/ssl/certs/ca-certificates.crt include the self-signed cert? | 16:42 |
| *** sridhar_ram has joined #openstack-infra | 16:42 | |
| yolanda | fungi yes | 16:42 |
| *** scheuran has joined #openstack-infra | 16:42 | |
| *** mwhahaha has joined #openstack-infra | 16:43 | |
| *** tpeoples has joined #openstack-infra | 16:43 | |
| *** Ng has joined #openstack-infra | 16:43 | |
| *** piet has joined #openstack-infra | 16:43 | |
| fungi | sorry, too many conversations in different places, catching up again | 16:43 |
| *** isaacb has quit IRC | 16:43 | |
| *** jmccrory has joined #openstack-infra | 16:44 | |
| *** jbryce has joined #openstack-infra | 16:44 | |
| *** comstud has joined #openstack-infra | 16:44 | |
| *** zhenguo_ has joined #openstack-infra | 16:44 | |
| *** salv-orlando has quit IRC | 16:44 | |
| yolanda | fungi, even if i instruct curl to use it directly i get the problem curl --cacert /etc/ssl/certs/controller00.hpuseast.ic.openstack.org-keystone.pem https://controller00.hpuseast.ic.openstack.org:35357/v3/services | 16:45 |
| yolanda | curl: (60) SSL certificate problem: unable to get local issuer certificate | 16:45 |
| *** ams_ has joined #openstack-infra | 16:45 | |
| *** mikelk has quit IRC | 16:45 | |
| openstackgerrit | Max Lobur proposed openstack-infra/project-config: Create openstack/bareon-func-test repository https://review.openstack.org/279120 | 16:46 |
| fungi | yolanda: http://stackoverflow.com/questions/17597457/why-wont-curl-recognise-a-self-signed-ssl-certificate implies what you're doing should work | 16:46 |
| *** dizquierdo has quit IRC | 16:46 | |
| *** scheuran has quit IRC | 16:47 | |
| *** mrmartin has quit IRC | 16:47 | |
| yolanda | mm, it says to add the name to /etc/ca-certificates.conf | 16:47 |
| *** dtantsur is now known as dtantsur|afk | 16:47 | |
| openstackgerrit | Brian Demers proposed openstack-infra/project-config: Adding new channel #openstack-networking-cisco to bot lists https://review.openstack.org/279178 | 16:48 |
| *** hdd has joined #openstack-infra | 16:48 | |
| madorn | fungi: do you by any chance know the status of getting the openstack-dev/sandbox +2 issue resolved....it was this gerrit upstream bug btw: https://code.google.com/p/gerrit/issues/detail?id=3831 | 16:48 |
| yolanda | it that's on the bundle it shouldn't be like that | 16:48 |
| Clint | if you put it into /usr/local/share/ca-certificates you don't have to do that | 16:48 |
| Clint | but in that case you need to make the extension .crt | 16:48 |
| fungi | yolanda: well, that's if you're not explicitly pointing to the bundle with --cacert | 16:49 |
| *** smanjara has joined #openstack-infra | 16:49 | |
| yolanda | i tried several options an no one worked | 16:49 |
| *** piet has quit IRC | 16:49 | |
| pabelanger | Clint: then update-ca-certificates | 16:49 |
| fungi | madorn: i believe zaro indicated that the cleanup will require a gerrit outage. we're planning on for friday afternoon, so perhaps we should try to squeeze that into our maintenance plan | 16:49 |
| pabelanger | that's what I used to do | 16:49 |
| yolanda | openssl_client also shows Verify return code: 21 (unable to verify the first certificate) | 16:49 |
| Clint | pabelanger: right | 16:49 |
| fungi | er, planning one | 16:50 |
| madorn | fungi: great thank you! | 16:50 |
| *** nikhil_k is now known as nikhil | 16:50 | |
| pabelanger | yolanda: are we able to download the cacert to import locally? | 16:50 |
| yolanda | pabelanger yes, i can provide the details | 16:50 |
| Clint | maybe it's checking the CA flag in the cert | 16:51 |
| yolanda | going to paste key and cert | 16:51 |
| fungi | madorn: that is to say, it's my understanding the problem won't recur (not because of the upstream gerrit fix but because of the workaround we added to jeepyb to avoid triggering the bug), however it still left a mess in some dozens (hundreds?) of repositories which need to be cleaned up | 16:51 |
| fungi | one of which is the openstack-dev/sandbox repo | 16:51 |
| fungi | once we complete that offline cleanup, voting in those repos should work properly again | 16:51 |
| yolanda | pabelanger, or any others http://paste.openstack.org/show/486731/ | 16:52 |
| *** r-daneel has joined #openstack-infra | 16:52 | |
| *** ayoung has joined #openstack-infra | 16:52 | |
| *** amotoki has joined #openstack-infra | 16:52 | |
| *** mriedem_meeting is now known as mriedem | 16:52 | |
| *** tpsilva has joined #openstack-infra | 16:53 | |
| *** kzaitsev_mb has quit IRC | 16:53 | |
| *** kzaitsev_mb has joined #openstack-infra | 16:53 | |
| prometheanfire | anyone able to look at https://review.openstack.org/#/c/278211/ (dib) | 16:55 |
| *** amrith is now known as _amrith_ | 16:55 | |
| *** moravec has quit IRC | 16:55 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Add query for horizon connection refused test bug 1544619 https://review.openstack.org/279182 | 16:55 |
| openstack | bug 1544619 in OpenStack Dashboard (Horizon) "test_create_delete_instance race fails in gate-horizon-dsvm-integration with "error: [Errno 111] Connection refused"" [Undecided,Confirmed] https://launchpad.net/bugs/1544619 | 16:55 |
| *** mrmartin has joined #openstack-infra | 16:56 | |
| *** rcernin has quit IRC | 16:56 | |
| *** bnemec has joined #openstack-infra | 16:56 | |
| yolanda | mm, i'm reading something about copying hash of cert | 16:56 |
| yolanda | going to try | 16:56 |
| openstackgerrit | Martin Hickey proposed openstack/requirements: Update Oslo VersionedObjects to version 1.5.0 https://review.openstack.org/279003 | 16:56 |
| *** krtaylor has joined #openstack-infra | 16:57 | |
| *** daemontool_ has quit IRC | 16:57 | |
| *** slogan621 has quit IRC | 16:58 | |
| madorn | fungi: thanks | 16:58 |
| *** amotoki has quit IRC | 16:58 | |
| *** esker has quit IRC | 16:58 | |
| *** moravec has joined #openstack-infra | 16:59 | |
| *** nelsnels_ has quit IRC | 16:59 | |
| fungi | zaro: when you're around, we should see if we can squeeze cleanup of all those rogue "registered users" groups into the maintenance plan for tomorrow | 16:59 |
| *** nelsnelson has joined #openstack-infra | 17:00 | |
| yolanda | no luck | 17:00 |
| yolanda | infra-root , if you want to play with that, controller is on 15.126.48.53 | 17:01 |
| yolanda | you have keys there | 17:01 |
| *** samuelBartel has joined #openstack-infra | 17:01 | |
| *** tiswanso has quit IRC | 17:01 | |
| *** bpokorny_ has quit IRC | 17:01 | |
| anteaya | zaro: also I have an etherpad prepared with the current steps of our rename process before adding content, I want to confirm with you these steps are still accurate for 2.11: https://etherpad.openstack.org/p/repo-renaming-2016-02-12 | 17:02 |
| *** esker has joined #openstack-infra | 17:02 | |
| *** bpokorny has joined #openstack-infra | 17:02 | |
| greghaynes | prometheanfire: Do you have a use case in mind for that patch? | 17:02 |
| greghaynes | prometheanfire: I think its fine, just wondering | 17:02 |
| *** tiswanso has joined #openstack-infra | 17:02 | |
| clarkb | yolanda: how are you testing? | 17:03 |
| prometheanfire | greghaynes: ya, atm the gentoo cleanup script is at 99-cleanup, which runs before 99-package-uninstalls | 17:03 |
| greghaynes | prometheanfire: ah, ok | 17:03 |
| yolanda | clarkb, tried wget or curl, and openssl s_client to verify | 17:03 |
| clarkb | yolanda: might help if we have a write up of the crrent steps that have been taken so that we know what was done | 17:03 |
| clarkb | yolanda: and how was the ca cert installed? | 17:04 |
| yolanda | testing the https://controller00.hpuseast.ic.openstack.org:35357 endpoint | 17:04 |
| greghaynes | prometheanfire: We have had an idea for a while we haven't followed through on that all our scripts should be between 10- and 90- so that users can always add things before and after | 17:04 |
| greghaynes | prometheanfire: not really important here, just remembered it though | 17:04 |
| prometheanfire | greghaynes: that would be nice, would move it to 85 then | 17:04 |
| greghaynes | prometheanfire: I wouldn't do that yet, I think we should do the whole world at one time | 17:04 |
| prometheanfire | cleanup would be 89 | 17:04 |
| prometheanfire | ya, not yet | 17:04 |
| yolanda | clarkb so cert and key are configured in apache, for 35357 endpoint. the self signed cert has been copied to /etc/ssl/certs, and i ran update-ca-certificates | 17:04 |
| prometheanfire | just saying how it would be if so | 17:05 |
| greghaynes | prometheanfire: yep | 17:05 |
| yolanda | i am testing even passing the path, wget https://controller00.hpuseast.ic.openstack.org:35357 --ca-certificate=/etc/ssl/certs/controller00.hpuseast.ic.openstack.org-keystone.pem | 17:05 |
| yolanda | even with that fails complaining with Unable to locally verify the issuer's authority. | 17:05 |
| *** gyee has joined #openstack-infra | 17:05 | |
| *** sabeen3 has quit IRC | 17:06 | |
| *** esker has quit IRC | 17:07 | |
| BobBall | asselin__: Any way to get the nodepool yaml file read the provider_username/password from heira? | 17:07 |
| *** mrmartin has quit IRC | 17:08 | |
| asselin_ | BobBall, absolutelyyes | 17:08 |
| asselin_ | in fact we need to make that the only way | 17:08 |
| crinkle | yolanda: can you put `openssl s_client -connect controller00.hpuseast.ic.openstack.org:35357 -CAfile /path/to/cert` in a paste? | 17:08 |
| clarkb | yolanda: you need to copy it to /usr/local/share/ca-certificates then run update-ca-certificates according to the manpage | 17:08 |
| yolanda | clarkb, i did that, i have a patch in puppet that does the same, and i even did manually to double check | 17:09 |
| clarkb | ok that is not what you said you did above | 17:10 |
| yolanda | yep, missed that bits | 17:10 |
| clarkb | (which is why I think it is important we clarify exactly what w ehave done to test) | 17:10 |
| *** lucasagomes is now known as lucas-dinner | 17:10 | |
| yolanda | clarkb http://paste.openstack.org/show/486734/ | 17:10 |
| asselin_ | BobBall, a bit of that here, but we should make that the default https://review.openstack.org/#/c/237365/ | 17:11 |
| *** sabeen has joined #openstack-infra | 17:11 | |
| *** mrmartin has joined #openstack-infra | 17:12 | |
| *** derekh has quit IRC | 17:12 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default https://review.openstack.org/276581 | 17:12 |
| *** sdake has quit IRC | 17:12 | |
| *** ociuhandu has quit IRC | 17:12 | |
| *** sfinucan has quit IRC | 17:13 | |
| clarkb | crinkle: I think that paste was meant for you | 17:13 |
| *** e0ne has quit IRC | 17:14 | |
| *** esker has joined #openstack-infra | 17:14 | |
| *** esker has quit IRC | 17:14 | |
| *** esker has joined #openstack-infra | 17:14 | |
| yolanda | clarkb, the paste was for you, the output you requested | 17:14 |
| clarkb | that was crinkle | 17:14 |
| yolanda | doh | 17:15 |
| *** fhubik has quit IRC | 17:15 | |
| clarkb | yolanda: the .pem that is in /etc/ssl/certs, did you copy that or was that the result of update-ca-certificates operating on /usr/local/share/ca-certificates? | 17:15 |
| yolanda | puppet on infra cloud, copies the certs direclty on /etc/ssl/certs | 17:16 |
| zaro | morning | 17:16 |
| clarkb | I wonder if it would work if we put the crt in /usr/local/share/ca-certificates then run the update | 17:17 |
| yolanda | let me try | 17:17 |
| *** apoorvad has joined #openstack-infra | 17:17 | |
| clarkb | (basically I want to do what the manpage says you shoudl do, not depend on puppet) | 17:17 |
| BobBall | asselin_: Ah ok. Will look into that. I saw oscc_file_contents but the template said it would be generated by puppet so I ignored it :) | 17:17 |
| asselin_ | BobBall, yeah...you're the 2nd person to express confusion with that | 17:18 |
| yolanda | clarkb, same result | 17:18 |
| asselin_ | BobBall, perhaps we should change it. That's to notify the reader of the deployed version not to update it. But it needs to be populated by a human | 17:18 |
| clarkb | yolanda: did you remove the -CAfile so that it would use the system default? | 17:19 |
| clarkb | (that should avoid permissions issues or file name mismatches) | 17:19 |
| yolanda | clarkb yes | 17:20 |
| *** _nadya_ has quit IRC | 17:20 | |
| BobBall | asselin_: FWIW defaulting log_server to undef did give a full deployment, but I'm currently trying to figure out why my jobs.yaml wasn't read by jjb :) | 17:21 |
| BobBall | asselin_: I'm going to get this CI fully set up before I remove the workflow -1 in case there are some error messages lurking somewhere... | 17:21 |
| asselin_ | BobBall, I'm expecting issues with the zuul comment with the log file links | 17:22 |
| pabelanger | yolanda: back now, going to give it a try | 17:22 |
| yolanda | pabelanger thanks | 17:22 |
| BobBall | asselin_: Well I have to change that zuul comment in my setup | 17:22 |
| yolanda | i need to go now , but i'll connect later at night | 17:22 |
| asselin_ | BobBall, ok...perhaps you can use your patch to fully add swift support? | 17:23 |
| asselin_ | I never tested that | 17:23 |
| *** regXboi has joined #openstack-infra | 17:24 | |
| BobBall | asselin_: my swift support is custom hackery because swift middleware doesn't (didn't?) let you set container types, so if you use the regular swift upload support then the log files look ugly and n-cpu.log.gz gets downloaded to your system rather than opening in a browser. | 17:25 |
| pabelanger | yolanda: works here | 17:25 |
| pabelanger | $ curl --cacert infra.crt https://controller00.hpuseast.ic.openstack.org:35357/v3/services | 17:25 |
| pabelanger | {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}} | 17:25 |
| pabelanger | that is centos7 | 17:25 |
| yolanda | going to try pasting that on my local box | 17:25 |
| pabelanger | you testing from trusty? | 17:25 |
| yolanda | pabelanger, can you add the cert to global trust, in centos and check without the cacert ? | 17:26 |
| yolanda | yes, trusty | 17:26 |
| pabelanger | yolanda: sure, one sec | 17:26 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Periodic Devstack job against oslo.* from master https://review.openstack.org/279198 | 17:26 |
| *** annegentle_ has quit IRC | 17:26 | |
| anteaya | who is the un-named maroon person on https://etherpad.openstack.org/p/repo-renaming-2016-02-12? | 17:28 |
| anteaya | I'm guessing its fungi | 17:28 |
| pabelanger | yolanda: $ curl https://controller00.hpuseast.ic.openstack.org:35357/v3/services{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}} | 17:29 |
| pabelanger | import worked | 17:29 |
| crinkle | yolanda: this seems to be a problem in west as well now (it wasn't a problem two days ago) | 17:29 |
| yolanda | that's fine | 17:29 |
| yolanda | pabelanger | 17:29 |
| yolanda | weird | 17:29 |
| pabelanger | yolanda: also, you'll need to recreate your cert, since you posted a private key :) | 17:29 |
| fungi | anteaya: not me | 17:29 |
| anteaya | fungi: okay thanks | 17:29 |
| pabelanger | yolanda: let me spin up a trusty box | 17:29 |
| yolanda | pabelanger yes, no problems | 17:29 |
| yolanda | it's just a dummy cert not used anywhere | 17:29 |
| anteaya | must be either clarkb or jeblair then | 17:29 |
| fungi | anteaya: possibly zaro | 17:30 |
| AJaeger | team, I saw yesterday - a liberty page on http://docs.openstack.org/developer/neutron/devref/alembic_migrations.html, today it's master again. The page had the 7.0.3 tag from yesterday's neutron release. Where can I find the log files from neutron from the release queue? | 17:30 |
| pabelanger | yolanda: fedora 23 works fine too, just tested | 17:30 |
| anteaya | zaro: are you the un-named maroon person? | 17:30 |
| *** smanjara has quit IRC | 17:30 | |
| *** esker has quit IRC | 17:31 | |
| clarkb | not me | 17:31 |
| anteaya | fungi: ah thank you, yes you usually make a point of signing in, and are currently fancying black as your colour | 17:31 |
| mtreinish | anteaya, AJaeger: any ideas why the unit test jobs show UNSTABLE: https://review.openstack.org/#/c/267824/ | 17:31 |
| anteaya | clarkb: okay thanks | 17:31 |
| clarkb | mtreinish: http://logs.openstack.org/24/267824/2/check/gate-os-testr-python27/eb1409d/console.html#_2016-02-11_01_29_16_957 | 17:32 |
| fungi | AJaeger: i'm going to bet their docs job is using zuul-cloner and running on the tag, which z-c can't find so falls back to checking out master | 17:32 |
| AJaeger | fungi: that's what I want to figure out... | 17:32 |
| clarkb | mtreinish: possibly something funny on the logs server | 17:32 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP Search Changes https://review.openstack.org/279204 | 17:32 |
| anteaya | mtreinish: http://logs.openstack.org/24/267824/2/check/gate-os-testr-python27/eb1409d/console.html#_2016-02-11_01_29_16_957 | 17:32 |
| mtreinish | clarkb: hmm, that's weird. I did 2 or 3 rechecks and they all were like that | 17:32 |
| fungi | AJaeger: do `git show-ref 7.0.3` to get the tag's sha, and then use that to formulate the logs.openstack.org url | 17:32 |
| AJaeger | mtreinish: "ERROR: Failed to upload files: 2 No such file or directory" - no die | 17:33 |
| *** samuelBartel has quit IRC | 17:33 | |
| zaro | anteaya: i'm looking at the etherpad now | 17:33 |
| *** ociuhandu has joined #openstack-infra | 17:33 | |
| AJaeger | fungi: thanks, will do | 17:33 |
| anteaya | zaro: thanks are you maroon? | 17:33 |
| *** esker has joined #openstack-infra | 17:33 | |
| *** fawadkhaliq has quit IRC | 17:33 | |
| zaro | fushia? | 17:33 |
| clarkb | mtreinish: ps2's dir is owned by root and 755 | 17:33 |
| anteaya | okay I'll buy fushia | 17:33 |
| clarkb | mtreinish: so the jenkins user cannot create subdirs | 17:33 |
| anteaya | zaro: thanks, so the additions you made to the etherpad, are they improvements to our current workflow or changes/additions for 2.11? | 17:34 |
| clarkb | fungi: any idea why that may happen? | 17:34 |
| mtreinish | clarkb: well that would do it | 17:34 |
| zaro | fungi: i think we may not need to do offline indexing this time. | 17:34 |
| *** jsavak has quit IRC | 17:34 | |
| zaro | anteaya: improvements. | 17:35 |
| fungi | anteaya: i believe those additions are the cleanup for verify voting on openstack-dev/sandbox (and cleaning up all the various other repos that mentioned registered users in their acls too) | 17:35 |
| anteaya | zaro: okay so not 2.11 related, great | 17:35 |
| anteaya | fungi: ah thank you | 17:35 |
| anteaya | awesome | 17:35 |
| clarkb | if I were going to guess we have a rogue find updating ownership/perms? | 17:35 |
| *** arthus has joined #openstack-infra | 17:36 | |
| anteaya | zaro: did you come across any changes we should make now we are on 2.11? | 17:36 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: WIP Task Links https://review.openstack.org/277882 | 17:36 |
| fungi | clarkb: that's the best idea i can come up with | 17:36 |
| zaro | fungi: no, i haven't gotten to that yet. that's cleanup for projects that are kept lingering from renames | 17:36 |
| zaro | anteaya: i don't see anything that should be different | 17:36 |
| fungi | zaro: oh... now i'm looking at the script, yeah | 17:36 |
| clarkb | drwxr-xr-x 3 root root 4096 Jan 27 08:45 2 | 17:37 |
| anteaya | zaro: so you did a rename workflow on review-dev and our current process (for the gerrit steps) work as is? | 17:37 |
| fungi | zaro: so we still had some lingering cleanup beyond the deletions we performed during the upgrade? | 17:37 |
| *** sshnaidm has joined #openstack-infra | 17:37 | |
| *** abregman has quit IRC | 17:37 | |
| zaro | fungi: i don't think so, but if you change foo to bar. then foo is left in the db correct? don't we want to clean that out? | 17:38 |
| clarkb | crinkle: does that mean west has regressed and isn't functioning or just that some behavior has changed but it is clouding ok? | 17:38 |
| *** jlanoux has quit IRC | 17:38 | |
| zaro | fungi: isn't that why we had to run that cleanup script on the last gerrit upgrade? | 17:38 |
| *** esker has quit IRC | 17:39 | |
| fungi | zaro: i'm not following | 17:39 |
| *** esker has joined #openstack-infra | 17:39 | |
| anteaya | fungi zaro here is the etherpad from the last rename for reference: https://etherpad.openstack.org/p/repo-renaming-2015-10-17 | 17:40 |
| fungi | zaro: during the upgrade maintenance we ran that script you linked to clean up a mess left behind during some very old renames which had been done incorrectly | 17:40 |
| zaro | fungi: so if we change asta to astana. isn't asta still in the DB? | 17:40 |
| clarkb | mtreinish: fungi http://logs.openstack.org/24/267824/2/check/gate-os-testr-python27/d84038a/console.html.gz#_2016-01-26_09_45_50_028 that is where the dir got created | 17:40 |
| clarkb | mtreinish: fungi but that runs as the jenkins user so would've been jenkins:jenkins at that point | 17:40 |
| fungi | zaro: our rename process normally takes care of that | 17:40 |
| *** mrmartin has quit IRC | 17:40 | |
| fungi | zaro: it happens during step #5 | 17:40 |
| * anteaya notes another mis-spelling of astara, markmcclain | 17:40 | |
| zaro | fungi: ahh, ok. i didn't know that included cleanup. then no need for 5.5 | 17:41 |
| mtreinish | clarkb: hmm, so something changed the ownership between then and the next time the tests ran | 17:41 |
| crinkle | clarkb: yolanda it seemed like puppet was misbehaving due to ssl but I think I had environment variables set that were messing with it | 17:41 |
| clarkb | ya, looking at our file maintenance script it only rm's files and dirs and gzips files | 17:42 |
| clarkb | none of those operations should change ownership | 17:42 |
| fungi | though the documentation at http://docs.openstack.org/infra/system-config/gerrit.html#renaming-a-project is still some hybrid involving ansible playbooks which we've never actually tried | 17:42 |
| zaro | anteaya: i haven't tested a rename on review-dev but not sure that's necessary. | 17:42 |
| crinkle | clarkb: yolanda so puppet is functioning normally, the clients are functioning normally, but openssl does report return code 21 for neutron and nova endpoints | 17:42 |
| fungi | i thought we'd reverted that | 17:42 |
| *** jsavak has joined #openstack-infra | 17:42 | |
| Sam-I-Am | AJaeger: hi | 17:42 |
| *** kozhukal` has joined #openstack-infra | 17:42 | |
| anteaya | zaro: do you have time for a test today? would make me feel better if you did | 17:42 |
| *** esikachev has quit IRC | 17:42 | |
| clarkb | well gzip may change ownership of the file but not that higher level dir | 17:42 |
| crinkle | yolanda: does puppet work for you? | 17:43 |
| zaro | anteaya: i think if that stuff in etherpad worked previoiusly then should continue to work. | 17:43 |
| anteaya | zaro: would save us time if we hit something different tomorrow | 17:43 |
| fungi | zaro: so the database schema between 2.8 and 2.11 hasn't really changed in ways which would impact the database queries we used to perform during renames? | 17:43 |
| yolanda | crinkle sorry, i was just stepping out. Have you tried puppet from scratch? when it needs to create all services in keystone? a fresh deploy? | 17:43 |
| clarkb | crinkle: maybe openssl is acting stricter? | 17:43 |
| yolanda | it worked for me with the Infra CA. Not working with self signed, complains in keystone | 17:44 |
| *** kozhukalov_ has quit IRC | 17:44 | |
| zaro | anteaya: that would be a big test. | 17:44 |
| anteaya | zaro: what makes it big? | 17:44 |
| *** kozhukalov has quit IRC | 17:44 | |
| crinkle | yolanda: no I was verifying against the production one | 17:44 |
| anteaya | a usual rename takes 30 minutes or did on 2.8 | 17:44 |
| AJaeger | fungi, thanks, that helped. We indeed published to the wrong location | 17:44 |
| zaro | anteaya: all those steps. | 17:44 |
| *** esker has quit IRC | 17:45 | |
| yolanda | crinkle it will work, as is not contacting keystone to create tenants , endpoints, etc | 17:45 |
| clarkb | the last change to maintenance script merged last september | 17:45 |
| yolanda | to verify it needs to be tested on a fresh deploy | 17:45 |
| zaro | fungi: where are the scripts/queries? i can check those. | 17:45 |
| yolanda | at your point everything should be created so it will skip most of the steps | 17:45 |
| AJaeger | team, I have a problem with run-docs.sh: The neutron 7.0.3 (liberty) release yesterday pushed content to both docs.o.o./developer/neutron/7.0.3 *and* docs.o.o/developer/neutron/ - thus overriding the master documents there. | 17:45 |
| AJaeger | Problem is http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-docs.sh#n46 | 17:45 |
| AJaeger | We check for the latest tag and then publish. | 17:45 |
| crinkle | yolanda: it contacts keystone to check for existing resources | 17:45 |
| fungi | zaro: unless someone gets time to revert the documentation "update" that switched it to ansible we haven't actually used yet, we end up copying from previous maintenances like https://etherpad.openstack.org/p/repo-renaming-2015-10-17 | 17:45 |
| yolanda | crinkle, with self signed cert or infra ca as we have on hiera? | 17:46 |
| clarkb | mtreinish: I am stumped, short of someone modifying those file perms I don't know what could be doing that | 17:46 |
| AJaeger | We shouldn't do that for neutron that publishes after each merge, only for clients that publish only from release job. Correct? | 17:46 |
| fungi | zaro: that example is the stackforge rename maintenance, so the largest example we have | 17:46 |
| crinkle | yolanda: with nothing changed, will try without ca in a dev env | 17:46 |
| yolanda | crinkle so with ca and chain works here, with self signed fails | 17:47 |
| clarkb | actually | 17:47 |
| yolanda | can you test the same? | 17:47 |
| clarkb | there was a volume error when I was out traveling | 17:47 |
| clarkb | yolanda: when was that? could something done then have changed file perms on the logs fs? | 17:47 |
| anteaya | fungi zaro here is a smaller rename: https://etherpad.openstack.org/p/project-renames-Septemeber-2015 | 17:47 |
| *** dtardivel has joined #openstack-infra | 17:47 | |
| fungi | zaro: mostly want to make sure steps 5-7 are still valid | 17:47 |
| yolanda | clarkb, so i tested this morning with infra ca. That works fine. Then, per conversation with fungi on the channel, i changed to simply self signed certs. That fail | 17:48 |
| fungi | zaro: it sounds like you're saying step 7 is no longer needed? | 17:48 |
| yolanda | i can reproduced and make it work again with infra ca + chain, system is ok | 17:48 |
| clarkb | yolanda: sorry I am crossing the streams. Last month the logs server lost a cinder volume | 17:48 |
| fungi | yolanda: what chain? | 17:48 |
| yolanda | clarkb, not talking about that | 17:49 |
| clarkb | yolanda: I know, I am | 17:49 |
| fungi | yolanda: how did the "infra ca" have a chain/intermediate certificate and what would even have been in it? | 17:49 |
| *** mrmartin has joined #openstack-infra | 17:49 | |
| *** ifarkas has quit IRC | 17:49 | |
| clarkb | yolanda: trying to debug mtreinish's log problem. There was a directory on the logs fs created on january 27t | 17:49 |
| clarkb | yolanda: it is owned by root and causing future uploads to that dir to fail bceause uploads happen as the jenkins user | 17:49 |
| clarkb | anyways we can come back to that to avoid too many conversations at once | 17:50 |
| SpamapS | fungi: I was just reading your response to the apt mirror questions on the ML. I am a bit confused. Are we running an actual mirror of Ubuntu, or a partial one with different lists files (otherwise, why wouldn't we have signed package lists?) | 17:50 |
| fungi | ahh, right, the logs.o.o filesystem failure from a couple weeks ago | 17:50 |
| *** annegentle_ has joined #openstack-infra | 17:50 | |
| zaro | anteaya, fungi : thanks for narrowing it down. i can verify those steps. i'm pretty sure it's still ok on 2.11 but will verify anyway. | 17:50 |
| yolanda | clarkb, it can match the dates. However what i did is unmount, fsck + mount | 17:51 |
| yolanda | not updating anything else | 17:51 |
| *** annegentle_ has quit IRC | 17:51 | |
| anteaya | zaro: thanks so much | 17:51 |
| fungi | SpamapS: it's possible reprepro simply doesn't copy the signatures. anyway it's not a _full_ mirror since we omit i386 and restricted/multiverse | 17:51 |
| SpamapS | why would we use reprepro? | 17:51 |
| fungi | SpamapS: and reprepro regenerates the indices | 17:51 |
| fungi | so that they're consistent | 17:51 |
| clarkb | yolanda: thanks | 17:51 |
| SpamapS | err | 17:51 |
| zaro | fungi: yeah, step 7 might not be needed. we should be able to start gerrit then initiate the online indexing and let it index while gerrit is running. | 17:51 |
| clarkb | I wonder if it is possible for fsck to recover things that way | 17:51 |
| yolanda | fungi, an infra ca i generated manually. I created a CA , and intermediate, for openstack-infra ca, just a dummy one. Then i created a cert in top of that. using that setup, it works | 17:52 |
| fungi | SpamapS: our biggest issue to date with "other people's mirrors" has been inconsitent indices | 17:52 |
| zaro | fungi: initiate the index with ssh command. | 17:52 |
| fungi | yolanda: i'm still not understanding why you created (or even needed) an intermediate certificate | 17:52 |
| SpamapS | fungi: until we have the new apt from 16.04/unstable, there is no way to make apt consistent. :-/ | 17:52 |
| dims | AJaeger : one last job and i'll be done for now :) this time it's a devstack+tempest+oslo.* from master - https://review.openstack.org/#/c/279198/ | 17:52 |
| clarkb | SpamapS: the system we have in place should be pretty darn ood | 17:53 |
| fungi | SpamapS: whatcha mean by that? an index generated of the packages you have should be consistent | 17:53 |
| yolanda | fungi, following that https://jamielinux.com/docs/openssl-certificate-authority/ | 17:53 |
| openstackgerrit | Dmitriy Ukhlov proposed openstack-infra/project-config: Adds neutron tempest tests for oslo.messaging drivers https://review.openstack.org/279215 | 17:53 |
| *** annegentle_ has joined #openstack-infra | 17:53 | |
| fungi | SpamapS: rather than copying indices from somewhere else when they might be in flux | 17:53 |
| yolanda | in my server, then pasting the generated contents on hiera | 17:53 |
| *** esker has joined #openstack-infra | 17:53 | |
| mtreinish | clarkb: I've never heard of fsck changing ownership, but I can't really rule it out | 17:53 |
| AJaeger | dims: famous last words ;) | 17:53 |
| SpamapS | fungi: caching fail. But, in our context, that is N/A, so I see where reprepro + having to reconfigure apt to respect infra's key might be better than dealing with apt-mirror fail. | 17:53 |
| dims | AJaeger : i did say "for now" :) | 17:54 |
| * AJaeger ignored that | 17:54 | |
| clarkb | mtreinish: ya I hvaen't either | 17:54 |
| *** mrmartin has quit IRC | 17:54 | |
| SpamapS | fungi: note that Ubuntu's mirrors actually set Expires headers that avoid the problem 99.9% of the time. | 17:55 |
| fungi | yolanda: i see. yes this is a guide for people who are trying to maintain a secured certificate authority. adding a "signing key" below your primary ca key is certainly excessive for a low/no-security ca | 17:55 |
| SpamapS | fungi: it's just the other mirrors, that aren't archive.ubuntu.com, that don't do that right :-P | 17:55 |
| clarkb | I wonder how long a chown -R jenkins:jenkins /srv/static/logs would take | 17:55 |
| SpamapS | clarkb: ACK, I agree, it's actually quite close to perfect, and if you can guarantee no caching ever, then it should be 100%. | 17:56 |
| fungi | SpamapS: three nines of uptime for a package mirror is pretty terrible in our situation | 17:56 |
| SpamapS | fungi: archive.ubuntu.com has been that bad? | 17:56 |
| zaro | anteaya, fungi : improvement suggestion in etherpad. | 17:56 |
| SpamapS | ssuuuuuck | 17:56 |
| yolanda | fungi yes, but it worked without much problems with openstack, apart from neutron. If we can solve the problem with self-signed it will be better, but we can have this as an alternative if we hit a blocker | 17:56 |
| yolanda | anyway, just leaving home now, | 17:56 |
| yolanda | sorry | 17:56 |
| *** fawadkhaliq has joined #openstack-infra | 17:56 | |
| fungi | SpamapS: bad enough that we have separate elastic-recheck patterns tracking failures in archive.ubuntu.com vs mirror.rackspace.com or whatever | 17:56 |
| AJaeger | dims: one question inline on your change... | 17:57 |
| SpamapS | fungi: trrrble. Anyway, that explains it, I didn't know we were reprepro'ing, thanks. | 17:57 |
| dims | AJaeger : ah, good point, will do | 17:58 |
| fungi | SpamapS: also, we're currently _not_ signing the indices at all. it's something of a feature, as it discourages anyone using it outside our ci | 17:58 |
| SpamapS | fungi: yeah, +1 for not making our own ubuntu fork. :) | 17:59 |
| fungi | SpamapS: yesterday it turned up an ansible role testing bug, where the role was set to check whether you've disabled signature verification on your package lists. which means that the test was actually validating the apt configuration of the test host! crazy | 18:00 |
| *** SumitNaiksatam has joined #openstack-infra | 18:00 | |
| *** sabeen has quit IRC | 18:01 | |
| SpamapS | fungi: I replied to the ML for the diskimage-builder based things. apt-conf element, DIB_APT_CONF=/etc/apt/apt/conf .. done | 18:01 |
| fungi | awesome | 18:01 |
| BobBall | asselin_: I'm confused by using clouds.yaml with nodepool... What's the link from nodepool's yaml file into clouds.yaml to get the credentials? | 18:01 |
| SpamapS | err | 18:01 |
| SpamapS | there are too many slashes there | 18:01 |
| *** kozhukalov_ has joined #openstack-infra | 18:01 | |
| clarkb | BobBall: the cloud: key under the provider. It names a cloud inside of your clouds.yaml | 18:01 |
| fungi | SpamapS: so many slashes | 18:02 |
| asselin_ | BobBall, nodepool osclient usage looks at clouds.yaml file for credentials | 18:02 |
| fungi | SpamapS: but i read your reply, so good enough | 18:02 |
| fungi | thanks! | 18:02 |
| *** shardy has joined #openstack-infra | 18:02 | |
| *** Nikolay_St has quit IRC | 18:02 | |
| *** rossella_s has quit IRC | 18:03 | |
| BobBall | clarkb / asselin_: Yes but https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/templates/nodepool/clouds.yaml.erb, for example, suggests we should have a "cloud: rax" but there are 3 regions defined in the rax cloud. Will nodepool just use all three regions? (whereas before I had to specify a region to use) | 18:03 |
| *** rossella_s has joined #openstack-infra | 18:04 | |
| AJaeger | dougwig: could you review https://review.openstack.org/#/c/278051/ , please? | 18:04 |
| dougwig | AJaeger: yes | 18:04 |
| clarkb | BobBall: no, https://github.com/openstack-infra/project-config/blob/master/nodepool/nodepool.yaml#L168-L169 you specify both | 18:04 |
| mtreinish | mordred, Shrews: I was looking at the results from the shade functional jobs and I noticed it's not copying the subunit stream from the test run | 18:04 |
| mtreinish | so the testr_results.html isn't really useful: http://logs.openstack.org/16/278816/5/check/gate-shade-dsvm-functional-nova/14df2cd//logs/testr_results.html.gz | 18:05 |
| BobBall | ohhhhhhhhh - sorry. Makes sense. Thanks. | 18:05 |
| *** kzaitsev_mb has quit IRC | 18:05 | |
| BobBall | clarkb: I'd assumed that cloud: rax was enough to define the region+credentials etc. Awesome, thanks. | 18:05 |
| AJaeger | dougwig: and also https://review.openstack.org/#/c/278061/ , please | 18:05 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Add undercloud idempotency test to periodic job https://review.openstack.org/279218 | 18:05 |
| clarkb | BobBall: you can problaby make a cloud called rax-iad and another called rax-ord then drop the regions from nodepool config | 18:06 |
| pabelanger | yolanda: something is different with trusty, not working as I would expect | 18:06 |
| BobBall | ok, thanks. I'll test the better way first then fallback to that if I can't get it working. | 18:06 |
| *** jsavak has quit IRC | 18:07 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Periodic Devstack job against oslo.* from master https://review.openstack.org/279198 | 18:07 |
| dougwig | AJaeger: done | 18:07 |
| *** jsavak has joined #openstack-infra | 18:08 | |
| AJaeger | thanks, dougwig ! | 18:08 |
| pabelanger | yolanda: have to step out. But certs are good | 18:09 |
| *** esker has quit IRC | 18:10 | |
| clarkb | nibalizer: ping when you want to work on getting mirrors going (though I recognize this cert stuff may have to be sorted first) | 18:11 |
| *** _nadya_ has joined #openstack-infra | 18:11 | |
| fungi | zaro: any thoughts on doing the "registered users" group cleanup in the database and in the groups files of all the affected repos? | 18:11 |
| fungi | you had mentioned that due to the database change it needs to happen offline, so tomorrow might be a good time to get it out of the way while we're already going to be down to do project renames | 18:12 |
| *** hashar has quit IRC | 18:13 | |
| *** infra-red has joined #openstack-infra | 18:13 | |
| AJaeger | I have four patches for review: Frist two small devstack-gate cleanups, then two linters-pep renames one. Could any project-config core review them, please? 278051, 278061, 278641, 278627 yolanda gave her +2s already | 18:13 |
| *** esikachev has joined #openstack-infra | 18:13 | |
| fungi | AJaeger: did you find the doc publishing log you were looking for? any insights? i probably missed it in scrollback | 18:14 |
| *** mrmartin has joined #openstack-infra | 18:14 | |
| *** e0ne has joined #openstack-infra | 18:15 | |
| *** mrmartin has quit IRC | 18:15 | |
| clarkb | mtreinish: I went ahead and chowned that particular dir so you should be able to recheck and get uploads to work | 18:15 |
| mtreinish | clarkb: ok, I'll do that now | 18:16 |
| AJaeger | fungi, yes, I mentioned it at the wrong time. Full story at http://paste.openstack.org/show/486741/ | 18:16 |
| clarkb | mtreinish: I basically wans't getting anywhere with further debug so preserving the old state wasn't helpful | 18:16 |
| AJaeger | fungi: So, found the problem thanks to the ref but I'm lost on how to fix it | 18:16 |
| AJaeger | any help is welcome! | 18:16 |
| clarkb | I couldn't find anything that indicated fsck may cause ownership to change | 18:16 |
| *** rcernin has joined #openstack-infra | 18:17 | |
| *** infra-red has quit IRC | 18:18 | |
| *** esikachev has quit IRC | 18:18 | |
| *** infra-red has joined #openstack-infra | 18:19 | |
| *** dolphm has joined #openstack-infra | 18:19 | |
| *** baoli_ has quit IRC | 18:20 | |
| *** flepied has quit IRC | 18:20 | |
| fungi | AJaeger: so it's a logic issue in run-docs.sh? | 18:20 |
| AJaeger | fungi: that's my assessment | 18:21 |
| AJaeger | fungi: but I'd like confirmation for that - and an idea how to fix it (or a patch ;) | 18:21 |
| *** esker has joined #openstack-infra | 18:21 | |
| NobodyCam | morning infra folk | 18:22 |
| fungi | AJaeger: briefly skimming the conditional at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/run-docs.sh#n46 it seems like it should do what you're wanting | 18:22 |
| mtreinish | clarkb: I'm fine with just chalking it up to cosmic rays | 18:22 |
| fungi | AJaeger: make a copy from branch to tag if the tag is latest, otherwise move content from branch to tag | 18:22 |
| *** mriedem has quit IRC | 18:22 | |
| fungi | clarkb: did you start a find to look for other incorrect ownership in that tree? | 18:23 |
| clarkb | fungi: I have not | 18:23 |
| *** sabeen1 has joined #openstack-infra | 18:23 | |
| AJaeger | fungi: tag is on a stable branch and neutron publishes from master as well. | 18:23 |
| Sam-I-Am | AJaeger: hi | 18:23 |
| AJaeger | fungi: And thus the content gets published on the master location, overriding it | 18:23 |
| AJaeger | fungi: until the next master change merges and overrides again... | 18:24 |
| *** piet has joined #openstack-infra | 18:24 | |
| fungi | AJaeger: right, which suggests we're behaving like the "$TAG" = "$LATEST" part of teh conditional | 18:24 |
| AJaeger | fungi: yes, correct. We're in that if | 18:24 |
| *** mriedem has joined #openstack-infra | 18:24 | |
| fungi | AJaeger: http://logs.openstack.org/52/5271f1a4477b80e7ee3683bbebcea0b40dd4b33b/release/neutron-docs/ae6f77e/console.html#_2016-02-10_17_09_55_917 | 18:24 |
| fungi | "2016-02-10 17:09:55.917 | + LATEST=7.0.3" | 18:25 |
| NobodyCam | I have a question: I need to adjust the build_timeout for gate-tempest-dsvm-large-ops test but am unable to locate the correct place to do this. can someone point /me inthe right direction (this is for stable liberty btw) | 18:25 |
| anteaya | NobodyCam: hey there | 18:25 |
| NobodyCam | hey anteaya | 18:25 |
| fungi | AJaeger: so something is wrong with the definition of LATEST in line 43 of run-docs.sh | 18:25 |
| AJaeger | fungi, That if sounds correct to me for repos that use openstack-client-publish-jobs and do not publish from post. But here we use openstack-server-publish-jobs and do publish from post as well | 18:25 |
| anteaya | NobodyCam: why do you want to adjust the build_timeout? | 18:26 |
| clarkb | fungi: something like find /srv/static/logs -uid 0 -gid 0 ? should we exec a chown or just collect a list? | 18:26 |
| AJaeger | fungi: I don't think so. 7.0.3 is the lastest tag | 18:26 |
| NobodyCam | anteaya: https://review.openstack.org/#/c/278792 | 18:26 |
| clarkb | hrm may want to or those two | 18:26 |
| AJaeger | fungi: but you're right 8.0.0.0b1 is a newer tag... | 18:27 |
| NobodyCam | a backport is timing out at 196 seconds ... we need to bump it up | 18:27 |
| fungi | AJaeger: oh... yes this is a logic error, you're right. we should only update the branch if we're not normally updating it in post | 18:27 |
| AJaeger | fungi: but we would be in this situation even if there wouldn't be a newer tag | 18:27 |
| NobodyCam | so I'm looking for the correct place to set BUILD_TIMEOUT for the ironic test there | 18:27 |
| *** eil397 has joined #openstack-infra | 18:28 | |
| *** zeih_ has joined #openstack-infra | 18:28 | |
| Sam-I-Am | AJaeger: why isnt devref published off master like we do for things like the admin guide? | 18:28 |
| AJaeger | Sam-I-Am: it is. We're just overriding it. | 18:28 |
| fungi | AJaeger: the regex on line 43 omits prerelease tag patterns (by design) which is why 7.0.3 is considered LATEST | 18:28 |
| anteaya | NobodyCam: okay I think this is a change you need to make within the test | 18:28 |
| *** ihrachys has quit IRC | 18:28 | |
| anteaya | NobodyCam: I wouldn't classify this as a build_timeout error from an infra perspective | 18:29 |
| AJaeger | fungi: but even if we just branched off 7.0 and have opened master with no tags on it, it would be wrong. | 18:29 |
| anteaya | since the build completed and tests were executed | 18:29 |
| fungi | Sam-I-Am: the only good reason i can think of is that the commit for 7.0.3 would have been published in post as 7.0.2+ because the 7.0.3 tag wasn't pushed at the time | 18:29 |
| fungi | but if a newer commit than the tagged one has also merged we really shouldn't touch the master docs | 18:29 |
| AJaeger | fungi, do we have to create a new {name}-server-docs job? | 18:30 |
| Sam-I-Am | hmmmm | 18:30 |
| AJaeger | fungi: and then change behaviour based on job name? | 18:30 |
| *** Jeffrey4l has quit IRC | 18:30 | |
| fungi | AJaeger: i think we should just be able to check whether the master branch tip is what the tag is pointing at. if so, publish master and tag docs, if not, only publish tag docs | 18:31 |
| Sam-I-Am | here's a q - if someone does a devref update in liberty, does the job publish the liberty branch instead of master? | 18:31 |
| NobodyCam | anteaya: we do this for master branch tests: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L340 | 18:31 |
| Sam-I-Am | s/branch/tag | 18:31 |
| *** sdake has joined #openstack-infra | 18:31 | |
| *** kozhukal` has quit IRC | 18:31 | |
| clarkb | fungi: I am running that find with an -or on a subset of the logs dir | 18:32 |
| AJaeger | fungi: but tagging is not necessarily the tip - with releases repo, the team can tag an older change | 18:32 |
| clarkb | find /srv/static/logs/24 \( -uid 0 -or -gid 0 \) | 18:32 |
| NobodyCam | but I not sure where to set it for the large opts test that is failing on the backport I put up | 18:32 |
| clarkb | nothing yet | 18:32 |
| Sam-I-Am | AJaeger: i think thats what i'm saying here | 18:32 |
| anteaya | NobodyCam: oh okay well you know more than I do then | 18:32 |
| Sam-I-Am | tags are arbitrary values | 18:32 |
| NobodyCam | :p | 18:32 |
| *** gordc has joined #openstack-infra | 18:32 | |
| *** kozhukalov_ has quit IRC | 18:33 | |
| Sam-I-Am | so latest may be 7.0.3 instead of 8.0.0b3 or whatever | 18:33 |
| fungi | clarkb: optionally you could have done -not \( -user jenkins -group jenkins \) i think | 18:33 |
| clarkb | fungi: good point | 18:33 |
| fungi | clarkb: but really, it's likely either correct or 0 | 18:33 |
| fungi | so what you have should be fine | 18:33 |
| *** zhenguo_ has quit IRC | 18:33 | |
| AJaeger | fungi: waht about: check if the tag is pointing at any change in master? | 18:33 |
| sdague | dstufft: I'm finding really weird behavior with virtualenv 14.0.6 | 18:34 |
| dstufft | sdague: yea? | 18:34 |
| sdague | it's scrubing out /usr/local... so it won't work on my linux box | 18:34 |
| sdague | with tox | 18:34 |
| fungi | AJaeger: well, it's possible to push a tag for something which isn't the branch tip, so that would cause it to roll back the master documentation to an earlier commit (though it would update the version number). i guess that's a trade-off, i don't really know which is better | 18:34 |
| anteaya | NobodyCam: sorry I'm not more help | 18:34 |
| *** pvaneck has joined #openstack-infra | 18:34 | |
| *** ashtokolov has quit IRC | 18:34 | |
| Sam-I-Am | fungi: devref should always reference master, imo | 18:35 |
| *** kushal has quit IRC | 18:35 | |
| *** edwarnicke has quit IRC | 18:35 | |
| *** mtanino__ has quit IRC | 18:35 | |
| *** teran has quit IRC | 18:35 | |
| NobodyCam | anteaya: don't be at all | 18:35 |
| Sam-I-Am | i dont think we publish stable branches of it | 18:35 |
| Sam-I-Am | or if there's even a reason | 18:35 |
| *** esikachev has joined #openstack-infra | 18:35 | |
| anteaya | NobodyCam: I hope you find what you need, though, if that is helpful | 18:35 |
| *** rhallisey has quit IRC | 18:35 | |
| AJaeger | Sam-I-Am: it should - and that's what we discuss here. The problem is that a liberty change overwrote it | 18:35 |
| *** mtanino__ has joined #openstack-infra | 18:35 | |
| *** teran has joined #openstack-infra | 18:35 | |
| AJaeger | Sam-I-Am: We publish the complete docs tree... | 18:35 |
| *** zhenguo_ has joined #openstack-infra | 18:35 | |
| dstufft | sdague: what do you mean it's "scrubing out /usr/local" | 18:36 |
| fungi | AJaeger: i think what Sam-I-Am is saying is we shouldn't run this job in the release pipeline, only post | 18:36 |
| Sam-I-Am | seems like a devref change for a prior release is pointless | 18:36 |
| Sam-I-Am | unless we're going to start publishing releases of it | 18:36 |
| Sam-I-Am | i'm curious which patch did that... could have been a backport? | 18:36 |
| AJaeger | Sam-I-Am: we publish http://docs.openstack.org/developer/neutron/7.0.3/ | 18:36 |
| AJaeger | Sam-I-Am: not only devref | 18:36 |
| *** baoli has joined #openstack-infra | 18:37 | |
| *** ccrouch has quit IRC | 18:37 | |
| *** ashtokolov has joined #openstack-infra | 18:37 | |
| *** rhallisey has joined #openstack-infra | 18:37 | |
| *** shashank_hegde has joined #openstack-infra | 18:37 | |
| AJaeger | fungi: but indeed, I wonder why we need these... | 18:37 |
| Sam-I-Am | AJaeger: how does one get to those other versions? | 18:38 |
| Sam-I-Am | they dont use the conventional release names that docs.o.o does | 18:38 |
| AJaeger | Sam-I-Am, fungi : That's why I wonder whether we need them - nobody knows about these... | 18:38 |
| *** jlvillal has quit IRC | 18:38 | |
| Sam-I-Am | to me, it seems developers would be updating docs for master because thats where the active development occurs | 18:38 |
| AJaeger | There's also http://docs.openstack.org/developer/neutron/liberty/ | 18:38 |
| Sam-I-Am | AJaeger: which is even another tag heh | 18:38 |
| sdague | dstufft: http://paste.openstack.org/show/486749/ | 18:38 |
| *** angdraug has quit IRC | 18:39 | |
| AJaeger | and we have http://docs.openstack.org/developer/python-neutronclient/4.0.0/ and http://docs.openstack.org/developer/python-neutronclient/ -but no liberty | 18:39 |
| Sam-I-Am | AJaeger: i dont think we have ways to get to any of those "old" versions | 18:39 |
| *** jlvillal has joined #openstack-infra | 18:39 | |
| fungi | AJaeger: yeah, the logic in that job seems to assume all tags are related to the master branch, so stable branch docs are only updated for stable commits in post | 18:39 |
| Sam-I-Am | unless you happen to know the link exists | 18:39 |
| *** mrmartin has joined #openstack-infra | 18:40 | |
| fungi | the script relies on `echo $ZUUL_REFNAME | grep stable/` which will never match on a tag | 18:40 |
| AJaeger | yep | 18:40 |
| dstufft | sdague: I bet this is something to do with the sys.path munging that virtualenv does :/ | 18:40 |
| fungi | since $ZUUL_REFNAME in this case is refs/tags/7.0.3 not refs/heads/stable/liberty | 18:41 |
| *** gordc has quit IRC | 18:41 | |
| clarkb | Sam-I-Am: AJaeger projects could publish links as part of their index on the master side that grows as new tags and releases are made | 18:41 |
| *** rhallisey has quit IRC | 18:41 | |
| clarkb | not sure that that is a problem with our publishing of them | 18:42 |
| sdague | dstufft: yeh, I don't know, I followed the homepage link for virtualenv and got - https://virtualenv.pypa.io/en/latest/ which is a 404 | 18:42 |
| AJaeger | clarkb: I doubt any of them does... | 18:42 |
| sdague | so I defaulted to asking you | 18:42 |
| clarkb | AJaeger: they don't, my point is we do our best to make sure the things are published | 18:42 |
| clarkb | AJaeger: indexing those things is up to the projects themselves | 18:42 |
| fungi | clarkb: the problem in this case is that the run-docs.sh logic around tags is naive | 18:43 |
| sdague | 14.0.5 seems to work | 18:43 |
| dstufft | sdague: open an issue on github.com/pypa/virtualenv please, I have to run to the store but I'll take a look when I get back | 18:43 |
| sdague | dstufft: ok, cool. It would be nice if that was linked in the pypi record somewhere | 18:43 |
| AJaeger | clarkb: yes, we give them the option - and I expect nobody knows about it | 18:43 |
| *** moravec has quit IRC | 18:43 | |
| *** piet has quit IRC | 18:43 | |
| fungi | clarkb: i _think_ someone tried to "fix" the script to work around the "bug" of docs for master not mentioning the latest release once a tag is pushed, but didn't think through the implications | 18:44 |
| crinkle | yolanda: I was able to reproduce your issue and solve it, i have a number of possible troubleshooting steps | 18:45 |
| Sam-I-Am | we only need to publish devref changes to master | 18:45 |
| fungi | clarkb: AJaeger: Sam-I-Am: it may be easiest to just accept that cosmetic nuisance and make the script either publish tag documentation or branch documentation but not try to suss out when it should do both at the same time | 18:45 |
| Sam-I-Am | if someone backports a patch that includes a devref change... it should really do nothing? | 18:45 |
| AJaeger | fungi: but then python-XXXclient would never publish to developer/python-XXXclient ;( | 18:46 |
| AJaeger | Since the clients only publish from release queue | 18:46 |
| Sam-I-Am | sounds like we have 2 cases here. things that use a release queue, and things that should always be master? | 18:47 |
| AJaeger | Sam-I-Am: for me devref is neutron/doc/source/devref - but we talk about all of neutron/doc/source. | 18:47 |
| anteaya | fungi zaro I'm going to start populating that etherpad | 18:47 |
| fungi | anteaya: thanks! | 18:47 |
| Sam-I-Am | AJaeger: thats what i mean doc/source | 18:47 |
| anteaya | thank you | 18:47 |
| fungi | AJaeger: it may make more sense to not publish two copies of client docs, and instead publish a redirect to the tag (which gets updated when the tag is updated) | 18:48 |
| *** kushal has joined #openstack-infra | 18:48 | |
| AJaeger | fungi: you've lost me, I'm not following | 18:49 |
| clarkb | fungi: a good chunk of /srv/static/logs/24/271124/1 is also root owned | 18:49 |
| fungi | AJaeger: let me dig up a specific example, hold on | 18:49 |
| AJaeger | thanks, fungi | 18:49 |
| *** edwarnicke has joined #openstack-infra | 18:50 | |
| *** eliqiao_ has quit IRC | 18:50 | |
| clarkb | four of the five gate jobs for that change have root owned dirs and files | 18:50 |
| clarkb | but not all of them making it even weirder | 18:50 |
| *** esker has quit IRC | 18:51 | |
| AJaeger | clarkb: permissions? Non-writable directory? | 18:51 |
| *** infra-re_ has joined #openstack-infra | 18:51 | |
| *** zeih_ has quit IRC | 18:51 | |
| fungi | AJaeger: so http://docs.openstack.org/developer/python-novaclient/ could be a temporary (not permanent) http redirect to http://docs.openstack.org/developer/python-novaclient/3.2.0/ and when a tag for python-novaclient 3.2.1 is pushed, publish only http://docs.openstack.org/developer/python-novaclient/3.2.1/ and update the redirect to point at that version | 18:51 |
| clarkb | AJaeger: it is 755 and 644 so the jenkins user can't write to it | 18:52 |
| *** eliqiao_ has joined #openstack-infra | 18:52 | |
| clarkb | AJaeger: this is unrelated to the publishing of docs conversation | 18:52 |
| clarkb | fungi: dougwig I have held the slave running https://jenkins02.openstack.org/job/gate-rally-dsvm-neutron-neutron/2502/ and ssh'd into it | 18:52 |
| clarkb | hopefiully we see what is going on | 18:52 |
| AJaeger | clarkb: yes, I know that you're fixing the logs issues | 18:52 |
| fungi | AJaeger: that way the docs are only updated when a new release happens, and "main" copy of the documentation (as determined by the http redirect) always mentions the right version number | 18:52 |
| *** mrmartin has quit IRC | 18:53 | |
| *** gyee has quit IRC | 18:53 | |
| clarkb | they were also updated last at january 27 0845 | 18:53 |
| zaro | fungi: will add steps to etherpad on the registered users cleanup. i think it's pretty isolated and safe to do. I will not be around tomorrow so will provide info so you can decide whether you'de like to do it. | 18:53 |
| jeblair | clarkb, fungi, AJaeger: https://review.openstack.org/#/c/13725/ | 18:53 |
| AJaeger | fungi: yes, that would work for the clients - but not for the server | 18:53 |
| clarkb | something happened at 0845 to do this | 18:53 |
| fungi | zaro: that's awesome, thanks for the help! | 18:53 |
| AJaeger | jeblair: tahnks | 18:53 |
| clarkb | jeblair: yup iirc annegentle_ didn't want to confuse people consuming the client iwth unreleased feature documentation | 18:53 |
| *** mrmartin has joined #openstack-infra | 18:54 | |
| *** infra-red has quit IRC | 18:54 | |
| jeblair | viewing that change required that i sign in -- is that true for other people too? | 18:54 |
| *** daemontool has joined #openstack-infra | 18:54 | |
| jeblair | clarkb: i don't see any discussion about why that wouldn't apply to server docs though... | 18:54 |
| clarkb | jeblair: you can "go anonymous" | 18:54 |
| fungi | AJaeger: right, i'm agreeing that we probably need _that_ behavior for client publication (things which only publish on release) and a branch-or-tag-but-not-both publication mechanism for "servers" (things that have branch versions of documentation as well as tag versions of documentation) | 18:54 |
| AJaeger | the nice thing with our discussion is that we have a self-healing problem. The next merge for master will fix it ;) | 18:55 |
| anteaya | jeblair: I can see 137725 in chrome without signing in | 18:55 |
| pabelanger | jeblair: I can view it and not logged in | 18:55 |
| jeblair | clarkb: oh ok, so not related | 18:55 |
| *** amotoki has joined #openstack-infra | 18:55 | |
| fungi | AJaeger: if we want to not duplicate too much of the logic, we just have two jobs which call the script with different parameters | 18:55 |
| clarkb | the thought with the server dev docs was people consuming them are hacking on the tip of $brnach | 18:55 |
| clarkb | which is slightly different than someone wanting to consume the nova api with novaclient and installing the latest version on pypi | 18:56 |
| AJaeger | fungi, so a new {name}-server-docs job | 18:56 |
| *** tzn has quit IRC | 18:56 | |
| fungi | jeblair: thanks, yep clarkb's change is what implemented the current behavior we see | 18:57 |
| jeblair | clarkb: no i understand that -- what i'm saying is that it looks like we had no discussion..a AHA! | 18:57 |
| jeblair | i said "(If we start doing something sensible with tags on the main projects, this will need updating since we probably don't want a tag overwriting trunk docs, but that's not a concern we have to solve now.)" | 18:57 |
| clarkb | fungi: dougwig and I have a tail -f on syslog as I figure if anything happens that is catchable that will catch it | 18:57 |
| fungi | and wow, 3.5 years ago | 18:57 |
| jeblair | apparently, at the time, we were not doing anything sensible with tags on the server projects :) | 18:57 |
| fungi | pretty sure that predated pbr | 18:58 |
| *** infra-re_ has quit IRC | 18:58 | |
| jeblair | and since then, we started doing something "sensible" and we have run into the problem we anticipated :) | 18:58 |
| AJaeger | jeblair: ;) | 18:58 |
| dougwig | clarkb: awesome. | 18:58 |
| jeblair | clarkb: we kicked that can down the road a LONG TIME. | 18:58 |
| *** mrmartin has quit IRC | 18:58 | |
| clarkb | september 2012 practically another lifetime | 18:59 |
| fungi | if we're only just _now_ getting annoyed by it, then successful punt! | 18:59 |
| AJaeger | as I said before: It's self-healing ;) | 18:59 |
| *** amotoki has quit IRC | 18:59 | |
| fungi | usually when i punt on solving something thoroughly, it comes back to bite me the next week not 3.5 years later | 18:59 |
| *** kzaitsev_mb has joined #openstack-infra | 18:59 | |
| fungi | so anyway, yes i think the sane solution is two jobs. we can still have both use the same script with just a little logic tweak | 19:00 |
| Sam-I-Am | ^ this | 19:00 |
| *** jsavak has quit IRC | 19:00 | |
| jeblair | ++ | 19:00 |
| *** _nadya_ has quit IRC | 19:01 | |
| jeblair | so my next question is, why does https://review.openstack.org/changes/?q=13725 return [] ? | 19:01 |
| *** FallenPegasus has quit IRC | 19:01 | |
| zaro | anteaya, fungi : i added the info on cleaning up the registered users group. please look to see if it makes sense. | 19:01 |
| AJaeger | I can give it a try but like to trade in four reviews for project-config, please: 278051, 278061, 278641, 278627 | 19:01 |
| anteaya | zaro: thank you | 19:01 |
| * anteaya looks | 19:02 | |
| *** FallenPegasus has joined #openstack-infra | 19:02 | |
| clarkb | fungi: dougwig also my d-g change did not merge and after a first recheck it tripped on something there too :/ | 19:02 |
| clarkb | the gate fail was cinder related but it didn't look like disk space problems so I dunno | 19:03 |
| clarkb | rechecked again for more data gathering | 19:03 |
| dougwig | clarkb: yeah i saw the oslo messaging timeout error from nova, on a floating ip. just a few projects in the mix in that failure. :) | 19:03 |
| *** jsavak has joined #openstack-infra | 19:03 | |
| dims | dougwig : was it grenade? | 19:03 |
| anteaya | zaro: in step 2 of your added steps, can that patch merge prior to gerrit and puppet being stopped? | 19:03 |
| dougwig | dims: yes it was | 19:04 |
| *** harlowja has joined #openstack-infra | 19:04 | |
| *** sabeen3 has joined #openstack-infra | 19:04 | |
| dims | dougwig : got a url to the log? | 19:04 |
| fungi | zaro: 270537 only looks like it handles the database part of the work (cleaning up references in 4 tables). that helps, but what about the groups files? is it safe to ignore them or do we need to clean them up in all the affected repos? | 19:04 |
| dougwig | dims: http://logs.openstack.org/57/278557/1/check/gate-grenade-dsvm/f5f5546/ | 19:04 |
| *** arthus has quit IRC | 19:05 | |
| *** _nadya_ has joined #openstack-infra | 19:06 | |
| zaro | fungi: it's safe to ignore. i tested that. | 19:06 |
| *** infra-red has joined #openstack-infra | 19:06 | |
| zaro | fungi: but would be nice to cleanup :) | 19:06 |
| fungi | zaro: okay, so if there is a group mentioned in a groups file with a uuid that is not referenced in the database, gerrit just ignores it? | 19:06 |
| jeblair | zaro: many of the older system-config changes are not searchable in gerrit | 19:06 |
| zaro | fungi: yes. | 19:07 |
| jeblair | zaro: example: https://review.openstack.org/#/q/change:13725 | 19:07 |
| *** sabeen3 has quit IRC | 19:07 | |
| *** jsavak has quit IRC | 19:07 | |
| *** sabeen1 has quit IRC | 19:07 | |
| fungi | zaro: that's very simple then--thanks! i was worried this would get a lot messier | 19:07 |
| dims | dougwig : i'll take a quick peek will report back if i spot something | 19:07 |
| jeblair | zaro: or by it's change-id: https://review.openstack.org/#/q/Ia3cf07d860b10214ed54fcf9748330866cb47f5d,n,z | 19:07 |
| *** mknawabi has joined #openstack-infra | 19:07 | |
| *** infra-re_ has joined #openstack-infra | 19:08 | |
| *** yamahata has quit IRC | 19:08 | |
| jeblair | zaro: do you have any idea why or how we could fix that? | 19:08 |
| zaro | jeblair: do you happen to know how old? like from 2.2 or 2.4? | 19:09 |
| jeblair | zaro: it's worth noting, though i do not know if it's related, that was during the time where we were cherry-picking as a merge strategy | 19:09 |
| jeblair | zaro: i do not know the version (though changes to other repos from the same time period are searchable) | 19:09 |
| *** rhallisey has joined #openstack-infra | 19:10 | |
| *** infra-red has quit IRC | 19:11 | |
| jeblair | zaro: i think it was 2.4 | 19:11 |
| *** annegentle_ has quit IRC | 19:11 | |
| zaro | jeblair: how are you finding these non-searchable changes? | 19:11 |
| jeblair | zaro: git history | 19:11 |
| zaro | jeblair: does it work with direct url? | 19:11 |
| jeblair | zaro: yes | 19:11 |
| Shrews | mtreinish: that's unfortunate (re: shade test_results) and i've never noticed. patch would be welcomed | 19:12 |
| *** fawadkhaliq has quit IRC | 19:12 | |
| anteaya | it would be great to get another jeepyb +2 on this prior to tomorrow's gerrit downtime, it helps to clean up the errand 'registered users' group when never should have been created: https://review.openstack.org/#/c/248126/ | 19:12 |
| *** nelsnels_ has joined #openstack-infra | 19:12 | |
| zaro | jeblair: ok. i have no idea what's going on off the top of my head but will dig deeper to see. | 19:12 |
| anteaya | *which | 19:12 |
| jeblair | zaro: i happened upon 13725, then just started incrementing the number from there; 13726, 7, 8 all work; 13729 is another system-config change which does not... etc. | 19:13 |
| jeblair | (6,7,8 are not system-config changes) | 19:13 |
| *** akshai has quit IRC | 19:13 | |
| *** akshai has joined #openstack-infra | 19:13 | |
| *** infra-re_ has quit IRC | 19:14 | |
| *** nelsnelson has quit IRC | 19:15 | |
| *** annegentle_ has joined #openstack-infra | 19:18 | |
| *** moravec has joined #openstack-infra | 19:18 | |
| *** flepied has joined #openstack-infra | 19:22 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix tag publishing of server projects https://review.openstack.org/279238 | 19:22 |
| AJaeger | jeblair, clarkb, fungi: here's the docs change, please review carefully ^ | 19:22 |
| *** Sukhdev has joined #openstack-infra | 19:23 | |
| *** sabeen has joined #openstack-infra | 19:23 | |
| *** jsavak has joined #openstack-infra | 19:23 | |
| *** notnownikki has quit IRC | 19:24 | |
| *** jsavak has joined #openstack-infra | 19:24 | |
| *** sabeen3 has joined #openstack-infra | 19:25 | |
| *** boris-42 has joined #openstack-infra | 19:26 | |
| dims | anteaya : one last job for this round of running stuff against oslo.* master - https://review.openstack.org/#/c/279198/ | 19:26 |
| fungi | nibalizer: any reason for me not to approve 275477 (Add OmfraCloud to puppetmaster_clouds) yet? | 19:26 |
| *** electrofelix has quit IRC | 19:27 | |
| *** moravec has quit IRC | 19:27 | |
| nibalizer | only if we wanted to flatten auth | 19:27 |
| *** moravec has joined #openstack-infra | 19:28 | |
| *** sabeen has quit IRC | 19:28 | |
| anteaya | dims: this adds a job to devstack, please get mtreinish to review the patch | 19:28 |
| dims | ack thanks | 19:28 |
| fungi | anteaya: that change doesn't clean up anything, just prevents manage-projects from making new messes | 19:29 |
| anteaya | fungi: I beg your pardon | 19:29 |
| fungi | nibalizer: oh, right, that's why i held off. you said something last night about a new patchset to do that | 19:29 |
| anteaya | 248126 prevents manage-projects from making new messes | 19:30 |
| nibalizer | ya i didn't get around to that last night | 19:30 |
| nibalizer | I can do today | 19:30 |
| fungi | anteaya: it's the other change (270537) which has the cleanup script for the present mess | 19:30 |
| anteaya | fungi: very good | 19:30 |
| anteaya | I'm having such a hard time keeping anything straight in my head today | 19:30 |
| fungi | nibalizer: no worries, i think my memory is just going | 19:30 |
| anteaya | sorry to add to the confusion | 19:30 |
| fungi | anteaya: me too! | 19:30 |
| clarkb | dougwig: fungi of course as soon as I hold a node to debug the test decides to pass | 19:30 |
| fungi | hah | 19:31 |
| fungi | heisenbug | 19:31 |
| dougwig | clarkb: you just need to login to every rally job. | 19:31 |
| *** jsavak has quit IRC | 19:31 | |
| *** moravec1 has joined #openstack-infra | 19:32 | |
| *** jsavak has joined #openstack-infra | 19:32 | |
| *** markus_z has quit IRC | 19:32 | |
| *** peterlisak has quit IRC | 19:32 | |
| *** dkaigarodsev_ has quit IRC | 19:32 | |
| *** moravec has quit IRC | 19:32 | |
| *** annegentle_ has quit IRC | 19:33 | |
| clarkb | ya I will have to be more aggressive | 19:33 |
| mriedem | when we hit something like this, the job is just automatically restarted isn't it? http://logs.openstack.org/03/275003/6/check/gate-functional-dsvm-magnum-api/a9a0eb9//console.html.gz#_2016-02-05_02_56_33_343 | 19:33 |
| *** peterlisak has joined #openstack-infra | 19:33 | |
| clarkb | mriedem: yes | 19:34 |
| fungi | AJaeger: starting to look at 279238, what happens if a stable point release is tagged on a client while there's not yet a newer tag on master? | 19:34 |
| *** FallenPe_ has joined #openstack-infra | 19:34 | |
| *** dkaigarodsev has joined #openstack-infra | 19:34 | |
| mriedem | clarkb: ok, someone was trying to classify that in e-r but i don't see a point | 19:34 |
| fungi | AJaeger: i think in that case you'll end up with the master documentation replaced by the stable branch documentation? | 19:34 |
| clarkb | mriedem: I am trying to debug that with neutron rally tests. Theory is it is related to the sparse swapfile but my change to switch that to preallocated swapfile hasn't been able to merge yet | 19:34 |
| fungi | AJaeger: wait, nevermind. if there's no newer tag on master then there's no newer documentation already published for "master" | 19:35 |
| fungi | AJaeger: since those projects only run this job on release and not in post | 19:35 |
| clarkb | grr just missed one | 19:36 |
| *** FallenPegasus has quit IRC | 19:37 | |
| *** markusz has joined #openstack-infra | 19:38 | |
| crinkle | yolanda: i wrote up things i went through to get the non-ca certs to work, one of these might help https://etherpad.openstack.org/p/debug-infracloud-ssl | 19:38 |
| clarkb | dougwig: fungi ok watching two more now | 19:39 |
| *** annegentle_ has joined #openstack-infra | 19:39 | |
| *** shardy has quit IRC | 19:41 | |
| anteaya | zaro fungi I have incorporated your manage-projects clean up steps in the rename workflow on the etherpad | 19:42 |
| fungi | thanks anteaya | 19:42 |
| AJaeger | fungi: I'm not getting your problem. The extra parameter is only used for *server* repos, and there's always content published | 19:43 |
| clarkb | now two in a row passing on me :) | 19:44 |
| AJaeger | fungi: ah, you figured it out, I think... | 19:44 |
| *** ajmiller_ has joined #openstack-infra | 19:44 | |
| clarkb | neutron just needs people to stare at these tests to make them pass | 19:44 |
| clarkb | 100% effective so far | 19:44 |
| mtreinish | Shrews: sure, I'll push a fix for that in a bit | 19:44 |
| mtreinish | Shrews: also do you know why: https://review.openstack.org/278816 is failing the ansible job | 19:45 |
| *** ccrouch has joined #openstack-infra | 19:45 | |
| Shrews | mtreinish: yeah. ansible broke us | 19:45 |
| Shrews | mtreinish: fix put up for review | 19:46 |
| mtreinish | ah, ok | 19:46 |
| Shrews | though now i have to fix ansible | 19:46 |
| fungi | AJaeger: spotted one obvious error (see inline comment) but i'm also still thinking through the logic | 19:46 |
| AJaeger | fungi: thanks for spotting - but yes, please think through it properly... | 19:46 |
| *** ajmiller has quit IRC | 19:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix tag publishing of server projects https://review.openstack.org/279238 | 19:47 |
| *** tzn has joined #openstack-infra | 19:51 | |
| *** Sukhdev has quit IRC | 19:54 | |
| *** Sukhdev has joined #openstack-infra | 19:54 | |
| *** dimtruck is now known as zz_dimtruck | 19:55 | |
| *** zz_dimtruck is now known as dimtruck | 19:55 | |
| *** sridhar_ram1 has joined #openstack-infra | 19:55 | |
| *** tzn has quit IRC | 19:55 | |
| *** eliqiao_ has quit IRC | 19:56 | |
| *** amotoki has joined #openstack-infra | 19:56 | |
| *** sridhar_ram has quit IRC | 19:56 | |
| Sam-I-Am | AJaeger: ^^^ thanks! | 19:57 |
| *** Swami has quit IRC | 19:57 | |
| openstackgerrit | Volodymyr Samotiy proposed openstack-infra/project-config: Add openstack/broadview-lib project https://review.openstack.org/275873 | 19:57 |
| *** eliqiao_ has joined #openstack-infra | 19:58 | |
| openstackgerrit | Dmitriy Ukhlov proposed openstack-infra/project-config: Adds neutron tempest tests for oslo.messaging drivers https://review.openstack.org/279215 | 19:59 |
| *** ociuhandu has quit IRC | 19:59 | |
| *** amotoki has quit IRC | 20:00 | |
| *** fawadkhaliq has joined #openstack-infra | 20:02 | |
| *** baoli has quit IRC | 20:03 | |
| clarkb | https://jenkins03.openstack.org/job/gate-rally-dsvm-neutron-neutron/5008/console did fail but not due to the thing we are trying to catch | 20:04 |
| *** tzn has joined #openstack-infra | 20:05 | |
| *** ayoung has quit IRC | 20:06 | |
| *** aysyd has quit IRC | 20:06 | |
| *** redixin has joined #openstack-infra | 20:06 | |
| *** vgridnev has joined #openstack-infra | 20:07 | |
| clarkb | I have 4 in progress rally neutron neutron jobs I am watching now. eventually we should get one | 20:08 |
| *** esker has joined #openstack-infra | 20:09 | |
| *** austin81 has joined #openstack-infra | 20:11 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/shade: Prepare functional test subunit stream for collection https://review.openstack.org/279246 | 20:12 |
| *** nwkarsten has joined #openstack-infra | 20:12 | |
| *** ddieterly has quit IRC | 20:13 | |
| mtreinish | Shrews: ^^^ I'm not sure if that'll work or not though | 20:13 |
| *** hashar has joined #openstack-infra | 20:14 | |
| *** ldnunes has quit IRC | 20:14 | |
| clarkb | jenkins 05 is no longer running jenkins | 20:16 |
| clarkb | I am restarting the service now | 20:16 |
| *** esikachev has quit IRC | 20:16 | |
| anteaya | clarkb: thank you | 20:17 |
| samueldmq | pabelanger: looks like stackalytics.o.o has been fixed | 20:19 |
| fungi | heh. it wasn't much of a jenkins server i guess | 20:19 |
| samueldmq | pabelanger: thanks for looking at it | 20:19 |
| *** kushal has quit IRC | 20:19 | |
| *** sbalukoff has quit IRC | 20:19 | |
| openstackgerrit | Austin Clark proposed openstack-infra/project-config: Add javascript jobs to stackviz https://review.openstack.org/279254 | 20:20 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Make propose_translation_update more robust https://review.openstack.org/279255 | 20:20 |
| clarkb | something seems very broken though | 20:21 |
| clarkb | so much red on zuul status page | 20:21 |
| clarkb | http://logs.openstack.org/25/207625/48/check/gate-tempest-dsvm-neutron-full/1704525/logs/devstacklog.txt.gz#_2016-02-11_20_11_38_086 | 20:22 |
| clarkb | I guess I should stop trying to catch the break | 20:22 |
| mordred | clarkb: that does seem to be an issue | 20:22 |
| *** _dpaterson has joined #openstack-infra | 20:22 | |
| *** ayoung has joined #openstack-infra | 20:22 | |
| *** andreykurilin__ has quit IRC | 20:22 | |
| AJaeger | clarkb: and lots of jobs in post and check - and few in gate | 20:23 |
| clarkb | I guess it is a good time to get lunch | 20:23 |
| mtreinish | dims: ^^^ that seems like your territory | 20:23 |
| clarkb | and then maybe not lookt at a compute rfor a while | 20:23 |
| austin81 | timothyb89, mtreinish, clarkb, jeblair: https://review.openstack.org/279254 | 20:23 |
| *** rcernin has quit IRC | 20:24 | |
| *** annegentle_ has quit IRC | 20:26 | |
| flaper87 | Is there a way (or would it make sense) to not run CI jobs for patches that "just" fix docstrings/comments ? | 20:26 |
| flaper87 | Like, just run pep8 and move on | 20:26 |
| *** jlk has quit IRC | 20:26 | |
| clarkb | flaper87: not really since zuul doesn't introspect change diffs | 20:26 |
| *** annegentle_ has joined #openstack-infra | 20:26 | |
| flaper87 | What am I missing? :D | 20:26 |
| clarkb | well you have to know if a change only touches docstrings | 20:27 |
| *** jlk has joined #openstack-infra | 20:27 | |
| *** jlk has quit IRC | 20:27 | |
| *** jlk has joined #openstack-infra | 20:27 | |
| clarkb | that requires parsing the diff and the source to determine if the string is a dosctring | 20:27 |
| clarkb | mtreinish: dims I think everything is fialing on that now, I am digging in | 20:27 |
| clarkb | but yall should definitely too because ya | 20:27 |
| *** esikachev has joined #openstack-infra | 20:28 | |
| openstackgerrit | Austin Clark proposed openstack-infra/project-config: Add javascript jobs to stackviz https://review.openstack.org/279254 | 20:28 |
| *** slagle has quit IRC | 20:28 | |
| flaper87 | clarkb: right, if there's no way to do that now, would it make sense (or useful) to have it ? I'm happy to help with that, TBH. | 20:28 |
| clarkb | mtreinish: dims actually I may have conflated the no logger with the error when that is just output from the command, the failure is the traceback and is squarely in neutron-db-manage | 20:28 |
| clarkb | flaper87: my initial opinion is no, we shouldn't put significant effort into turning zuul into a python CI scheduler | 20:29 |
| *** ddieterly has joined #openstack-infra | 20:29 | |
| *** slagle has joined #openstack-infra | 20:29 | |
| clarkb | but maybe that is something that could be made pluggable in zuulv3 if you want to attach internal checks to short circuit a run | 20:30 |
| clarkb | I dunno | 20:30 |
| flaper87 | clarkb: roger. I just feel it's a waste of CI resources to have jobs running for docstring changes | 20:30 |
| AJaeger | flaper87: but how often does it happen? | 20:30 |
| pabelanger | samueldmq: np | 20:30 |
| flaper87 | I wonder if there's a generic enough way to do this. Generic as in multi-lingual. I'd assume there is | 20:30 |
| *** fawadkhaliq has quit IRC | 20:30 | |
| flaper87 | AJaeger: TBH, it doesn't matter how often it happens. It's still a waste and OpenStack is getting bigger. We should sum all the cases in every project using CI resources | 20:31 |
| anteaya | fungi: for the possible requirements patches for tomorrow's rename, specs would not be in requirements, nor would tests, and I don't think anything fuel related is in requirements | 20:31 |
| clarkb | flaper87: it actually isnt a waste if the results are still valuable | 20:31 |
| *** jtomasek has quit IRC | 20:31 | |
| flaper87 | Also, the frequency increases every time we're 2 or 3 months away from the summit ;) | 20:31 |
| clarkb | flaper87: and they definitely are from a trending standpoint | 20:31 |
| fungi | anteaya: yeah, that's just there as a double-check | 20:31 |
| clarkb | imo fixing bugs in our software is much more valuable | 20:31 |
| fungi | anteaya: it often won't be needed | 20:32 |
| clarkb | see the current db manage thing that is making every job fail | 20:32 |
| *** kzaitsev_mb has quit IRC | 20:32 | |
| anteaya | fungi: awesome, just double checking my addled thinking, I'll move on, thank you | 20:32 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for horizon connection refused test bug 1544619 https://review.openstack.org/279182 | 20:32 |
| openstack | bug 1544619 in OpenStack Dashboard (Horizon) "test_create_delete_instance race fails in gate-horizon-dsvm-integration with "error: [Errno 111] Connection refused"" [Undecided,Confirmed] https://launchpad.net/bugs/1544619 | 20:32 |
| flaper87 | clarkb: I'm volunteering to help here, not asking anyone to do it | 20:32 |
| flaper87 | just wondering if it makes sense from an infra/ci perspective | 20:32 |
| flaper87 | if you don't think it does, then fine | 20:32 |
| flaper87 | :) | 20:32 |
| fungi | flaper87: i have to agree with clarkb on this one, random openstack bugs cause far more collateral damage in wasted workers than changes which fix a docstring | 20:33 |
| *** Swami has joined #openstack-infra | 20:33 | |
| fungi | flaper87: also how many languages to you want to write a parser for to identify comment formats? | 20:33 |
| clarkb | flaper87: and I am sayin your volunteered help is better directed at improving our software | 20:33 |
| flaper87 | sure, not saying that's not true. there will always be bugs, fwiw. This is just a micro optimization in our process | 20:33 |
| fungi | also spelled premature optimization | 20:34 |
| clarkb | oh derp this is a change that is broken in a giant stack | 20:34 |
| flaper87 | ok, I get it. not useful | 20:34 |
| clarkb | so that giant stack is all going to fail but that explains why the gate isn't dying quicker | 20:34 |
| anteaya | fungi: I'm wrong sahara-scenario is in there | 20:34 |
| anteaya | I'll make a patch | 20:34 |
| openstackgerrit | Lingxian Kong proposed openstack-infra/project-config: Add distil jobs back https://review.openstack.org/278827 | 20:35 |
| clarkb | mtreinish: sorry for the alarm, I just started seeing many many neutron tests all fail the same way without realizing they are part of the same stack | 20:35 |
| *** |-paul-| has joined #openstack-infra | 20:35 | |
| mtreinish | clarkb: ah ok, no worries | 20:35 |
| *** dpaterson has quit IRC | 20:35 | |
| lane_kong | AJaeger: hi, i want to know if there is any guidance for the configurations in gate jobs | 20:36 |
| AJaeger | lane_kong: did you read the Infra Manual? http://docs.openstack.org/infra/manual/creators.html | 20:36 |
| fungi | flaper87: basically it would be useful at least to first do some math on the percentage of changes which only touch comments in the languages you're interested in doing this for, and then the average number of patchsets those types of changes incur (compared to our overall average). i'd wager it ends up being a reasonably small percentage as to be best ignored | 20:37 |
| AJaeger | lane_kong: for official projects, there's the PTI as well: http://governance.openstack.org/reference/cti/python_cti.html | 20:37 |
| AJaeger | lane_kong: for a project like distil: Start with one job and evolve ;) | 20:38 |
| flaper87 | fungi: fair enough. I'll do this and report back either way. You guys know the gate and CI usage way better than me for sure :) | 20:38 |
| lane_kong | AJaeger: ok, we wil | 20:38 |
| lane_kong | AJaeger: thanks! | 20:38 |
| fungi | flaper87: the relative patchset count is also almost certainly further amplified by people who make code-altering changes which are broken rechecking their change a few times before deciding they should actually look at the failing job logs | 20:39 |
| openstackgerrit | Anita Kuno proposed openstack/requirements: Change project name from sahara-scenario to sahara-tests https://review.openstack.org/279257 | 20:40 |
| anteaya | fungi: does that have to wait for the repo rename or can it go in anytime? | 20:40 |
| *** julim has quit IRC | 20:40 | |
| clarkb | dougwig: https://jenkins03.openstack.org/job/gate-rally-dsvm-neutron-neutron/5007/console that job looks to be hung and my ssh connection is not responsive | 20:40 |
| clarkb | dougwig: the last thing in syslog is some ovs-vsctl thing | 20:41 |
| dougwig | clarkb: ovs shouldn't mess up your ssh connection. kernel panic? or did eth0 get messed with somehow? | 20:41 |
| fungi | anteaya: needs to happen during or soon after the maintenance | 20:41 |
| clarkb | dougwig: I am going to check nova console-log shortly | 20:42 |
| anteaya | very good | 20:42 |
| anteaya | fungi: I'll workflow -1 it | 20:42 |
| clarkb | dougwig: that should tlel us if kernel panic or similar hapapned | 20:42 |
| fungi | anteaya: otherwise we just end up breaking the list before the maintenance rather than having it get broken after the maintenance | 20:42 |
| anteaya | fun | 20:43 |
| clarkb | dougwig: http://paste.openstack.org/show/486756/ this again | 20:44 |
| clarkb | which is what showed up in the magnum jobs | 20:44 |
| clarkb | so something is really making the kernel unhappy, if we can get that d-g change in we can rule out the swap sparse file | 20:44 |
| dougwig | clarkb: or really slow. how does memory pressure look? lots of swapping to a slow vm disk? | 20:46 |
| nibalizer | flaper87: we already have some stuff so that changes that only touch unit tests don't run tempest , or if you changed only docs under /doc | 20:46 |
| *** ybathia has joined #openstack-infra | 20:46 | |
| bkero | clarkb: Looks like memory or disk problems | 20:46 |
| nibalizer | anyways if someone did modify python code Id want the test to validate that its still python | 20:46 |
| clarkb | dougwig: no idea, I can't get on the box | 20:46 |
| clarkb | bkero: right thats the theory with the sparse swap file, you can get into a situation where you have no disk and no memory and bam you lose | 20:46 |
| bkero | jbd = journaling block device = kernel thread responsible for the maintenance of | 20:47 |
| dougwig | clarkb: that's an internap node, not ovh like last night. | 20:47 |
| clarkb | dougwig: yes but it also uses sparse swapfile | 20:47 |
| bkero | clarkb: Doesn't the linux kernel explicitly prevent you from using a sprarse swapfile? | 20:47 |
| clarkb | bkero: nope, you loopback mount it | 20:47 |
| bkero | So you have to loopback-mount to create a level of indirection | 20:47 |
| clarkb | yes | 20:47 |
| clarkb | I have a change up to stop doing that and fallocate a file instead | 20:47 |
| clarkb | but that has broken on other things | 20:48 |
| austin81 | greghaynes: I am in need of your nodepool expertise. Should I create an entirely new element for stackviz in /nodepool/elements/ ... or just add my npm/tar/gulp steps to an existing element (probably node-devstack?) | 20:48 |
| bkero | clarkb: Is disk space that tight that you can't just pre-allocate it? | 20:48 |
| bkero | Or is this ironic bring-up time? | 20:48 |
| fungi | dangerous in production obviously, but was a workaround for platforms where we have no second partition to use for swap and not enough space on the rootfs for some jobs to take up a lot of it with a preallocated swapfile | 20:48 |
| clarkb | bkero: that was the concern but it may be biting us hard here | 20:48 |
| bkero | Maybe ramzswap at that point? | 20:48 |
| clarkb | bkero: that likely won't help at all :) | 20:49 |
| clarkb | its jobs like magnum with 1.2GB of head room booting many 1gb ram or bigger VMs | 20:49 |
| bkero | Just trading CPU for memory :) | 20:49 |
| clarkb | (I don't recommend this but hey) | 20:49 |
| fungi | the theory was we have some jobs which need pretty much all of our smallest root filesystems, but other jobs which need swap on providers that don't give us a non-swapfile option, so sparse should allow each of them to use the system in the way they need. however this ignores the possibility of some jobs wanting to use all disk and swap | 20:49 |
| bkero | KSM? | 20:49 |
| clarkb | bkero: that is up to them to do | 20:50 |
| clarkb | but that would possibly be an option for them | 20:50 |
| bkero | So...if you need the sparsefile for bring-up time, and KVM isn't running as root, why not set your reserved blockcount on FS to match your swapfile size? | 20:50 |
| dougwig | preallocate should help. but vm disk swap; fine for page outs, but if a nova vm has to actively use it, man that job is doomed either way. | 20:50 |
| clarkb | bkero: or just preallocate... it is what we are doing | 20:50 |
| bkero | Yeah, that works if you can afford the time to preallocate | 20:51 |
| clarkb | bkero: its dirt cheap on ext4 | 20:51 |
| bkero | although depending on the filesystem that might be really quick | 20:51 |
| fungi | bkero: yeah, so much of these impacted jobs run as root anyway that reserved block count isn't useful | 20:51 |
| zaro | anteaya, fungi : here's my ranming projects test: https://etherpad.openstack.org/p/gerrit_project_rename_test | 20:51 |
| clarkb | * it is what we are trying to do | 20:52 |
| zaro | it didn't work right for me. so could you please verify that my testing is valid before we continue to debug? | 20:52 |
| clarkb | dougwig: they apparently had great success until we started adding these new clouds without swap in their flavors | 20:52 |
| fungi | i think probably the ultimate solution is to stop providing swap, but perhaps provide a hint as to the most effective place on a given system to add swap, and then let jobs which need to mkswap something at an appropriate size before they run their main payload | 20:52 |
| clarkb | dougwig: for rally I wonder if rally is using a larger flavor than tempest | 20:52 |
| clarkb | dougwig: since tempest uses a 64MB ram flavor iirc | 20:52 |
| *** daemontool_ has joined #openstack-infra | 20:53 | |
| *** e0ne has quit IRC | 20:53 | |
| clarkb | boris-42: any idea what flavor rally will use on the rally neutron jobs? | 20:53 |
| dougwig | clarkb: i know octavia uses 512, one at a time. let me check on rally | 20:53 |
| zaro | fungi, anteaya : it doesn't look like projects are saved to account_project_watches table anymore. | 20:54 |
| anteaya | zaro: thanks for testing, looking at your paste | 20:54 |
| anteaya | zaro: thats good we found out now, thank you | 20:54 |
| fungi | zaro: they only appeared in account_project_watches if people set their accounts to watch that project | 20:54 |
| anteaya | zaro: sorry it is an etherpad, looking at your etherpad | 20:54 |
| * dims reading scroll back | 20:54 | |
| clarkb | dougwig: fwiw https://jenkins03.openstack.org/job/gate-rally-dsvm-neutron-neutron/5010/console looks very similar to the internap one | 20:55 |
| zaro | fungi: ok. that's good. | 20:55 |
| clarkb | dougwig: so Ithink this is the same issue with different symptoms | 20:55 |
| fungi | zaro: but that looks like our process to me. is it possible the project name is now tracked in a second place besides the changes table? | 20:55 |
| dims | clarkb : ack on the "failure is the traceback and is squarely in neutron-db-manage" | 20:55 |
| *** daemontool has quit IRC | 20:55 | |
| zaro | fungi: i took a look but didn't find any other projects tables or columns in the db. | 20:55 |
| anteaya | zaro fungi well except the part where zaro manually created the project that it will be renamed to | 20:55 |
| *** Sukhdev has quit IRC | 20:56 | |
| clarkb | dims: you can ignore that | 20:56 |
| fungi | anteaya: oh! good eye | 20:56 |
| clarkb | dims: I didn't realize that all of the failing jobs belonged to a large related stack that was broken | 20:56 |
| anteaya | as our process doesn't have any existance of the project that has the new name prior to the process commencing | 20:56 |
| dims | clarkb : no worries :) | 20:56 |
| *** [1]Thelo has joined #openstack-infra | 20:56 | |
| zaro | anteaya: ahh, ok. i thought we actually created new project first? | 20:57 |
| anteaya | no | 20:57 |
| *** ociuhandu has joined #openstack-infra | 20:57 | |
| anteaya | the old project exists | 20:57 |
| fungi | zaro: yeah, for this test you should have only created yprojects/ytest1 and then after the process is complete yprojects/ytest1 will no longer exist and yprojects/ytest2 will be there instead | 20:57 |
| dougwig | clarkb: rally-neutron looks to be using the "tiny" flavor from devstack. | 20:57 |
| anteaya | the existing project is _renamed_ | 20:57 |
| anteaya | no project is created | 20:57 |
| clarkb | dougwig: it should use nano iirc | 20:57 |
| dims | anteaya : mtreinish has blessed https://review.openstack.org/#/c/279198/ | 20:58 |
| anteaya | and test2 should have exactly the same git log as test1 | 20:58 |
| zaro | anteaya, fungi : ok. i thought it was just moving changes from one project to another. | 20:58 |
| anteaya | no | 20:58 |
| clarkb | dougwig: so that may be a simple fix if that is why we have memory pressure there | 20:58 |
| zaro | i'll try without creating ytest2 first | 20:58 |
| *** Thelo has quit IRC | 20:58 | |
| *** [1]Thelo is now known as Thelo | 20:58 | |
| anteaya | it is taking one existing project with history and giving the repo a new name | 20:58 |
| anteaya | and maintaining the history | 20:58 |
| *** moravec1 has quit IRC | 20:59 | |
| anteaya | zaro: thank you for good clear notes | 20:59 |
| anteaya | zaro: glad we caught this now, thank you | 20:59 |
| *** baoli has joined #openstack-infra | 21:00 | |
| anteaya | dims: does jenkins like it yet? | 21:01 |
| anteaya | dims: jobs are running should finish in a few minutes | 21:02 |
| *** notmorgan is now known as morganfainberg | 21:02 | |
| therve | If someone can review https://review.openstack.org/#/c/279099/ that would unblock kilo requirements (and heat kilo gate) | 21:03 |
| therve | Thanks! | 21:03 |
| *** dkranz has quit IRC | 21:03 | |
| *** jsavak has quit IRC | 21:03 | |
| mordred | yay my patch was useful | 21:04 |
| mordred | clarkb: ^^ that requirements patch is the type of patch infra folks can do | 21:04 |
| *** jsavak has joined #openstack-infra | 21:04 | |
| mordred | as it relates to our afs mirrors | 21:04 |
| mordred | fungi: ^^ | 21:04 |
| dims | anteaya : that's just periodic, so we'll only know when the first run runs | 21:04 |
| openstackgerrit | Doug Wiegley proposed openstack-infra/project-config: Remove neutron grenade from lbaas jobs lists https://review.openstack.org/279266 | 21:05 |
| dims | anteaya : it's a cut-n-paste from the standard template with extra PROJECTS and LIBS_FROM_GIT, so fairly safe | 21:05 |
| *** Gordoni has joined #openstack-infra | 21:05 | |
| dims | and won't break anything else :) | 21:05 |
| *** jsavak has quit IRC | 21:06 | |
| *** jsavak has joined #openstack-infra | 21:06 | |
| fungi | mordred: indeed, no apt-get build-dep with our mirrors, unless we add deb-src lines | 21:06 |
| *** yamahata has joined #openstack-infra | 21:06 | |
| mordred | fungi: yup | 21:06 |
| fungi | er, rather, unless we start mirroring source packages _and_ add deb-src lines to the sources.list | 21:06 |
| * morganfainberg waits to see who will be the first to fall prey to mor<tab> | 21:06 | |
| mordred | fungi: TIL that build-dep requires deb-src lines | 21:06 |
| dougwig | clarkb: not finding the rally folks i know, so i filed a bug on the tiny/nano issue. | 21:07 |
| fungi | morganfainberg: you're not not morgan any more again? | 21:07 |
| *** ddieterly has quit IRC | 21:07 | |
| morganfainberg | fungi: only for a bit :P | 21:07 |
| olaph | now he's just plain not mordred | 21:07 |
| fungi | morganfainberg is indeed notmordred | 21:08 |
| * morganfainberg resists urge to make it even more confusing | 21:08 | |
| *** mfalatic has joined #openstack-infra | 21:08 | |
| anteaya | dims: I was waiting for the xml and config-layout job to finish | 21:09 |
| morganfainberg | ok. so.. i missed a ping earlier someone was maybe saying swiftclient things? | 21:09 |
| morganfainberg | it's out of my scrollback buffer... maybe clarkb ? | 21:09 |
| * morganfainberg is trying to be good at responding. | 21:10 | |
| dims | anteaya : thanks! | 21:10 |
| clarkb | was mordred | 21:10 |
| anteaya | dims: thank you | 21:10 |
| morganfainberg | clarkb: ooh | 21:11 |
| *** ihrachys has joined #openstack-infra | 21:11 | |
| morganfainberg | mordred: oh hai, about what you said here earlier when i was asleep...or getting teeth drilled | 21:11 |
| anteaya | morganfainberg: was this it? http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-02-11.log.html#t2016-02-11T16:36:13 | 21:11 |
| morganfainberg | oh that | 21:12 |
| morganfainberg | yes | 21:12 |
| morganfainberg | !! | 21:12 |
| openstack | morganfainberg: Error: "!" is not a valid command. | 21:12 |
| morganfainberg | that was it. | 21:12 |
| morganfainberg | anteaya: ok not important | 21:12 |
| anteaya | morganfainberg: awesome | 21:12 |
| morganfainberg | mordred: nvm. figured it out thanks to anteaya | 21:12 |
| *** morganfainberg is now known as notmorgan | 21:12 | |
| openstackgerrit | Merged openstack/os-testr: Enable testr run passthrough arguments https://review.openstack.org/267824 | 21:12 |
| *** _nadya_ has quit IRC | 21:12 | |
| anteaya | fungi zaro I'm offline in about 45 minutes for about an hour or so, will pick up when I get back, depending on what zaro's next round of testing reveals | 21:13 |
| anteaya | fungi: I'm at the point of generating the db commands but will hold off until I hear from zaro | 21:14 |
| *** kzaitsev_mb has joined #openstack-infra | 21:14 | |
| zaro | anteaya, fungi : ok. revisted testing worked like a charm. i have updated the etherpad for reference. so looks like we are good. | 21:15 |
| openstackgerrit | Merged openstack-infra/project-config: Periodic Devstack job against oslo.* from master https://review.openstack.org/279198 | 21:17 |
| *** kgiusti has left #openstack-infra | 21:19 | |
| fungi | anteaya: zaro: thanks for all the planning and testing/validation | 21:21 |
| anteaya | zaro: thank you! | 21:23 |
| *** flip214 has quit IRC | 21:23 | |
| *** esikachev has quit IRC | 21:23 | |
| *** esker has quit IRC | 21:23 | |
| *** flip214 has joined #openstack-infra | 21:24 | |
| *** flip214 has quit IRC | 21:24 | |
| *** flip214 has joined #openstack-infra | 21:24 | |
| *** ybathia has quit IRC | 21:24 | |
| mriedem | is it possible to disable people from voting on changes? https://review.openstack.org/#/c/278521/ -- apparently people can't vote on that | 21:26 |
| *** ajmiller_ is now known as ajmiller | 21:26 | |
| *** ihrachys has quit IRC | 21:26 | |
| *** esker has joined #openstack-infra | 21:26 | |
| *** ihrachys has joined #openstack-infra | 21:27 | |
| yolanda | hi crinkle, so i read your notes . The update of the certs, with update-ca-certificates is running fine for me, i'm thinking that the problems is the self-signed cert missing some setting, to act as CA | 21:27 |
| fungi | mriedem: we're fixing that tomorrow during a gerrit maintenance | 21:27 |
| yolanda | even if i do a verify cert using the same cert as path, fails | 21:28 |
| fungi | yolanda: there's a constraint you can set. let me find it | 21:28 |
| yolanda | fungi some flag in openssl.cnf? | 21:28 |
| mriedem | fungi: ah ok | 21:28 |
| fungi | yolanda: yeah, you need to set a basic constraint to allow signing, which is how a cert is identified as an authority by some software | 21:29 |
| yolanda | fungi i think the problem may be around these flags yes | 21:29 |
| *** dprince has quit IRC | 21:31 | |
| fungi | yolanda: the corresponding rfc section is https://tools.ietf.org/html/rfc5280#section-4.2.1.9 | 21:31 |
| *** esker has quit IRC | 21:31 | |
| yolanda | mm | 21:32 |
| yolanda | v3_req | 21:32 |
| yolanda | basicConstraints = CA:FALSE | 21:32 |
| yolanda | looks as it may be that | 21:32 |
| yolanda | fungi, key usage looks fine keyUsage = nonRepudiation, digitalSignature, keyEncipherment | 21:32 |
| fungi | right, i think you may need to flip that to true. still digging in openssl docs | 21:33 |
| yolanda | but i bet ca needs to be true | 21:33 |
| yolanda | going to try that | 21:33 |
| crinkle | yolanda: I didn't need to do anything special in openssl.cnf to generate self-signed certs | 21:36 |
| *** _amrith_ is now known as amrith | 21:37 | |
| yolanda | but crinkle you are not using pure self-signed certs, but a root ca, and intermediate cert right? | 21:37 |
| zaro | mtreinish: yes, you can deny groups of from voting. it's done with gerrit acls. | 21:37 |
| yolanda | i was following same tutorial as you i think | 21:37 |
| crinkle | yolanda: no I've reproduced your issue using just the self-signed certs | 21:37 |
| mtreinish | zaro: ? | 21:37 |
| mtreinish | zaro: oh, you meant mriedem, nm | 21:37 |
| crinkle | yolanda: without the CA | 21:37 |
| clarkb | hrm not getting any of the stuff done today that I wanted to get done, but fixing gate is important | 21:38 |
| clarkb | nibalizer: when is a good time to do mirror stuff? | 21:38 |
| *** MarkAtwood has joined #openstack-infra | 21:39 | |
| yolanda | crinkle if you have self-signed certs working in openstack, can you create one with alt names for east and west and share? if that works we could use that in our hiera, and test on east cloud | 21:39 |
| clarkb | grrrr https://jenkins03.openstack.org/job/gate-grenade-dsvm-multinode/2609/console | 21:39 |
| yolanda | i may have something wrong on my local openssl.cnf, don't know | 21:40 |
| clarkb | cacti says we are well below bw thresholds http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=862&rra_id=all | 21:40 |
| crinkle | yolanda: sure | 21:40 |
| *** jsavak has quit IRC | 21:41 | |
| clarkb | bhs is this side of the atlantic at least | 21:41 |
| yolanda | crinkle, fungi, it works now | 21:41 |
| yolanda | it was the flag | 21:41 |
| crinkle | yolanda: oh yay | 21:42 |
| yolanda | i still have the funny neutron bug, but that's something known, and i need to test a patch that degorenko created | 21:42 |
| yolanda | crinkle, mine contain the alt names, do you want to test? | 21:42 |
| openstackgerrit | Nate Johnston proposed openstack-infra/irc-meetings: Adjust Ops Guide Specialty Team meeting https://review.openstack.org/279276 | 21:42 |
| clarkb | I guess I should check haproxy logs for thta IP | 21:42 |
| crinkle | yolanda: sure | 21:42 |
| yolanda | going to paste you in private | 21:43 |
| yolanda | as you need the key | 21:43 |
| nibalizer | clarkb: um | 21:43 |
| clarkb | grep 158.69.65.86 /var/log/haproxy.log returns nothing | 21:43 |
| nibalizer | clarkb: i haave an appointment until 3 | 21:43 |
| nibalizer | you should maybe just go without me | 21:43 |
| clarkb | nibalizer: no rush I am pulling on the tests are failing thread and being distrated | 21:43 |
| nibalizer | i still haven't had time to redo the auth | 21:43 |
| fungi | yolanda: awesome work! | 21:44 |
| clarkb | fungi: that implies packets never reached git.o.o | 21:44 |
| clarkb | fungi: which means it could be anything between git.o.o and that ip ? | 21:44 |
| clarkb | firewall, asymettric route, cosmic rays | 21:44 |
| yolanda | fungi, thanks! i hope we have it 100% functional tomorrow, with the neutron fix | 21:44 |
| yolanda | crinkle, we may need to update the puppet code, to don't use chain file in apache, and add the certs to global trust | 21:45 |
| crinkle | yolanda: i'm working on that now | 21:45 |
| clarkb | I will try running some tests from bhs1 to git.o.o | 21:45 |
| yolanda | also we can set only one cert for all components, as they are wildcard certs | 21:45 |
| fungi | clarkb: yeah, no ipv6 on that worker, so would have to be something on the network between them dropping packets | 21:45 |
| *** moravec has joined #openstack-infra | 21:46 | |
| *** esker has joined #openstack-infra | 21:46 | |
| clarkb | motd on bhs1 mirror is fun | 21:46 |
| fungi | clarkb: back in my isp days, i was frequently plagued by too-small flow tables and state tables on various sorts of routing devices | 21:47 |
| fungi | which would cause exactly these sorts of symptoms | 21:47 |
| *** |-paul-| has quit IRC | 21:47 | |
| nibalizer | clarkb: ok | 21:47 |
| *** jsavak has joined #openstack-infra | 21:48 | |
| openstackgerrit | John L. Villalovos proposed openstack-infra/devstack-gate: Set PS4 to provide additional debug info https://review.openstack.org/271434 | 21:48 |
| *** alivigni has quit IRC | 21:48 | |
| *** jsavak has quit IRC | 21:48 | |
| yolanda | leaving for today, thanks for help with the certs! | 21:49 |
| clarkb | https://jenkins02.openstack.org/job/gate-tempest-dsvm-neutron-large-ops/59032/console is another case of this | 21:49 |
| fungi | yolanda: thanks for working on it! | 21:49 |
| clarkb | I should run a logstash queruy to see how bad it is | 21:49 |
| *** jsavak has joined #openstack-infra | 21:49 | |
| fungi | also ovh-bhs1 | 21:50 |
| fungi | wonder if they have a routing issue? except git.o.o and jenkins*.o.o are all in the same rax region | 21:50 |
| *** bpokorny has quit IRC | 21:51 | |
| craige | o/ | 21:51 |
| fungi | though entirely different netblocks so i guess could be a problem with a particular prefix in bgp | 21:51 |
| clarkb | there are ~22 instance in the last week all from today | 21:51 |
| clarkb | and ovh-gra1 has some occureances as well | 21:51 |
| fungi | but all in ovh? | 21:52 |
| *** eharney has quit IRC | 21:52 | |
| clarkb | yes | 21:52 |
| *** ociuhandu has quit IRC | 21:52 | |
| fungi | we don't have any ovh'ers hanging out in channel i guess | 21:52 |
| *** dpaterson has joined #openstack-infra | 21:52 | |
| *** ayoung has quit IRC | 21:52 | |
| *** weshay has quit IRC | 21:53 | |
| *** ociuhandu has joined #openstack-infra | 21:53 | |
| clarkb | mtr is unhappy about a level3 hop in dallas | 21:53 |
| clarkb | from ovh bhs mirror | 21:53 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Install docker for tests https://review.openstack.org/278021 | 21:53 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Fail functests if refusing to run tests https://review.openstack.org/177002 | 21:53 |
| clarkb | but no loss behind it so that may just be a device that doesn't like us | 21:54 |
| fungi | yeah, it may just throttle/filter icmp ttl exceeded messages | 21:54 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Update SSL configuration https://review.openstack.org/279286 | 21:54 |
| fungi | or it may be filtering udp to its loopback (does mtr use udp by default like traditional traceroute?) | 21:55 |
| fungi | er, no, not to its loopback | 21:55 |
| dims | fungi : clarkb : a zuul irc reporter is easy to do? (wanted to see if the oslo-master periodic jobs can report back on channel) | 21:55 |
| fungi | its loopback may not be routable | 21:55 |
| fungi | from the internet | 21:55 |
| greghaynes | ianw: http://logs.openstack.org/39/204639/25/check/gate-dib-dsvm-functests-devstack-centos7/2adec24/console.html#_2016-02-09_20_28_07_892 that is a really wierd fail... | 21:55 |
| fungi | common cause | 21:56 |
| greghaynes | ianw: yum related so wondering if you might have any insigt | 21:56 |
| fungi | if you're getting 100% loss on an intermediate hop | 21:56 |
| clarkb | dims: shouldn't be too hard, reporters are pretty pluggable and self contained | 21:56 |
| *** daemontool_ has quit IRC | 21:56 | |
| *** amotoki has joined #openstack-infra | 21:56 | |
| dims | clarkb : hmmm :) | 21:57 |
| clarkb | dims: we do already have an email reporter which we use to report periodic job results | 21:57 |
| *** moravec has quit IRC | 21:57 | |
| clarkb | dims: not sure if that is useulf to you though | 21:57 |
| *** rcarrillocruz has joined #openstack-infra | 21:57 | |
| *** rcarrillocruz has left #openstack-infra | 21:58 | |
| dims | clarkb : right i saw that, may be better to start with that | 21:58 |
| clarkb | fungi: 98.8% so ya it likely isn't a real problem there | 21:58 |
| clarkb | fungi: no loss behind it | 21:58 |
| *** nwkarsten has quit IRC | 21:58 | |
| *** sdake has quit IRC | 21:58 | |
| *** nwkarsten has joined #openstack-infra | 21:59 | |
| greghaynes | Is there somewhere I can look to figure out the URI's for the new apt-mirrors? | 21:59 |
| greghaynes | I am hoping to switch DIB over to using them.. | 22:00 |
| mtreinish | clarkb: we've actually turned off the email reporting for a bunch of the periodic jobs recently | 22:00 |
| *** sridhar_ram1 is now known as sridhar_ram | 22:00 | |
| *** ayoung has joined #openstack-infra | 22:00 | |
| clarkb | greghaynes: mirror.providername.regionname.openstack.org/apt/ I think | 22:01 |
| greghaynes | clarkb: ah, cool | 22:01 |
| *** amotoki has quit IRC | 22:01 | |
| *** sdake has joined #openstack-infra | 22:01 | |
| clarkb | greghaynes: er s/apt/ubuntu/ | 22:01 |
| clarkb | because we can't colocate ubuntu and debian | 22:01 |
| austin81 | mtreinish: fixed https://review.openstack.org/#/c/279254/ | 22:01 |
| greghaynes | clarkb: successs: http://mirror.ord.rax.openstack.org/ | 22:01 |
| clarkb | greghaynes: http://mirror.dfw.rax.openstack.org/ubuntu/ for example | 22:01 |
| clarkb | greghaynes: and the region/provider info is in /etc/nodepool on the test slaves | 22:02 |
| greghaynes | perfect | 22:02 |
| *** esker has quit IRC | 22:02 | |
| *** rossella_s has quit IRC | 22:03 | |
| *** chlong has quit IRC | 22:03 | |
| *** nwkarsten has quit IRC | 22:03 | |
| *** burgerk has quit IRC | 22:03 | |
| *** esker has joined #openstack-infra | 22:03 | |
| *** rossella_s has joined #openstack-infra | 22:03 | |
| fungi | greghaynes: switch dib over how? | 22:03 |
| greghaynes | fungi: in our CI to use those mirrors when we build images | 22:04 |
| greghaynes | (we build a bunch of images as part of our gate) | 22:04 |
| fungi | greghaynes: there was already a dib-related bug reported earlier today where an element was copying the sources.list from the host but not the apt.conf to go along with it | 22:04 |
| fungi | so i guess try not to make that same mistake | 22:04 |
| greghaynes | fungi: oh, that sounds like something we would do | 22:05 |
| fungi | our mirrors are not signed, so you need the apt configuration we're using to disable signing | 22:05 |
| SlickNik | fungi: do you know where the apt-conf is on the host node? | 22:05 |
| greghaynes | fungi: ah! makes sense | 22:05 |
| fungi | SlickNik: in /etc/apt | 22:05 |
| clarkb | SlickNik: /etc/apt/ | 22:05 |
| *** ajmiller_ has joined #openstack-infra | 22:05 | |
| fungi | greghaynes: er, to disable signature verification i mean | 22:06 |
| jhesketh | Morning | 22:06 |
| clarkb | jhesketh: good day | 22:06 |
| fungi | SlickNik: greghaynes: specifically, we're using run-parts type snippets in the *.d directories under /etc/apt | 22:06 |
| clarkb | fungi: I am thinking the only thing we can really do here is collect VM uuids and IP addrs then apss that along to ovh? | 22:07 |
| *** _dpaterson has quit IRC | 22:07 | |
| clarkb | fungi: I will work on collecting that info unless you think there is something more direct we can do | 22:07 |
| greghaynes | fungi: cool, thanks | 22:07 |
| jhesketh | clarkb: "I said, good day!" | 22:07 |
| fungi | clarkb: likely. if we see a pattern on some netblocks but not others that might be useful for them | 22:07 |
| fungi | clarkb: though also possible it's just certain nova hosts which are impacted and not others | 22:08 |
| *** Sukhdev has joined #openstack-infra | 22:08 | |
| fungi | in which case i wouldn't expect any pattern in the ip addresses | 22:08 |
| *** ajmiller has quit IRC | 22:08 | |
| ianw | greghaynes: checking... | 22:08 |
| *** tiswanso has quit IRC | 22:10 | |
| *** thorst_ has quit IRC | 22:11 | |
| crinkle | yolanda: your cert doesn't work for me, I see CN=controller00.hpuseast.ic.openstack.org instead of the wildcard | 22:11 |
| *** thorst_ has joined #openstack-infra | 22:11 | |
| *** dimtruck is now known as zz_dimtruck | 22:11 | |
| *** nwkarsten has joined #openstack-infra | 22:11 | |
| SlickNik | clarkb / fungi: strange we're using the apt-conf element from dib to override the apt-conf while image-building, but it seems to fail: http://paste.openstack.org/raw/486764/ | 22:14 |
| *** ajmiller_ is now known as ajmiller | 22:14 | |
| SlickNik | Full output at http://logs.openstack.org/20/279220/1/check/gate-trove-functional-dsvm-mysql/67a8df0/console.html if it's helpful at all | 22:14 |
| mtreinish | dims: what's wrong with using openstack-health to track the job you're adding? | 22:14 |
| ianw | greghaynes: hmm, so i'm not sure about the error, but yeah yum actually got hung up | 22:15 |
| mtreinish | dims: because if there's a gap there I'd like to figure it out so we can fix it | 22:15 |
| greghaynes | ianw: seems like another potentially bad package somewhere in the mix | 22:15 |
| ianw | greghaynes: firstly, i don't think it's worth our time debugging the centos6 builds, because they're just not used | 22:15 |
| mtreinish | that's why I stopped the email reporting on periodic-qa | 22:15 |
| SlickNik | Ah, maybe I think we might need to copy the file to the chroot before we can use it as an override? | 22:15 |
| fungi | SlickNik: i wonder if it doesn't handle /etc/apt/apt.conf.d/* files? | 22:15 |
| ianw | greghaynes: we should switch it to 7 | 22:15 |
| greghaynes | ianw: I think that patch is to switch on centos 7 | 22:16 |
| greghaynes | ianw: so, yes :) | 22:16 |
| mtreinish | pleia2: that reminds me want to help with locking up the openstack-qa list on mailman? | 22:16 |
| greghaynes | ianw: we can remove the 6 tests if we want | 22:16 |
| *** thorst_ has quit IRC | 22:16 | |
| ianw | greghaynes: it's building *on* 7, but testing building 6 | 22:16 |
| pleia2 | mtreinish: yep! | 22:16 |
| pleia2 | mtreinish: did you track down the list admins? | 22:16 |
| *** tpsilva has quit IRC | 22:16 | |
| *** araji has joined #openstack-infra | 22:17 | |
| ianw | greghaynes: yeah, i want us to functional test centos-minimal and fedora-minimal latest | 22:17 |
| SlickNik | will look into it some more — thanks for confirming that the file is indeed at /etc/apt/apt.conf. | 22:17 |
| ianw | greghaynes: we are *so* close! | 22:17 |
| greghaynes | ianw: Centos is just odd here because there are separate elements for 6 and 7, so we have to explicitly turn them on/off | 22:17 |
| pleia2 | mtreinish: as discussed, we'll just shut it down administratively through the UI instead of deleting on the infra side | 22:17 |
| *** esker has quit IRC | 22:17 | |
| mtreinish | pleia2: well it's sdague, jaypipes, and dkranz | 22:17 |
| mtreinish | pleia2: yeah, that's fine | 22:18 |
| ianw | greghaynes: so this is why i wrote -> https://review.openstack.org/#/c/276563/ | 22:18 |
| pleia2 | mtreinish: just have one of them give you the admin password and I can tell you what to change | 22:18 |
| ianw | greghaynes: it gives you a nice interface to the functional tests, and we can change defaults for ci in a list | 22:18 |
| greghaynes | ianw: cool, I'll have a look | 22:18 |
| *** annegentle_ has quit IRC | 22:18 | |
| greghaynes | ianw: perfect | 22:18 |
| mtreinish | pleia2: sdague probably has called it a day already, dkranz isn't on irc, and dunno if jaypipes is still online | 22:19 |
| *** annegentle_ has joined #openstack-infra | 22:19 | |
| ianw | greghaynes: i saw your change to install docker / do those image output tests ... i think that's fine but i think we can run it separately | 22:19 |
| *** vgridnev has quit IRC | 22:19 | |
| *** ganso has quit IRC | 22:19 | |
| mtreinish | pleia2: I also doubt any of them remember the admin password, we haven't used that list for anything real in at least a couple years | 22:20 |
| *** zz_dimtruck is now known as dimtruck | 22:20 | |
| pleia2 | mtreinish: ah, I doubt I can retrieve the password, but I can probably set a new one and give that to you, I'll look into it now | 22:21 |
| clarkb | fungi: http://paste.openstack.org/show/486766/ is what I have so far | 22:21 |
| clarkb | if someone is able to grep haproxy logs to see if any of those show up that would be awesome | 22:21 |
| clarkb | I need t ofind the remaining IPs and their uuids | 22:21 |
| *** amitgandhinz has quit IRC | 22:21 | |
| fungi | SlickNik: /etc/apt/apt.conf.d/* actually, i think | 22:22 |
| *** dims has quit IRC | 22:23 | |
| mtreinish | pleia2: ok, thanks | 22:23 |
| *** hdd has quit IRC | 22:23 | |
| fungi | clarkb: will check them in just a sec | 22:23 |
| *** annegentle_ has quit IRC | 22:23 | |
| clarkb | fungi: there is also at least one suspect net block | 22:23 |
| *** araji has quit IRC | 22:24 | |
| *** markvan_ has quit IRC | 22:24 | |
| *** markvan has joined #openstack-infra | 22:25 | |
| *** markvan has quit IRC | 22:25 | |
| *** Swami has quit IRC | 22:26 | |
| *** dims has joined #openstack-infra | 22:26 | |
| jeblair | mtreinish, dims: ++ using the dashboard for that | 22:27 |
| *** markvan has joined #openstack-infra | 22:27 | |
| dims | mtreinish : +1 to openstack-health from me as well, currently the jobs just don't show up because of the restriction you mentioned mtreinish (https://etherpad.openstack.org/p/dims-periodic-jobs) | 22:27 |
| *** pradk has quit IRC | 22:28 | |
| dims | mtreinish : only additional thing that would be useful is to be able to specify the list of specific jobs somehow to show up on one page (if we don't have that already) | 22:29 |
| dims | mtreinish : jeblair : the reporter seems useful as a push mechanism to get in front of folks (just like the elastic recheck bot) | 22:30 |
| mtreinish | dims: well right now it does metadata grouping | 22:31 |
| mtreinish | so you get: http://status.openstack.org/openstack-health/#/g/build_queue/periodic?groupKey=build_queue&resolutionKey=hour | 22:31 |
| mtreinish | which is all the periodic jobs | 22:31 |
| *** thorst_ has joined #openstack-infra | 22:31 | |
| mtreinish | jeblair: is working on adding arbitrary tags to jobs so we can build custom groups | 22:32 |
| jeblair | dims: we're adding job tags to zuul so you'll be able to group them in the zuul config and health will be able to group on that | 22:32 |
| *** annegentle_ has joined #openstack-infra | 22:32 | |
| jeblair | https://review.openstack.org/#/c/277981/ | 22:32 |
| jeblair | job tags patch ^ | 22:32 |
| clarkb | fungi: http://paste.openstack.org/show/486770/ that is a complete list | 22:33 |
| *** thorst_ has quit IRC | 22:33 | |
| clarkb | 158.69.64.0/24 seems very suspect | 22:33 |
| *** thorst_ has joined #openstack-infra | 22:33 | |
| dims | jeblair : ah nice | 22:33 |
| openstackgerrit | Sean M. Collins proposed openstack-infra/devstack-gate: [WIP] Switch to new Neutron devstack https://review.openstack.org/278415 | 22:33 |
| dims | mtreinish : y i need just the oslo.* master jobs, hopefully we can do that with the zuul tags | 22:34 |
| clarkb | grep says we have used more IPs in that range that presumeably did not exhibit this issue | 22:34 |
| *** daemontool has joined #openstack-infra | 22:34 | |
| jeblair | dims: i like the irc reporter in principle, but i also don't want that to cause us to want to make lots of little pipelines so that certain jobs get reported to certain channels, etc (which is what prompted the job tags thing but for email) | 22:34 |
| *** mfalatic has quit IRC | 22:34 | |
| jeblair | dims: if we did do an irc reporter, we'd probably want to do something where we specified which channels were used in the jobs | 22:35 |
| clarkb | jeblair: fyi http://paste.openstack.org/show/486770/ are ovh nodes that exhibited trouble connecting to git.o.o | 22:35 |
| dims | jeblair : right. | 22:35 |
| jeblair | dims: but at any rate, health is probably better for this specific use case | 22:35 |
| jeblair | clarkb: thanks; do you have an email to send that to? | 22:36 |
| jeblair | (i'm wondering if we should record contacts in our secrets file too...) | 22:36 |
| clarkb | jeblair: I do, but still in the collecting data phase | 22:36 |
| jeblair | clarkb: cool | 22:37 |
| dims | jeblair : agree. if we can get past the "subunit2sql data pruning" problem mtreinish mentioned | 22:37 |
| anteaya | zaro: can you check what I have on https://etherpad.openstack.org/p/repo-renaming-2016-02-12 to incorporate the cleanup registered users work? can we get rid of that section on the end now? | 22:37 |
| clarkb | haproxy says the first one on that list did a cD which I think is client side disconnect? | 22:37 |
| fungi | clarkb: i found these hits http://paste.openstack.org/show/486771/ | 22:37 |
| *** ayoung has quit IRC | 22:37 | |
| mtreinish | dims: I just need to figure out why the cron job I added doesn't do anything | 22:37 |
| fungi | clarkb: those are the addresses which actually show up in the current haproxy log and how many times | 22:37 |
| dims | mtreinish : does it run? :) | 22:37 |
| *** dtardivel has quit IRC | 22:38 | |
| anteaya | SergeyLukjanov: can you look over https://etherpad.openstack.org/p/repo-renaming-2016-02-12 in preperation for the rename tomorrow, please? | 22:38 |
| fungi | clarkb: start time on the file is Feb 11 03:18:01 so presumably all the entries in your list are since then? | 22:38 |
| mtreinish | dims: well fungi and I were looking at it a while ago and IIRC there wasn't any indication one way or the other | 22:38 |
| mtreinish | dims: I've got: https://review.openstack.org/273141 up to try and add more verbosity when it's doing something | 22:38 |
| dims | mtreinish and where do the logs go? | 22:39 |
| clarkb | fungi: 0802 for the first one | 22:39 |
| clarkb | fungi: so yup | 22:39 |
| mtreinish | dims: on logstash.o.o | 22:39 |
| mtreinish | dims: so infra-root needs to check for us :) | 22:39 |
| *** thorst_ has quit IRC | 22:39 | |
| jeblair | mtreinish: what do i need to check? | 22:40 |
| clarkb | fungi: I will add the timestamps for each occurance to the list as well as the haproxy lines for each one that hit | 22:40 |
| clarkb | fungi: that should be a fairly complete set of data to hand to ovh | 22:40 |
| jeblair | mumble mumble local afs git mirror mumble | 22:41 |
| mtreinish | jeblair: if this cron job is working: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/manifests/server.pp#n64 | 22:41 |
| dims | mtreinish : ah. ok | 22:41 |
| mtreinish | the log file is /var/log/subunit2sql_migration.log | 22:41 |
| jeblair | subunit2sql-db-manage expire: error: argument --expire-age/-e: expected one argument | 22:41 |
| jeblair | 0 3 * * 7 subunit2sql-db-manage --config-file /etc/subunit2sql.conf expire --expire-age $expire_age >> /var/log/subunit2sql_migration.log 2>&1 & | 22:42 |
| jeblair | mtreinish: i think maybe change ' to " in puppet? | 22:42 |
| *** salv-orlando has joined #openstack-infra | 22:42 | |
| mtreinish | jeblair: that probably would do it :) | 22:42 |
| fungi | i wonder what the vos release time would be like if we were calling it on every gerrit ref-updated event | 22:43 |
| *** thorst_ has joined #openstack-infra | 22:43 | |
| jeblair | fungi: well, for that we could just use a rw volume; though then we lose fault tolerance | 22:43 |
| fungi | also warming the cache in our nodepool images would probably be the biggest challenge | 22:43 |
| jeblair | good thing that was just a mumble and not a real suggestion :) | 22:43 |
| *** salv-orl_ has quit IRC | 22:44 | |
| *** redixin has quit IRC | 22:45 | |
| openstackgerrit | Tim Burke proposed openstack/requirements: Update flake8 constraints https://review.openstack.org/279296 | 22:47 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Run all flake8 checks https://review.openstack.org/276500 | 22:47 |
| anteaya | fungi: also when you get a minute can you give https://etherpad.openstack.org/p/repo-renaming-2016-02-12 a once over, I filled in the commands | 22:47 |
| *** dingyichen has joined #openstack-infra | 22:47 | |
| *** thorst_ has quit IRC | 22:48 | |
| anteaya | I won't be around for a good chunk of tomorrow as I have to clean my house since I'll be traveling for 2 weeks | 22:48 |
| fungi | jeblair: yep. and musings on my part, not actual design ;) | 22:48 |
| anteaya | but I will be here for the renaming | 22:48 |
| jeblair | mtreinish: are you going to fix the '? | 22:48 |
| jeblair | or should i? | 22:48 |
| anteaya | just want to ensure my prep is accurate, if we have time tonight to do so | 22:48 |
| mtreinish | jeblair: I'm writing the commit msg right now | 22:48 |
| jeblair | mtreinish: cool | 22:49 |
| zaro | anteaya: etherpad LGTM and yes, you can get rid of section at the end. | 22:49 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Fix the expiration cron job https://review.openstack.org/279297 | 22:49 |
| mtreinish | jeblair: ^^^ | 22:49 |
| anteaya | zaro: thank you | 22:50 |
| anteaya | fungi: you made a spelling correction in the fuel plugin one, what did I have there? | 22:51 |
| anteaya | oh I didn't have anything at all | 22:51 |
| anteaya | d'oh | 22:51 |
| *** jsavak has quit IRC | 22:52 | |
| *** jpr has quit IRC | 22:52 | |
| anteaya | sorry about that, and thanks for fixing it | 22:53 |
| fungi | anteaya: yeah, missing openstack/ prefix throughout for that repo | 22:53 |
| clarkb | fungi: jeblair http://paste.openstack.org/show/486774/ how does that look? any other information that you think we should add? | 22:53 |
| *** mriedem has quit IRC | 22:53 | |
| anteaya | fungi: yeah, sorry, looks like you caught them all though, thank you | 22:53 |
| clarkb | note there have been 8 more since I started collecting info | 22:53 |
| clarkb | I probably won't add them to the list because etime | 22:54 |
| *** sdake_ has joined #openstack-infra | 22:54 | |
| jeblair | clarkb: you added the haproxy lines to indicate that it worked a little bit in close temporal proximity? | 22:55 |
| bknudson_ | this job failed: http://logs.openstack.org/72/278772/1/check/gate-requirements-integration-dsvm/f54fb26/ -- says the main setup script failed... how do I find the setup script log? | 22:56 |
| clarkb | jeblair: yes | 22:56 |
| jeblair | clarkb: though the first one is cD | 22:56 |
| clarkb | jeblair: for that subset of instances they did manage to connect to git.o.o to some degree | 22:56 |
| jeblair | which i guess is still interesting | 22:56 |
| clarkb | jeblair: but only those with the log lines | 22:56 |
| jeblair | clarkb: yeah. makes sense. | 22:56 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Remove job filters and add missing paths to subunit collection https://review.openstack.org/279299 | 22:57 |
| bknudson_ | hmm, maybe it's this: 2016-02-11 01:26:33.429 | E: You must put some 'source' URIs in your sources.list | 22:57 |
| *** sdake has quit IRC | 22:57 | |
| clarkb | jeblair: and grepping for that suspect /24 in nodepool log sows we use quite a few IPs in that range | 22:58 |
| clarkb | jeblair: so other instances with those IPs are working fine | 22:58 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Remove job filters and add missing paths to subunit collection https://review.openstack.org/279299 | 22:58 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:58 | |
| mtreinish | dims, jeblair: part 2 ^^^ | 22:58 |
| *** xyang1 has quit IRC | 22:58 | |
| jeblair | clarkb: yeah, so maybe they can localize to a switch/compute node or something | 22:58 |
| anteaya | bknudson_: this may help: http://logs.openstack.org/72/278772/1/check/gate-requirements-integration-dsvm/f54fb26/console.html#_2016-02-11_01_10_57_053 | 22:59 |
| dims | mtreinish : cool thanks | 22:59 |
| *** bpokorny has joined #openstack-infra | 22:59 | |
| mtreinish | dims: hopefully that won't blow up in our face | 23:00 |
| anteaya | bknudson_: this doesn't look good: http://logs.openstack.org/72/278772/1/check/gate-requirements-integration-dsvm/f54fb26/logs/devstack-gate-setup-workspace-new.txt.gz#_2016-02-11_01_17_43_212 | 23:00 |
| mtreinish | I've never tested any of this stack with more than a couple of test suites | 23:00 |
| *** sridhar_ram has quit IRC | 23:00 | |
| mtreinish | dunno how well the db is gonna scale when we throw all of them at it | 23:00 |
| mtreinish | I want to say perfectly, but who knows :) | 23:00 |
| fungi | anteaya: i don't see anything else obviously wrong, looks great! | 23:01 |
| bknudson_ | anteaya: fatal: Couldn't find remote ref refs/zuul/stable/kilo/Za987afe0f7504815a9a17535e4429a92 seems to happen pretty often | 23:01 |
| *** thorst_ has joined #openstack-infra | 23:03 | |
| bknudson_ | so I don't know if that's good or bad... seems to be ignored | 23:03 |
| *** gildub has joined #openstack-infra | 23:03 | |
| *** dpaterson has quit IRC | 23:03 | |
| bknudson_ | I'm going to go with "recheck" | 23:04 |
| anteaya | fungi: wonderful, thank you | 23:04 |
| anteaya | bknudson_: yeah I get thrown when fatal things aren't really fatal | 23:05 |
| fungi | clarkb: yeah, that looks like the basic useful detail i'd send them | 23:05 |
| fungi | along with some explanatory prose of course | 23:05 |
| anteaya | zaro: long live the blueman group | 23:06 |
| *** regXboi has quit IRC | 23:07 | |
| clarkb | jeblair: fungi sent with prose. You are cc'd | 23:10 |
| *** ihrachys has quit IRC | 23:11 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Allow zuul-release to push merge commits https://review.openstack.org/278967 | 23:12 |
| jhesketh | jeblair: if you have a moment ^ | 23:12 |
| jhesketh | I'd like to push up my merge of master -> feature/zuulv3 | 23:12 |
| austin81 | clarkb, fungi, jeblair, greghaynes: Adding stackviz config to nodepool/elements and nodepool/scripts. Can someone inform me on what nodepool/scripts/prepare_node.sh does? I'm not sure if I should be making a prepare_node_stackviz.sh script that calls it. | 23:13 |
| clarkb | austin81: prepare_node.sh is used by nodepools snapshot image builds to update the image with the contents we want | 23:13 |
| clarkb | austin81: you will likely want to add an element to nodepool/elements that installs stackviz too | 23:14 |
| anteaya | tonyb: so if you look in #openstack-meeting-3 the ops guide folks need some meetbot 101 | 23:14 |
| clarkb | we are moving everything to dib which uses the dib elements and moving away from snapshot builds but still use them until we have reliably image uploads to rax | 23:14 |
| *** dizquierdo has joined #openstack-infra | 23:14 | |
| anteaya | tonyb: at least for adding addiional chairs, ending a meeting, and popping into -infra for help if meetbot isn't don't as they expect | 23:14 |
| anteaya | tonyb: I'm not sure how to convey that to them | 23:15 |
| clarkb | speaking of reliable builds https://review.openstack.org/#/c/277941/ is going once going twice review it soon or I will self approve | 23:15 |
| austin81 | clarkb: I added an element to elements/node-devstack/install.d/99-prepare-stackviz | 23:15 |
| tonyb | anteaya: Thanks, I'll check the logs | 23:15 |
| austin81 | clarkb: Seemed like the right place for it, not sure where to add the corressponding script in nodepool/scripts | 23:15 |
| anteaya | tonyb: awesome | 23:15 |
| anteaya | tonyb: oh they aslo don't know how to start a meeting either it seems, :( | 23:16 |
| clarkb | austin81: prepare_devstack.sh is where I would add it for the snapshots | 23:16 |
| *** dizquierdo has quit IRC | 23:16 | |
| anteaya | not with confidence | 23:16 |
| clarkb | austin81: have a install_stackviz.sh or whatever and call it from prepare_devstack.sh | 23:16 |
| austin81 | clarkb: okay, great thanks! | 23:17 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Enable windmill-jobs for all ansible-roles https://review.openstack.org/279308 | 23:17 |
| mtreinish | austin81: probably take a look at what I did for the tempest testr preseeding for an example. I had to go through basically the same process for doing that | 23:17 |
| austin81 | mtreinish: That's been my main guide to figuring out nodepool :) | 23:18 |
| *** ybathia has joined #openstack-infra | 23:18 | |
| *** thorst_ has quit IRC | 23:20 | |
| clarkb | eventually we will trim that scripts dir to only things that need to run on the VM after they boot before testing | 23:20 |
| clarkb | but we haven't completely switched to dib yet | 23:20 |
| *** thorst_ has joined #openstack-infra | 23:21 | |
| *** ybathia_ has joined #openstack-infra | 23:21 | |
| austin81 | mtreinish: Mind if I add our code into prepare_node_devstack.sh? | 23:21 |
| mtreinish | austin81: you probably want to isolate the stackviz specific stuff into a seperate script, and just call that script from prepare_node_devstack.sh | 23:22 |
| austin81 | mtreinish: that works too. I'll call it after the subunit2sql stuff | 23:22 |
| *** ybathia has quit IRC | 23:23 | |
| *** ybathia has joined #openstack-infra | 23:24 | |
| *** ayoung has joined #openstack-infra | 23:25 | |
| tonyb | anteaya: So I'm assuming that the #endmeeting failed to work because it wasn't issueed by a meeting chair? and you have ops so that's why it worked for you? | 23:26 |
| *** ybathia_ has quit IRC | 23:26 | |
| clarkb | tonyb: there is a 60 minute timeout on it | 23:27 |
| clarkb | after that anyone can end it | 23:27 |
| tonyb | clarkb: okay thanks. That explains it. | 23:28 |
| *** thorst_ has quit IRC | 23:29 | |
| anteaya | yeah the timeout | 23:29 |
| anteaya | and yeah the chair thought she made other people chair but she didn't | 23:30 |
| anteaya | and the first attempt to start the meeting failed due to a whitespace before the # | 23:30 |
| *** chlong has joined #openstack-infra | 23:30 | |
| anteaya | but they thought it was because the successful person who did start the meeting had a registered nick | 23:30 |
| anteaya | I don't think they will figure this out on their own | 23:31 |
| anteaya | also the abandoned the meeting without asking for direction or support from infra | 23:31 |
| anteaya | I just noticed it in my channel flipping | 23:31 |
| *** baoli has quit IRC | 23:33 | |
| *** moravec has joined #openstack-infra | 23:34 | |
| openstackgerrit | Austin Clark proposed openstack-infra/project-config: Create stackviz element and script in nodepool (WIP) https://review.openstack.org/279317 | 23:36 |
| austin81 | mtreinish: ^ | 23:36 |
| *** baoli has joined #openstack-infra | 23:36 | |
| fungi | clarkb: jeblair had some test suggestions on 277941 that could be added at a later date, fyi. approved | 23:37 |
| clarkb | fungi: thanks, I meant to play with that more in a followup. The tricky thing is going to be races between first delete image being in a delete state and second image being in build state | 23:38 |
| clarkb | definitely doable but as an optimization I figured it could wait (see host of other things I have been battling the last couple days) | 23:38 |
| clarkb | I still havne't done the voip.ms thing :/ | 23:39 |
| openstackgerrit | Merged openstack-infra/nodepool: Handle image upload job failures https://review.openstack.org/277941 | 23:40 |
| lifeless | clarkb: for home? | 23:41 |
| fungi | lifeless: for pbx.openstack.org | 23:42 |
| fungi | clarkb: in your correlated list of connection failures, i also note that the successful hits in the haproxy log are not correlated to the failures on the client side, suggesting they are not from the connections which failed | 23:43 |
| clarkb | fungi: agreed | 23:43 |
| *** wolverin_ has joined #openstack-infra | 23:43 | |
| fungi | implying initial tcp/syn never arriving presumably | 23:43 |
| *** wolverin_ is now known as wolverineav | 23:43 | |
| *** annegentle_ has quit IRC | 23:44 | |
| fungi | clarkb: oh! i take that back | 23:44 |
| *** MarkAtwood has quit IRC | 23:44 | |
| fungi | in at least a couple instances, the hit logged to the haproxy is ~120 seconds before the failure on the client | 23:44 |
| *** FallenPe_ has quit IRC | 23:45 | |
| fungi | that smells like a connection timeout duration | 23:45 |
| *** wolverineav has quit IRC | 23:45 | |
| fungi | keeping in mind that haproxy is logging only successful establishment of sockets (hence full handshakes) we likely can't rule out that some of these connected initially and then timed out | 23:46 |
| fungi | i have to remind myself that log is not a packet capture and assume some application timing behaviors | 23:46 |
| openstackgerrit | Merged openstack/requirements: Stop using build-dep for numpy/pyyaml depends https://review.openstack.org/279099 | 23:47 |
| *** marcusvrn_ has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Preallocate swapfile https://review.openstack.org/278557 | 23:48 |
| fungi | clarkb: i take it no grenade jobs ran out of space on / with that ^ in your rechecks | 23:48 |
| clarkb | fungi: not that I saw | 23:48 |
| fungi | so hopefully we can just consider swap a sunk space cost on those machines and call it a day | 23:49 |
| clarkb | fungi: I even logged into one at one point an it had 50GB of free space | 23:49 |
| fungi | odd, because i thought that we were running close to max in ovh with grenade | 23:49 |
| fungi | but maybe that was early on and they upped our disk size? | 23:49 |
| *** ayoung has quit IRC | 23:50 | |
| clarkb | it is 80GB now I want ot say it was more problematic when it was 50GB? | 23:51 |
| clarkb | at the very least if there are real disk constraints we should shake them out with that d-g change in | 23:51 |
| clarkb | right now we ar ein this weird place where things break without being debuggable | 23:51 |
| clarkb | which I would prefer to transition to things break and are debuggable | 23:51 |
| *** dimtruck is now known as zz_dimtruck | 23:52 | |
| nibalizer | infra-root im gonna take the hiera lock to move aroud the omfra credentials some more | 23:52 |
| fungi | go for it | 23:52 |
| *** nwkarsten has quit IRC | 23:52 | |
| *** nwkarsten has joined #openstack-infra | 23:53 | |
| nibalizer | also i resolved the issue (with crinkle's help) about the infracloud compute hosts not running puppet | 23:53 |
| nibalizer | it was because i had group/infracloud.yml instead of group/infracloud.yaml | 23:53 |
| clarkb | nibalizer: ingy says you gotta use the yaml | 23:54 |
| clarkb | (he actually didn't seem to care but said that perosnally uses yaml) | 23:54 |
| nibalizer | our automation cares | 23:55 |
| nibalizer | in that case | 23:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from global requirements https://review.openstack.org/246114 | 23:55 |
| *** ggillies has quit IRC | 23:56 | |
| *** amotoki has joined #openstack-infra | 23:57 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Add gate-grenade-dsvm-neutron to tempest failure rate dashboard https://review.openstack.org/279326 | 23:57 |
| anteaya | tonyb: the original chair showed up again so I shared some explaination of meetbot commands | 23:58 |
| anteaya | no acknowledgement yet though | 23:58 |
| anteaya | in meeting-3 | 23:58 |
| *** nwkarsten has quit IRC | 23:58 | |
| tonyb | anteaya: okay. I'm also going to reach out to darrenc as I know him .... | 23:58 |
| anteaya | awesome | 23:58 |
| anteaya | good idea | 23:58 |
| fungi | nibalizer: worth noting, cronspam from all of those has been bouncing in a bunch of places because they're sending with unqualified hostnames as the domain part of the sender address rather than having an fqdn | 23:59 |
| Shrews | fungi: fwiw, i seem to have stopped receiving email notifications from gerrit, and i have not changed any settings | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!