| *** rbrndt_ has quit IRC | 00:01 | |
| *** austin81 has left #openstack-infra | 00:01 | |
| crinkle | tcpdump also shows it going to the mirror so it's not the puppet pip provider doing weird things | 00:03 |
|---|---|---|
| *** Sukhdev has joined #openstack-infra | 00:03 | |
| *** ashleighfarnham has quit IRC | 00:04 | |
| *** sputnik13 has quit IRC | 00:05 | |
| *** andymaier_ has quit IRC | 00:06 | |
| *** ayoung has quit IRC | 00:07 | |
| *** sarob has quit IRC | 00:08 | |
| docaedo | ianw: I use weechat running in a screen, but also connect to it with the weechat client on my phone and often glowing-bear web client too | 00:08 |
| *** sarob has joined #openstack-infra | 00:08 | |
| docaedo | ianw: I love it | 00:09 |
| *** sdake has quit IRC | 00:09 | |
| *** geekinutah has quit IRC | 00:09 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci: [DNM] Check pip.conf https://review.openstack.org/288186 | 00:09 |
| *** annegentle has joined #openstack-infra | 00:12 | |
| *** annegentle has quit IRC | 00:13 | |
| *** zeih has joined #openstack-infra | 00:14 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 00:15 | |
| *** zeih has quit IRC | 00:18 | |
| *** Sukhdev has quit IRC | 00:21 | |
| *** ybathia has joined #openstack-infra | 00:22 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Use unversioned identity endpoints https://review.openstack.org/288194 | 00:23 |
| *** notnownikki has quit IRC | 00:23 | |
| *** Sukhdev has joined #openstack-infra | 00:24 | |
| mordred | crinkle: ++ ^^ | 00:26 |
| *** Qiming has joined #openstack-infra | 00:26 | |
| *** ashleighfarnham has joined #openstack-infra | 00:27 | |
| *** Sukhdev has quit IRC | 00:27 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Let zuul-cloner work in post and release pipelines https://review.openstack.org/271585 | 00:28 |
| *** andymaier_ has joined #openstack-infra | 00:32 | |
| *** ashleighfarnham has quit IRC | 00:33 | |
| *** annegentle has joined #openstack-infra | 00:33 | |
| *** zhurong has joined #openstack-infra | 00:35 | |
| *** bastelfreak has quit IRC | 00:35 | |
| nibalizer | oh | 00:35 |
| nibalizer | oh i think i know | 00:35 |
| openstackgerrit | Qiming Teng proposed openstack/requirements: Bump python-senlinclient version to 0.3.0 https://review.openstack.org/263598 | 00:35 |
| nibalizer | i think puppets pip provder hits the pypi xmlrpc endpoint to determine if it has latest | 00:36 |
| nibalizer | is maybe that the issue? | 00:36 |
| crinkle | nibalizer: i tcpdumped and it looked like it was talking to the mirror | 00:37 |
| nibalizer | oh hrm | 00:38 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-openstackci: [DNM] Check pip.conf https://review.openstack.org/288186 | 00:40 |
| *** angdraug has quit IRC | 00:40 | |
| anteaya | may I have the following four gerrit accounts disabled please? 14208, 14212, 14213, 15443: http://paste.openstack.org/show/489263/ | 00:40 |
| *** tqtran is now known as tqtran-afk | 00:40 | |
| anteaya | they are all posting failed to merge messages on patches | 00:40 |
| anteaya | which is distracting noise for devs | 00:40 |
| *** ybathia has quit IRC | 00:41 | |
| anteaya | my attempt to get someone from cisco to turn them off from there end failed | 00:41 |
| anteaya | the person I was talking to disappeared without getting back to me | 00:41 |
| anteaya | their end | 00:41 |
| *** bastelfreak has joined #openstack-infra | 00:45 | |
| openstackgerrit | amrith proposed openstack/requirements: Update upper-constraints.txt for python-troveclient v 2.1.0 https://review.openstack.org/288201 | 00:51 |
| fungi | okay, here's a new one i don't have time to dig into but the release time might love someone figuring out what we broke: http://releases.openstack.org/ seems (to me) to be hitting our default vhost on static.o.o which redirects to the system-config docs | 00:51 |
| fungi | anteaya: i can disable those four ids, sure, just a sec | 00:51 |
| anteaya | fungi: thanks | 00:51 |
| openstackgerrit | amrith proposed openstack/requirements: Update upper-constraints.txt for python-troveclient v 2.1.0 https://review.openstack.org/288201 | 00:53 |
| *** ZZelle_ has quit IRC | 00:53 | |
| anteaya | I see this when I navigate to releases.o.o: http://imgur.com/FWrLOVN | 00:53 |
| *** rossella_s has quit IRC | 00:54 | |
| fungi | thanks anteaya. it looks like i probably have a cached 301 redirect in that case | 00:54 |
| anteaya | welcome | 00:54 |
| *** rossella_s has joined #openstack-infra | 00:54 | |
| *** ddieterly has joined #openstack-infra | 00:55 | |
| *** ddieterl_ has joined #openstack-infra | 00:56 | |
| *** jpr has quit IRC | 00:56 | |
| *** annegentle has quit IRC | 00:56 | |
| fungi | not sure how to clear this from the cache without blowing away the entire browser cache | 00:56 |
| anteaya | :( | 00:57 |
| anteaya | I have no suggestions | 00:57 |
| anteaya | well nothing I think will help | 00:57 |
| anteaya | I do have lots of unhelpful suggestions | 00:58 |
| bswartz | docaedo: ping | 00:58 |
| *** akshai has quit IRC | 00:58 | |
| fungi | anteaya: good news, i've disabled those four accounts, flushed gerrit's caches and disconnected the 8 connections they had open to it amongst them | 00:59 |
| anteaya | fungi: wonderful thank you | 00:59 |
| fungi | so the annoyance should cease | 00:59 |
| anteaya | I will compose an email and go to bed | 00:59 |
| anteaya | kevinbenton: you should soon cease to be annoyed | 00:59 |
| anteaya | well to the extent I have any say in the matter that is | 00:59 |
| *** ddieterly has quit IRC | 01:00 | |
| docaedo | bswartz: howdy! | 01:00 |
| bswartz | docaedo: I'm looking at http://apps.openstack.org/ now | 01:00 |
| bswartz | this is first time I've seen it | 01:00 |
| anteaya | kevinbenton: you might cease to be annoyed now | 01:00 |
| *** zeih has joined #openstack-infra | 01:00 | |
| kevinbenton | anteaya: yay :) | 01:00 |
| fungi | bswartz: there was a pretty awesome demo of it at a keynote in vancouver | 01:00 |
| * bswartz had to miss vancouver summit :-( | 01:00 | |
| fungi | which puts it at right about a year old now | 01:01 |
| fungi | or coming close | 01:01 |
| docaedo | bswartz: fire away with any questions - it DOES aim to be the authoritative place for having glance images | 01:01 |
| *** AndyU has joined #openstack-infra | 01:01 | |
| bswartz | docaedo: so our use case requires that we can pull images out of it for running check/gate jobs | 01:02 |
| *** akshai has joined #openstack-infra | 01:02 | |
| bswartz | and we need to be able to quickly push bugfixes to images as they're found, as they can affect manila | 01:02 |
| bswartz | so I guess I'm curious to know what is the upload process? | 01:02 |
| docaedo | that's kind of the sore spot right now | 01:02 |
| bswartz | can it be automated? are there humans involved? what quality checks are enforced? | 01:02 |
| fungi | if the primary purpose is to use the images in ci jobs then continuing to stash them on tarballs.o.o still probably makes sense, but apps.o.o would probably still be a good authoritative place for docs to refer for released images | 01:03 |
| bswartz | fungi: that's what I was starting to think | 01:03 |
| docaedo | To add an asset right now, a human checks out the repo, edits assets.yaml in it (and adds the info there), then submits a PR | 01:03 |
| AndyU | clarkb nibalizer jeblair fungi etc: have a couple more quick questions about potential infra donation | 01:03 |
| fungi | since the earlier comment you'd made was that you needed somewhere "official" for the install docs to link | 01:03 |
| docaedo | if there's a big binary chunk as well, there's a human involved | 01:03 |
| bswartz | most likely we don't want to expose HEAD commit of manila-image-elements to the whole world | 01:04 |
| docaedo | we're working to replace that by implementing glare at the back end, which would mean the entire process could be fully automated | 01:04 |
| fungi | AndyU: ask away | 01:04 |
| *** jaypipes has quit IRC | 01:04 | |
| *** zeih has quit IRC | 01:05 | |
| AndyU | Thanks! :-) | 01:05 |
| bswartz | fungi: our primary purpose is images for CI jobs, but a close second is customer use -- the main difference is that customers would use tagged builds and CI would use HEAD | 01:05 |
| *** ayoung has joined #openstack-infra | 01:05 | |
| AndyU | You said that if we host a cloud for your use it need not be plain vanilla openstack, right? Any concerns that it would contain some customizations that have been done on our end? | 01:06 |
| *** max_lobur has quit IRC | 01:07 | |
| *** Swami has quit IRC | 01:07 | |
| bswartz | docaedo: so if we output images from manila-image-elements to tarballs.o.o, could we take the tagged ones can push them into the app-catalog just by writing some yaml with a URL points to the tarballs.o.o site? | 01:07 |
| fungi | bswartz: yeah, so tarballs.o.o for a branch tip and versioned release images, but maybe also apps.o.o for the released images and have the install docs refer to the latter | 01:07 |
| *** Swami has joined #openstack-infra | 01:07 | |
| *** piet has quit IRC | 01:07 | |
| AndyU | In other words would our reference architecture for our openstack clouds be acceptable? Do you need us to be on a specific release? | 01:07 |
| bswartz | fungi: That sounds very appealing | 01:07 |
| docaedo | bswartz: yes that would work | 01:08 |
| *** akshai has quit IRC | 01:08 | |
| *** Qiming has quit IRC | 01:08 | |
| bswartz | docaedo: what is the turnaround time on reviewing new image submissions? I'm asking because we're working on the install guide right now and in order to test it, the image actually has to exist where we say it does | 01:08 |
| fungi | AndyU: you've sort of just described the state of openstack across existing providers. there are very few who just have a vanilla deployment (we sometimes like to jokingly quota randy bias who said "there is no such thing as vanilla openstack" though a lot of us disagree on that point) | 01:09 |
| bswartz | so if we're going to use app-catalog I need to hurry up and get a build done, tagged, and submitted | 01:09 |
| fungi | AndyU: so it would sort of depend on what kind of "customizations" you're talking about | 01:09 |
| anteaya | AndyU: and what release | 01:10 |
| fungi | AndyU: for example, if it prevents us being able to upload images to glance, or being able to reach instances we've booted, or being able to communicate between those instances, those would be problematic for our use case | 01:10 |
| docaedo | bswartz: can review tonight, turnaround is usually same day (within a few hours) | 01:10 |
| AndyU | Fungi: hard to really describe but it's all based on MOS openstack. | 01:10 |
| anteaya | kevinbenton: http://lists.openstack.org/pipermail/third-party-announce/2016-March/000288.html | 01:10 |
| anteaya | I don't know what mos openstack is | 01:11 |
| bswartz | okay well my project-config patch has to land first https://review.openstack.org/#/c/288149/ | 01:11 |
| AndyU | fungi: I can have Mike Wilson connect with you on specifics to see if anything would raise any red flags. | 01:11 |
| *** Swami has quit IRC | 01:11 | |
| anteaya | bswartz: I have reviewed that already | 01:11 |
| bswartz | once that merges I can tag a commit in manila-image-elements and ....HOPEFULLY..... out will pop a qcow2 file on tarballs site | 01:11 |
| bswartz | anteaya: I saw, ty | 01:12 |
| anteaya | welcome | 01:12 |
| jhesketh | jeblair: ping | 01:12 |
| docaedo | bswartz: I'm stepping away for a bit for family stuff but will be back later and am generally around - definitely ping me or someone else on #openstack-app-catalog if you have any more questions, or need to try to get something reviewed ASAP | 01:12 |
| AndyU | fungi: Assuming customization is not an issue, what about the version of openstack? Any concerns if we're a couple releases behind current? | 01:12 |
| bswartz | docaedo: thanks, it's fine if we don't get to this until tomorrow | 01:13 |
| docaedo | bswartz: cool | 01:13 |
| fungi | AndyU: as in kilo? juno? we've used them, i don't think we do anything that demands a particularly new openstack environment | 01:13 |
| fungi | AndyU: is mos mirantis openstack or something else? | 01:14 |
| AndyU | fungi: yes, mirantis | 01:14 |
| *** achanda has joined #openstack-infra | 01:14 | |
| AndyU | and yes... juno, kilo ;-) | 01:14 |
| *** blogan is now known as blogan-bot | 01:14 | |
| *** blogan-bot is now known as blogan | 01:14 | |
| AndyU | fungi: it's somewhat a matter of timing | 01:15 |
| anteaya | was hoping it was Mos Eisley openstack | 01:15 |
| fungi | i don't know that we've explicitly used environments based on mirantis openstack but i have a feeling they're still pretty openstacky | 01:15 |
| anteaya | that would be fun | 01:15 |
| fungi | mos eisley is dangerous. always carry a blaster | 01:15 |
| anteaya | so I hear | 01:15 |
| *** sridhar_ram has quit IRC | 01:16 | |
| * anteaya will be cautious | 01:16 | |
| AndyU | fungi: but I think we'd treat it like all our other clouds and periodically upgrade in an unending quest to catch up | 01:16 |
| fungi | AndyU: makes sense. we don't expect 100% uptime and our ci system is able to tolerate different clouds going offline for extended periods | 01:16 |
| fungi | it's one of the reasons we like to use multiple clouds. better resilience that way | 01:17 |
| AndyU | fungi: any upgrades would be done very efficiently... they have to be. And of course you'd have ample notice. | 01:17 |
| *** dprince has joined #openstack-infra | 01:18 | |
| AndyU | fungi: so what i hear from you is that In PRINCIPLE some customization is not a problem and being 2 or 3 releases of the trunk is not a problem. | 01:19 |
| *** dimtruck is now known as zz_dimtruck | 01:19 | |
| fungi | AndyU: yes, really we do very basic things, all via the api | 01:19 |
| fungi | upload images, boot instances from them, log into the instances over the internet, run stuff on them, delete the instances again | 01:20 |
| fungi | image uploads are refreshed once a day when things are working correctly | 01:20 |
| fungi | average lifetime of an instance is probably somewhere around 30 minutes | 01:20 |
| AndyU | fungi: Correct me if I have any of that wrong. I'll ask Mike Wilson to connect with you to explore some of those potential concerns and see where we end up. | 01:21 |
| AndyU | fungi: ok, I type slow. You were already there ahead of me ;-) | 01:21 |
| fungi | they're generally 8gb ram, hopefully with somewhere around 8 vcpus and at least 80gb disk each | 01:21 |
| AndyU | fungi: right | 01:22 |
| fungi | and we run as many in parallel as we can during peak load, though virtually none when things are quiet (we're very elastic) | 01:22 |
| *** kzaitsev_mb has quit IRC | 01:22 | |
| fungi | and that's basically it | 01:23 |
| AndyU | fungi I just was hit with a few questions and I wanted to double check with you before I responded. It's all in an exploratory stage at the moment so if he hit a bump, we'll adjust the plan | 01:23 |
| anteaya | things are quietening down, 30 in check, 20 in the gate | 01:23 |
| fungi | cool, and thanks for wanting to pitch in! | 01:23 |
| anteaya | AndyU: once the cloud is ready we start with one node and troubleshoot from there | 01:24 |
| AndyU | fungi: thanks for the info! My confidence in actually succeeding on this quest is growing ;-) I'll surely be in touch again. | 01:24 |
| anteaya | AndyU: then we up slowly and see what happens | 01:24 |
| *** zz_dimtruck is now known as dimtruck | 01:25 | |
| fungi | okay, i think i'm going to wrap up for the night | 01:25 |
| AndyU | fungi: Maybe we could get you access to a few tenants to just check things out. That would probably be a really good idea so that we don't get too far and find out it won't work. | 01:25 |
| AndyU | I'll look into it. Have a good night! | 01:26 |
| anteaya | fungi: yup, have a good night | 01:26 |
| fungi | AndyU: sounds like that would be a useful exploration | 01:26 |
| tonyb | fungi: do you have time for a quick election question? | 01:26 |
| fungi | i'm also not around much tomorrow, on the road a bunch of the day, though i'll pop in as i can in case there are emergencies | 01:26 |
| fungi | tonyb: sure, what's up? | 01:26 |
| anteaya | fungi: hope you have good driving conditions | 01:27 |
| tonyb | fungi: WRT generating the roles and having the correct email adress in gerrit | 01:27 |
| fungi | antqit's supposed to be rainy and windy, but at least nothing frozen | 01:27 |
| fungi | er, anteaya ^ | 01:27 |
| anteaya | oh good, no ice | 01:27 |
| *** sdake has joined #openstack-infra | 01:27 | |
| fungi | tonyb: electoral rolls? what's the question about addresses? | 01:27 |
| tonyb | fungi: that's only an issue for the generation of the roles. I'm assuming that as long as they're valid at that time as opposed to the start of the election all things are good | 01:27 |
| anteaya | well in the past I communicated that the electorate should confirm their gerrit prefered address | 01:28 |
| tonyb | fungi: Really I just need to pick a good date to say "make sure the email address in gerrit is right by $date" | 01:28 |
| *** thorst_ has quit IRC | 01:28 | |
| tonyb | anteaya: right I want to set that date | 01:28 |
| anteaya | as that would be the address the poll was sent to | 01:28 |
| fungi | tonyb: i suppose it's possible some will shift out from under us. when the rolls are generated they refledt the gerrit preferred e-mail addresses for each voter | 01:28 |
| tonyb | the current draft is today which is clearly uncool | 01:28 |
| *** thorst has joined #openstack-infra | 01:29 | |
| anteaya | the roll was created on the after the last day of eligibility | 01:29 |
| anteaya | which in the past was the day prior to the start of the ptl nomination period | 01:29 |
| bswartz | oh god is it election time already? | 01:30 |
| tonyb | anteaya: okay thanks. That's what I was looking for. | 01:30 |
| bswartz | did I miss a deadline? | 01:30 |
| *** dingyichen has quit IRC | 01:30 | |
| tonyb | bswartz: soon | 01:30 |
| anteaya | tonyb: yeah look for the archived emails about timelines | 01:30 |
| anteaya | and the wikipages | 01:30 |
| fungi | yeah, so basically if we have _too_ long of a delay between when rolls are generated and when the ballots are sent, it's possible some voters' addresses might change/become invalid | 01:30 |
| anteaya | it should all be there | 01:30 |
| bswartz | whew | 01:30 |
| anteaya | tonyb: I can dig this up for you tomorrow | 01:30 |
| tonyb | fungi, anteaya: thanks. | 01:30 |
| anteaya | unless you are tight for time and need it tonight | 01:30 |
| tonyb | anteaya: no need I'm pretty sure I have a solid plan. | 01:31 |
| *** dingyichen has joined #openstack-infra | 01:31 | |
| anteaya | how much time have you given yourself before you post an information email | 01:31 |
| anteaya | okay | 01:31 |
| *** akshai has joined #openstack-infra | 01:31 | |
| anteaya | well taht was what happened in the past | 01:31 |
| anteaya | it was up to the electorate to check their email | 01:31 |
| fungi | as for what anteaya is saying, the reasoning is 1. so that i have a day to be leisurely about roll generation without delaying the start of the election, and 2. so that i only have to do it once and you can use the rolls generated from that point in time for the tc election and for any ptl elections which come up | 01:31 |
| anteaya | yes, what fungi said | 01:32 |
| tonyb | anteaya, fungi: +1 | 01:32 |
| anteaya | I also used to give the electorate about 4 weeks to get themselves collected and confirm their email address was correct | 01:32 |
| anteaya | you always get some who can't find the email but it minimized that | 01:32 |
| tonyb | All the dates are public just not announced, so I'm going to set the "deadline" to PTL_start - 24 hours | 01:33 |
| fungi | also you need ttx, as the tc chair, to tag a point in time in the governance repo so that we can document the state of the reference material used to determine which project teams are involved and what deliverable repos count toward each of them | 01:33 |
| anteaya | yes that | 01:33 |
| tonyb | fungi: okay we'll line that up. | 01:34 |
| fungi | which also serves as a cut-off point for extra-atc additions counting toward electorate eligibility | 01:34 |
| tonyb | fungi: makes sense, but I have to admit it didn't occur to me that extra-atc's get to vote. | 01:35 |
| fungi | though since those tend to only get merged on tc meeting days, that cutoff is potentially as much as week earlier than when the tag is pushed | 01:35 |
| *** sdake has quit IRC | 01:35 | |
| tonyb | fungi: :) | 01:35 |
| openstackgerrit | Merged openstack/requirements: Bump upper constraint django_openstack_auth https://review.openstack.org/287991 | 01:35 |
| fungi | tonyb: the existence of extra-atcs tracking _primarily_ to enable them to vote | 01:35 |
| anteaya | tonyb: it is in the by-laws | 01:35 |
| fungi | that they also get free admission to the conference and summit is secondary | 01:36 |
| AndyU | anteaya: thanks for your input too. reading back I felt like I neglected to address you as well as fungi. I kinda loved the Mos Eisley comment ;) Bye for now. | 01:36 |
| tonyb | fungi: cool. Thanks so much for letting me de-rail your plans to unplug. | 01:37 |
| *** thorst has quit IRC | 01:37 | |
| *** mtanino has quit IRC | 01:37 | |
| anteaya | tonyb: 3.b.ii http://www.openstack.org/legal/technical-committee-member-policy/ | 01:37 |
| fungi | anyway, i concur with getting your public communication to the electoral body going as soon as you can. as anteaya says, you need to warm them up and get them prepared | 01:37 |
| *** yamahata has quit IRC | 01:37 | |
| tonyb | fungi: Yeah the plan is tomorrow. | 01:38 |
| anteaya | AndyU: thanks for coming back, yeah was feeling ignored, it happens sometimes, thanks for saying so | 01:38 |
| anteaya | AndyU: talk to you later | 01:38 |
| tonyb | anteaya: Thanks. | 01:39 |
| anteaya | tonyb: welcome | 01:39 |
| fungi | okay, night all | 01:39 |
| *** akshai_ has joined #openstack-infra | 01:39 | |
| anteaya | night | 01:39 |
| AndyU | anteaya: My brain was all over paying your great attention... but my fingers failed to follow! Sorry about that. | 01:39 |
| anteaya | no worries it happens | 01:39 |
| anteaya | just thanks for saying it wasn't intentional | 01:40 |
| anteaya | :) | 01:40 |
| anteaya | attend a tc meeting sometime | 01:40 |
| anteaya | noone can keep up | 01:40 |
| *** aeng has quit IRC | 01:40 | |
| AndyU | yw... my bad | 01:40 |
| anteaya | no worries | 01:40 |
| anteaya | I also have to offline | 01:40 |
| *** ashleighfarnham has joined #openstack-infra | 01:41 | |
| anteaya | night all | 01:41 |
| *** _sarob has joined #openstack-infra | 01:42 | |
| *** akshai has quit IRC | 01:42 | |
| *** AndyU has quit IRC | 01:43 | |
| *** sam_wan has joined #openstack-infra | 01:44 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/requirements: Bump u-c for python-ironicclient 1.2.0 https://review.openstack.org/288219 | 01:45 |
| *** sarob has quit IRC | 01:45 | |
| *** _sarob has quit IRC | 01:46 | |
| *** Qiming has joined #openstack-infra | 01:47 | |
| openstackgerrit | Merged openstack/requirements: Raise upper-constraint on python-congressclient https://review.openstack.org/288124 | 01:48 |
| *** sflanigan has quit IRC | 01:49 | |
| *** dmsimard has quit IRC | 01:50 | |
| *** dmsimard has joined #openstack-infra | 01:51 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Bump python-glanceclient to 2.0.0 https://review.openstack.org/287751 | 01:51 |
| openstackgerrit | Merged openstack/requirements: Raise manilaclient upper to 1.8.0 https://review.openstack.org/288129 | 01:52 |
| *** apoorvad has quit IRC | 01:53 | |
| openstackgerrit | Fei Long Wang proposed openstack/requirements: Bump upper-constraints for python-zaqarclient 1.0.0 https://review.openstack.org/288220 | 01:53 |
| *** sflanigan has joined #openstack-infra | 01:53 | |
| *** yamamoto has joined #openstack-infra | 01:53 | |
| *** tqtran-afk has quit IRC | 01:54 | |
| *** Sukhdev has joined #openstack-infra | 01:54 | |
| *** yuanying has quit IRC | 01:58 | |
| *** yuanying has joined #openstack-infra | 01:59 | |
| *** zeih has joined #openstack-infra | 02:01 | |
| *** esker has joined #openstack-infra | 02:01 | |
| *** Sukhdev has quit IRC | 02:04 | |
| *** zeih has quit IRC | 02:05 | |
| *** esker has quit IRC | 02:07 | |
| *** pvaneck has quit IRC | 02:08 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Fixes for latest cinder and neutron clients https://review.openstack.org/288110 | 02:13 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 02:13 |
| *** gyee has quit IRC | 02:15 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: IPv4 network isolation testing https://review.openstack.org/288163 | 02:15 |
| *** bpokorny_ has joined #openstack-infra | 02:16 | |
| *** thorst has joined #openstack-infra | 02:16 | |
| *** bpokorny has quit IRC | 02:19 | |
| *** tiswanso has quit IRC | 02:19 | |
| *** fawadkhaliq has quit IRC | 02:20 | |
| *** tiswanso has joined #openstack-infra | 02:20 | |
| openstackgerrit | Merged openstack/requirements: bump upper constraints of openstackclient https://review.openstack.org/288157 | 02:20 |
| *** bpokorny_ has quit IRC | 02:21 | |
| *** thorst has quit IRC | 02:23 | |
| *** yamamoto has quit IRC | 02:23 | |
| *** yamamoto has joined #openstack-infra | 02:23 | |
| clarkb | ok I am back again | 02:26 |
| *** andymaier_ has quit IRC | 02:26 | |
| clarkb | anything related to release/feature freeze need attention? | 02:26 |
| *** yamamoto has quit IRC | 02:29 | |
| *** ccrouch has quit IRC | 02:32 | |
| *** ccrouch has joined #openstack-infra | 02:33 | |
| *** apoorvad has joined #openstack-infra | 02:36 | |
| fungi | not that i've seen | 02:36 |
| *** sdake has joined #openstack-infra | 02:39 | |
| *** dprince has quit IRC | 02:40 | |
| *** esker has joined #openstack-infra | 02:43 | |
| *** dimtruck is now known as zz_dimtruck | 02:43 | |
| *** woodster_ has quit IRC | 02:47 | |
| openstackgerrit | Merged openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0 https://review.openstack.org/286587 | 02:47 |
| *** esker has quit IRC | 02:48 | |
| *** akshai_ has quit IRC | 02:51 | |
| *** sdake has quit IRC | 02:51 | |
| *** sdake has joined #openstack-infra | 02:52 | |
| *** sdake has quit IRC | 02:52 | |
| *** piet has joined #openstack-infra | 02:53 | |
| *** mgagne has left #openstack-infra | 02:56 | |
| *** akshai has joined #openstack-infra | 02:56 | |
| *** mgagne has joined #openstack-infra | 02:56 | |
| *** annegentle has joined #openstack-infra | 02:57 | |
| *** akshai has quit IRC | 03:00 | |
| *** links has joined #openstack-infra | 03:01 | |
| *** zeih has joined #openstack-infra | 03:02 | |
| openstackgerrit | Merged openstack/requirements: Add python-searchlightclient>=0.2.0 https://review.openstack.org/268394 | 03:02 |
| *** annegentle has quit IRC | 03:03 | |
| *** xiangxinyong has quit IRC | 03:04 | |
| *** kevinsho_ has quit IRC | 03:04 | |
| *** sflanigan is now known as sflanigan_afk | 03:04 | |
| *** sflanigan_afk is now known as sflanigan | 03:04 | |
| *** sflanigan is now known as sflanigan_afk | 03:05 | |
| *** lykinsbd has joined #openstack-infra | 03:05 | |
| *** lykinsbd has quit IRC | 03:06 | |
| *** zeih has quit IRC | 03:07 | |
| *** lykinsbd has joined #openstack-infra | 03:07 | |
| *** harlowja_at_home has quit IRC | 03:07 | |
| *** lykinsbd has quit IRC | 03:11 | |
| *** lykinsbd has joined #openstack-infra | 03:11 | |
| *** yamamoto has joined #openstack-infra | 03:11 | |
| *** lykinsbd has quit IRC | 03:12 | |
| *** kevinsho_ has joined #openstack-infra | 03:12 | |
| *** sflanigan_afk is now known as sflanigan | 03:13 | |
| *** esikachev has joined #openstack-infra | 03:13 | |
| *** piet has quit IRC | 03:13 | |
| *** piet has joined #openstack-infra | 03:14 | |
| *** amrith is now known as _amrith_ | 03:15 | |
| *** yuanying has quit IRC | 03:16 | |
| *** esikachev has quit IRC | 03:18 | |
| *** esker has joined #openstack-infra | 03:18 | |
| *** lykinsbd has joined #openstack-infra | 03:19 | |
| *** thorst has joined #openstack-infra | 03:21 | |
| openstackgerrit | Tim Buckley proposed openstack-infra/puppet-subunit2sql: Fix log_url parsing for new job types https://review.openstack.org/288234 | 03:23 |
| *** rhallisey has quit IRC | 03:27 | |
| *** thorst has quit IRC | 03:28 | |
| *** aeng has joined #openstack-infra | 03:37 | |
| *** fawadkhaliq has joined #openstack-infra | 03:38 | |
| openstackgerrit | Merged openstack/requirements: Update upper-constraints.txt for python-troveclient v 2.1.0 https://review.openstack.org/288201 | 03:38 |
| *** baoli has joined #openstack-infra | 03:39 | |
| *** dims has quit IRC | 03:40 | |
| *** yamahata has joined #openstack-infra | 03:41 | |
| openstackgerrit | Merged openstack/requirements: Bump ironic-lib to 1.1.0 https://review.openstack.org/287955 | 03:43 |
| *** dims has joined #openstack-infra | 03:43 | |
| *** dims has quit IRC | 03:44 | |
| *** links has quit IRC | 03:44 | |
| *** apoorvad has quit IRC | 03:45 | |
| *** _ody_ has joined #openstack-infra | 03:47 | |
| *** esker has quit IRC | 03:49 | |
| *** _ody has quit IRC | 03:50 | |
| *** rguillebert has quit IRC | 03:57 | |
| *** jamesmcarthur has joined #openstack-infra | 04:00 | |
| *** asselin__ has joined #openstack-infra | 04:02 | |
| *** zeih has joined #openstack-infra | 04:02 | |
| *** zeih has quit IRC | 04:07 | |
| *** yuanying has joined #openstack-infra | 04:08 | |
| openstackgerrit | Merged openstack/requirements: Update upper-constraints for barbianclient 4.0.0 https://review.openstack.org/287417 | 04:10 |
| *** yuanying has quit IRC | 04:12 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: long.defs/redat: set SYS_UID_MIN and SYS_GID_MIN https://review.openstack.org/288244 | 04:12 |
| EmilienM | ianw: hey! if you don't ming, please review ^ | 04:12 |
| *** jamesmcarthur has quit IRC | 04:15 | |
| *** jamesmcarthur has joined #openstack-infra | 04:20 | |
| *** Jeffrey4l has joined #openstack-infra | 04:22 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 04:27 |
| *** tojuvone_ has joined #openstack-infra | 04:27 | |
| *** sam_wan has quit IRC | 04:27 | |
| *** zz_dimtruck is now known as dimtruck | 04:27 | |
| *** piet has quit IRC | 04:27 | |
| *** thorst has joined #openstack-infra | 04:27 | |
| *** esker has joined #openstack-infra | 04:27 | |
| *** links has joined #openstack-infra | 04:27 | |
| *** baoli_ has joined #openstack-infra | 04:27 | |
| *** jamesmcarthur_ has joined #openstack-infra | 04:27 | |
| *** openstack has joined #openstack-infra | 04:29 | |
| *** jamesmcarthur has quit IRC | 04:29 | |
| *** jamesmcarthur_ is now known as jamesmcarthur | 04:29 | |
| *** markmcclain has joined #openstack-infra | 04:29 | |
| *** alex_xu has joined #openstack-infra | 04:29 | |
| *** dingyichen has quit IRC | 04:29 | |
| *** ZZelle has quit IRC | 04:29 | |
| *** yuanying has joined #openstack-infra | 04:30 | |
| *** baoli_ has quit IRC | 04:30 | |
| *** ccrouch has quit IRC | 04:30 | |
| *** dtantsur|afk has quit IRC | 04:30 | |
| *** cznewt has quit IRC | 04:30 | |
| *** gildub has quit IRC | 04:30 | |
| *** cznewt has joined #openstack-infra | 04:30 | |
| *** openstack has quit IRC | 04:30 | |
| *** openstack has joined #openstack-infra | 04:32 | |
| *** jlk has joined #openstack-infra | 04:33 | |
| *** jlk has quit IRC | 04:33 | |
| *** jlk has joined #openstack-infra | 04:33 | |
| *** thorst has quit IRC | 04:33 | |
| *** dangers_away has quit IRC | 04:33 | |
| *** esker has quit IRC | 04:33 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/system-config: login.defs/redhat: set SYS_UID_MIN and SYS_GID_MIN https://review.openstack.org/288244 | 04:33 |
| *** openstack has quit IRC | 04:34 | |
| *** openstack has joined #openstack-infra | 04:39 | |
| *** openstack has quit IRC | 04:39 | |
| *** openstack has joined #openstack-infra | 14:07 | |
| lucasagomes | hi all, if you have time can you please take a look at https://review.openstack.org/#/c/284036/ ? This patch is the final piece that enables the ipmi drivers in ironic to be tested in gate | 14:07 |
| lucasagomes | it already has a +2 and the patch is very small (3LOC) | 14:07 |
| lucasagomes | thanks | 14:07 |
| *** amitgandhinz has joined #openstack-infra | 14:07 | |
| *** _nadya_ has quit IRC | 14:08 | |
| *** zeih has joined #openstack-infra | 14:10 | |
| openstackgerrit | Merged openstack/requirements: Bump dogpile.cache to 0.5.7 https://review.openstack.org/288263 | 14:10 |
| pabelanger | AJaeger: going to use https://review.openstack.org/#/q/topic:bindep+status:open as topic | 14:10 |
| *** _nadya_ has joined #openstack-infra | 14:11 | |
| AJaeger | yeah, good idea. Can you change topics? | 14:11 |
| * AJaeger suggests to merge https://review.openstack.org/288446 and https://review.openstack.org/288427 next - then project-config, then python jobs (with manual JJB run). | 14:12 | |
| openstackgerrit | Merged openstack/requirements: Add couchdb to global requirements https://review.openstack.org/285191 | 14:12 |
| AJaeger | for python jobs I'd like to have some more infra-root around ;) | 14:12 |
| openstackgerrit | Merged openstack/requirements: Bump django-compressor to 2.0 https://review.openstack.org/286663 | 14:12 |
| openstackgerrit | Merged openstack/requirements: Add oauth2client to requirements https://review.openstack.org/284927 | 14:12 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Fix on category groups expand https://review.openstack.org/288464 | 14:13 |
| AJaeger | yolanda, mordred : could you review 288446, please? | 14:13 |
| *** mhu has quit IRC | 14:14 | |
| *** mhu has joined #openstack-infra | 14:14 | |
| *** mriedem has joined #openstack-infra | 14:15 | |
| yolanda | ok | 14:15 |
| *** piet has quit IRC | 14:16 | |
| pabelanger | AJaeger: Yup, already on the list | 14:17 |
| pabelanger | AJaeger: I'm working ansible-jobs patch | 14:17 |
| *** yamamoto_ has quit IRC | 14:17 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to ansible-func-ubuntu-trusty for ansible jobs https://review.openstack.org/288471 | 14:18 |
| openstackgerrit | Akihiro Motoki proposed openstack/requirements: Block neutronclient 4.1.0 to avoid gate breakage https://review.openstack.org/288321 | 14:19 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Switch to ansible-func-ubuntu-trusty for ansible jobs https://review.openstack.org/288471 | 14:19 |
| *** amotoki has quit IRC | 14:21 | |
| *** Daisy has joined #openstack-infra | 14:22 | |
| *** dkranz has quit IRC | 14:22 | |
| *** ildikov has joined #openstack-infra | 14:24 | |
| pabelanger | AJaeger: is there a bindep patch in flight ATM? | 14:25 |
| pabelanger | AJaeger: just to know what I should be watching for | 14:25 |
| *** moravec has joined #openstack-infra | 14:28 | |
| *** dkranz has joined #openstack-infra | 14:29 | |
| *** dansmith is now known as superdan | 14:29 | |
| *** rbrndt has joined #openstack-infra | 14:34 | |
| *** kencjohnston has joined #openstack-infra | 14:34 | |
| *** piet has joined #openstack-infra | 14:36 | |
| *** openstackgerrit_ has quit IRC | 14:36 | |
| *** openstackgerrit_ has joined #openstack-infra | 14:37 | |
| *** sorantis has joined #openstack-infra | 14:37 | |
| *** ddieterly has joined #openstack-infra | 14:38 | |
| SamYaple | sdake: http://logs.openstack.org/85/277085/3/check/gate-kolla-dsvm-deploy-centos-source/a8743b9/console.html#_2016-03-04_13_56_16_811 | 14:39 |
| *** dkranz has quit IRC | 14:39 | |
| SamYaple | we use pypi mirrors thats not whats slowing down centos gate | 14:39 |
| *** eharney has joined #openstack-infra | 14:39 | |
| SamYaple | we dont have package mirrors for rpm for centos | 14:39 |
| sdake | i recognize we dont have package mirrors for rpm | 14:40 |
| sdake | clarkb said earlier we are lacking pypi wheel prebuilt binaries on centos as well | 14:40 |
| sdake | in the infrastructure | 14:40 |
| sdake | not in our gating code | 14:40 |
| sdake | but in openstack inra | 14:40 |
| *** links has quit IRC | 14:41 | |
| *** icey has quit IRC | 14:42 | |
| *** icey has joined #openstack-infra | 14:43 | |
| AJaeger | pabelanger: nothiing in flight right now - and nothing coming from me for the moment | 14:43 |
| AJaeger | yolanda, thanks for the review. Do you have time for a +2A for https://review.openstack.org/#/c/288471, please? | 14:43 |
| yolanda | yep | 14:43 |
| AJaeger | pabelanger: 288471 is nice in that reduces jobs ;) | 14:44 |
| pabelanger | AJaeger: Ya, I think 288471 should be safe to do | 14:44 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for neutron/master https://review.openstack.org/286777 | 14:44 |
| pabelanger | AJaeger: do we want to ask for another +2 from somebody? | 14:44 |
| openstackgerrit | Merged openstack/requirements: Updated from generate-constraints https://review.openstack.org/287613 | 14:45 |
| pabelanger | I confirmed with odyssey4me that his team is ready for the flip to ubuntu-trusty | 14:45 |
| *** asettle has joined #openstack-infra | 14:45 | |
| *** nwkarsten has joined #openstack-infra | 14:45 | |
| AJaeger | pabelanger: yolanda just gave a +2. | 14:47 |
| AJaeger | pabelanger: thanks | 14:47 |
| pabelanger | AJaeger: yolanda: \o/ | 14:48 |
| yolanda | :) | 14:48 |
| pabelanger | will be testing sortly | 14:48 |
| AJaeger | pabelanger: that's what I did as well - run a couple of rechecks ;) | 14:49 |
| *** zz_dimtruck is now known as dimtruck | 14:49 | |
| AJaeger | I did this on the manual jobs we merged | 14:49 |
| *** ddieterly has quit IRC | 14:49 | |
| pabelanger | AJaeger: Ya odyssey4me already has patches up for other-requirements.txt so think we'll be good for rechecks | 14:50 |
| *** asettle has quit IRC | 14:50 | |
| AJaeger | pabelanger: do they really need other-requirements? Or is that an optimization? | 14:51 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for neutron/master https://review.openstack.org/286777 | 14:52 |
| *** ddieterly has joined #openstack-infra | 14:52 | |
| pabelanger | AJaeger: Ya, optimization. Since ansible installs packages by default, using empty other-requirements.txt will be a good way to ensure roles actually install dependencies | 14:52 |
| *** abregman|afk is now known as abregman | 14:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Use warlock in the glance v2 tests https://review.openstack.org/288502 | 14:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Mock glance v1 image with object not dict https://review.openstack.org/288503 | 14:53 |
| *** ddieterly has quit IRC | 14:53 | |
| *** _nadya_ has quit IRC | 14:53 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches https://review.openstack.org/288506 | 14:54 |
| AJaeger | pabelanger: ok, let ansible play with it ;) | 14:54 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for neutron/master https://review.openstack.org/286777 | 14:55 |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches https://review.openstack.org/288506 | 14:55 |
| *** piet has quit IRC | 14:56 | |
| AJaeger | ihrachys: is there one patch we should merge today? | 14:57 |
| ihrachys | AJaeger: not really, I still try to set all patches proper and pass gate. still in the middle of the process for the set. | 14:57 |
| ihrachys | I think we can wait till next week | 14:57 |
| ihrachys | as long as we clean it during the next week, we are good | 14:58 |
| *** abregman is now known as abregman|afk | 14:58 | |
| AJaeger | ihrachys: I'm not much available early in the week but can review before - and help push it through if needed later | 14:58 |
| *** mcarden has quit IRC | 14:59 | |
| *** jcoufal_ has quit IRC | 14:59 | |
| *** ccrouch has quit IRC | 15:00 | |
| *** mcarden has joined #openstack-infra | 15:00 | |
| *** tonyb has quit IRC | 15:01 | |
| *** tonyb has joined #openstack-infra | 15:01 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch to ansible-func-ubuntu-trusty for ansible jobs https://review.openstack.org/288471 | 15:02 |
| *** kgiusti has quit IRC | 15:02 | |
| AJaeger | pabelanger: Wait 30mins before you recheck so that those jobs are updated ^ | 15:03 |
| *** woodster_ has joined #openstack-infra | 15:03 | |
| *** piet has joined #openstack-infra | 15:04 | |
| pabelanger | AJaeger: ack | 15:04 |
| AJaeger | or wait: Enough to wait for next zuul reconfigure, you're not adding/changing jobs on jenkins | 15:05 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: IPv4 network isolation testing https://review.openstack.org/288163 | 15:05 |
| anteaya | smcginnis: so patches to cinder's unit tests don't run any integrated tests in the gate? | 15:05 |
| *** yamamoto has joined #openstack-infra | 15:05 | |
| *** yamamoto has quit IRC | 15:05 | |
| AJaeger | So, let's watch status.openstack.org/zuul for "Last reconfigured" | 15:05 |
| AJaeger | morning, anteaya | 15:06 |
| anteaya | AJaeger: morning Andreas | 15:06 |
| *** vgridnev has quit IRC | 15:06 | |
| smcginnis | anteaya: There was a change recently to filter out some of the tests based on if it is a docs/reno/tests only change. | 15:06 |
| *** yamamoto has joined #openstack-infra | 15:06 | |
| smcginnis | AJaeger: That was you, right? ^^ | 15:06 |
| anteaya | right I'm familiar with not running intregrated jobs in the gate on docs or reno tests | 15:07 |
| anteaya | just confirming what I am seeing | 15:07 |
| AJaeger | anteaya: we're currently moving jobs from bare-trusty to ubuntu-trusty, so if any questions about strange jobs show up, let's investigate. If you want to review, see https://review.openstack.org/#/q/topic:bindep+status:open | 15:07 |
| AJaeger | smcginnis: let me check... | 15:07 |
| *** zeih has quit IRC | 15:07 | |
| AJaeger | smcginnis: I673db66cbfba573182abaa200e13546715881c82 - mriedem did it | 15:08 |
| smcginnis | AJaeger: Oh, right. Thanks. THere you go anteaya ^^ | 15:08 |
| anteaya | so this behaviour is as intended? | 15:08 |
| smcginnis | anteaya: Does make the infra load a little lighter, so I do like it. | 15:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Fixes for latest cinder and neutron clients https://review.openstack.org/288110 | 15:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Use warlock in the glance v2 tests https://review.openstack.org/288502 | 15:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Mock glance v1 image with object not dict https://review.openstack.org/288503 | 15:09 |
| anteaya | smcginnis: okey dokey | 15:09 |
| anteaya | thanks | 15:09 |
| anteaya | AJaeger: can I ask a favour? | 15:09 |
| AJaeger | anteaya: sure | 15:09 |
| *** kgiusti has joined #openstack-infra | 15:09 | |
| anteaya | do you mind sharing the patch number of a patch rather than the change id in channel? | 15:09 |
| anteaya | when searching on a change id gerrit returns every reference for it | 15:09 |
| anteaya | including patches that reference that change id | 15:10 |
| AJaeger | anteaya: I can try - I was going from git ... | 15:10 |
| AJaeger | anteaya: 286180 | 15:10 |
| anteaya | thanks | 15:10 |
| *** markvoelker has quit IRC | 15:10 | |
| *** markvoelker has joined #openstack-infra | 15:10 | |
| anteaya | AJaeger: you are a bit of a trend setter | 15:11 |
| AJaeger | what have I done again? ;) | 15:11 |
| anteaya | just hoping the crowd wasn't follwing you into change ids :) | 15:11 |
| AJaeger | anteaya: I see... | 15:11 |
| anteaya | no just explaining myself further | 15:11 |
| anteaya | :) | 15:11 |
| anteaya | thanks | 15:11 |
| *** ccrouch has joined #openstack-infra | 15:12 | |
| *** claudiub has joined #openstack-infra | 15:12 | |
| *** cznewt has quit IRC | 15:12 | |
| jeblair | clarkb, mordred: yes, pabelanger and krotscheck were looking into centos afs... i think there was a wip patch | 15:12 |
| mordred | cool! | 15:12 |
| krotscheck | jeblair, mordred: I'm currently focused on fixing a mitaka release blocker. | 15:13 |
| krotscheck | And ironic things | 15:13 |
| anteaya | okay so here is my follow up argument, why do we need patches that don't run integrated tests in the integrated gate? | 15:13 |
| *** annegentle has quit IRC | 15:14 | |
| anteaya | they just line up with the others, risk resetting the gate and they aren't running integrated tests | 15:14 |
| *** esker has joined #openstack-infra | 15:14 | |
| anteaya | so what is the point? | 15:14 |
| EmilienM | fungi: when you have some time today, login.defs patch https://review.openstack.org/#/c/288244/ | 15:15 |
| jeblair | clarkb, mordred, krotscheck, pabelanger: apparently there's an abandoned patch: https://review.openstack.org/#/c/273155/ | 15:15 |
| *** esker has quit IRC | 15:15 | |
| *** jswarren_ is now known as jswarren | 15:15 | |
| *** esker has joined #openstack-infra | 15:15 | |
| krotscheck | I have un-abandoned that | 15:15 |
| krotscheck | people keep asking about it. | 15:16 |
| *** amitgandhinz has quit IRC | 15:16 | |
| krotscheck | But I'm not certain I'm comfortable with the way I'm selecting the version | 15:16 |
| morgan | krotscheck: quick reabandon it to confuse people more! :P | 15:16 |
| * morgan goes back under a rock. | 15:16 | |
| krotscheck | Naah | 15:17 |
| jeblair | anteaya: when you search using a change id, how do you go about doing that? | 15:17 |
| anteaya | I copy the change id and paste it into the gerrit search text box | 15:17 |
| anteaya | I do believe old gerrit would just return the patch that had that change id as its change id | 15:18 |
| jeblair | anteaya: that should only return results that are that change id | 15:18 |
| anteaya | new gerrit gives you every patch that mentions it | 15:18 |
| *** vgridnev has joined #openstack-infra | 15:18 | |
| anteaya | I would prefer if it did that | 15:18 |
| anteaya | old gerrit used to | 15:18 |
| jeblair | anteaya: (which is usually just one, in which case it goes directly to the change) | 15:18 |
| anteaya | yes | 15:18 |
| anteaya | that is the old behaviour | 15:19 |
| *** hashar has quit IRC | 15:19 | |
| anteaya | and my preference | 15:19 |
| mordred | krotscheck: fwiw, it's nowhere near the most terrible thing I've ever seen | 15:19 |
| anteaya | in which case sharing change ids was efficient | 15:19 |
| krotscheck | Hey, anyone want to collaborate on an April Fool's RFC? "TCP Packet engraving for ultra-long network delays"? Recommendations on materials on which your TCP packets can be etched based on how many years of latency you're expecting, and/or how long the government wants you to keep it. | 15:19 |
| mordred | hah | 15:19 |
| jeblair | https://review.openstack.org/#/q/Ibb88c64ddd899c09cd148f398e21ac613be9f15b | 15:20 |
| jeblair | anteaya: ^ | 15:20 |
| *** vgridnev has quit IRC | 15:20 | |
| jeblair | anteaya: https://review.openstack.org/#/q/message:Ibb88c64ddd899c09cd148f398e21ac613be9f15b | 15:20 |
| anteaya | jeblair: thank you | 15:20 |
| ihrachys | AJaeger: ok, I think I am set for the most part. it seems we really want to get https://review.openstack.org/#/c/286777/ in though it depends on another neutron patch, so I gotta chase for reviews there first. | 15:20 |
| anteaya | jeblair: if you look at https://review.openstack.org/#/c/288080/ | 15:20 |
| *** jpr has joined #openstack-infra | 15:20 | |
| *** jcoufal has joined #openstack-infra | 15:21 | |
| jeblair | anteaya: compare those two searches. the first is just on the change id and it returns exactly one result. the second is for the change id within the commit message and returns 3 | 15:21 |
| anteaya | jeblair: if you click on the change id in the commit message after "this reverts commit" | 15:21 |
| ihrachys | AJaeger: there is another patch to switch stable/liberty to regular target names, but that will need to wait for a backport | 15:21 |
| *** piet has quit IRC | 15:21 | |
| anteaya | jeblair: gerrit links to two patches | 15:21 |
| *** sorantis has quit IRC | 15:21 | |
| jeblair | anteaya: https://review.openstack.org/#q,I2316d13e90210d641541fdc18d9357d085971346,n,z ? | 15:21 |
| jeblair | anteaya: that link returns 2 patches for you? | 15:22 |
| mordred | https://review.openstack.org/#/q/6495027a8a2dac1870d39716c44a85348e8bffad,n,z | 15:22 |
| mordred | that does | 15:22 |
| anteaya | click on This reverts commit 6495027a8a2dac1870d39716c44a85348e8bffad. | 15:22 |
| mordred | the commit link - not the depends-on reference | 15:22 |
| anteaya | this is new gerrit behaviour | 15:22 |
| mordred | yah - that's a search for a git sha, not a gerrit change-id | 15:22 |
| AJaeger | ihrachys: I'll review later... | 15:22 |
| anteaya | gerrit used to just take you do the patch that it is reverting | 15:22 |
| *** jed56 has quit IRC | 15:23 | |
| anteaya | okay well I'm gerriting wrong then | 15:23 |
| anteaya | I'm fine if this is my fault and I should be doing something else | 15:23 |
| mordred | anteaya: I don't thnk it's about fault - just wanting to make sure we understand which things does which thing | 15:24 |
| AJaeger | jeblair, mordred: As next step for going from bare-trusty to ubuntu-trusty I'd like us to dogfoot project-config - https://review.openstack.org/285722 . Are you around in case any problems arise since a revert might need a manual merge? | 15:24 |
| anteaya | but I do find 6 digits easier to track than shas | 15:24 |
| *** cznewt has joined #openstack-infra | 15:25 | |
| anteaya | mordred: sure that's fine | 15:25 |
| anteaya | visually track them that is | 15:25 |
| mordred | AJaeger: yah | 15:25 |
| jeblair | anteaya: i don't know about that, but i think the old behavior of searching on a bare changeid still returns only changes that are that changeid. on the other hand, searching on a commit sha does indeed search within messages. | 15:25 |
| jeblair | anteaya: i agree about the shorter numbers :) | 15:26 |
| *** nwkarsten has quit IRC | 15:26 | |
| *** vgridnev has joined #openstack-infra | 15:26 | |
| *** kzaitsev_mb has quit IRC | 15:26 | |
| *** bhunter71 has quit IRC | 15:27 | |
| *** yamahata has joined #openstack-infra | 15:28 | |
| *** kencjohnston has quit IRC | 15:28 | |
| anteaya | I didn't realize that fungi's commit message about the revert included the git sha rather than the change id | 15:29 |
| anteaya | I guess since I had always used or seen the change id in commit messages about reverts I just assumed change id usage was universal | 15:30 |
| anteaya | so that is my incorrect assumption | 15:30 |
| AJaeger | Ok, then I +A it. | 15:30 |
| *** vgridnev has quit IRC | 15:30 | |
| anteaya | so perhaps I am mistaken about gerrit's behaviour | 15:31 |
| AJaeger | I think the changed behaviour is that gerrit inlcudes the git sha instead of the change ID in a revert message now. | 15:31 |
| jeblair | anteaya: you can tell a git commit sha from a gerrit change-id by whether it has a leading "I". if it does, it's a gerrit change id. | 15:31 |
| anteaya | oh okay thanks, I didn't know that | 15:32 |
| anteaya | that helps | 15:32 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Bumps os-win version to 0.2.3 https://review.openstack.org/288429 | 15:32 |
| anteaya | AJaeger: oh perhaps that is it | 15:32 |
| pabelanger | jeblair: mordred: Ya, working on it while bindep stuff merges now. I might have an initial patch today | 15:33 |
| jeblair | anteaya, AJaeger: some folks may create a revert commit with git itself, rather than gerrit. especially if they are creating it ahead of time. i don't know gerrit's behavior when creating a revert (i almost never use gerrit to do that), but git will only leave the sha, not the change-id, in the message). | 15:33 |
| *** Daisy has quit IRC | 15:33 | |
| *** jsavak has quit IRC | 15:34 | |
| AJaeger | jeblair: https://review.openstack.org/#/c/285356/ was done from review.openstack.org | 15:34 |
| *** david_lyle__ has joined #openstack-infra | 15:34 | |
| AJaeger | and it shows the git SHA | 15:34 |
| *** ZZelle has quit IRC | 15:34 | |
| mordred | gotcha. so we'll always get a git SHA in revert messages no matter where they come from | 15:35 |
| AJaeger | jeblair: https://review.openstack.org/#/c/285247/ has the usual "this change got reverted" message | 15:35 |
| *** dizquierdo has joined #openstack-infra | 15:35 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 15:36 |
| openstackgerrit | Claudiu Belu proposed openstack/requirements: Bumps os-win version to 0.2.3 https://review.openstack.org/288429 | 15:36 |
| *** vgridnev has joined #openstack-infra | 15:36 | |
| *** david-lyle has quit IRC | 15:37 | |
| *** vgridnev has quit IRC | 15:37 | |
| *** aysyd has quit IRC | 15:38 | |
| odyssey4me | oh dear, something seems to have changed in the docs publishing - http://docs.openstack.org/developer/openstack-ansible/ is now showing content from our liberty branch, instead of from the master branch | 15:38 |
| * AJaeger likes to increase min-ready for ubuntu-trusty while we convert jobs - are these values fine? Please review https://review.openstack.org/288427 | 15:38 | |
| *** aysyd has joined #openstack-infra | 15:38 | |
| *** yamamoto has quit IRC | 15:39 | |
| pabelanger | AJaeger: LGTM | 15:39 |
| *** vgridnev has joined #openstack-infra | 15:39 | |
| AJaeger | odyssey4me: is 12.0.7 your latest tag? When was it tagged? | 15:39 |
| odyssey4me | AJaeger today, earlier today | 15:40 |
| *** yamahata has quit IRC | 15:40 | |
| openstackgerrit | Merged openstack-infra/project-config: Use ubuntu-trusty for project-config https://review.openstack.org/285722 | 15:40 |
| *** regXboi has joined #openstack-infra | 15:40 | |
| AJaeger | odyssey4me: So, https://review.openstack.org/#/c/279238/ did not solve the problem it should have solve - any project-config expert interested to take it to the drawing board again? | 15:40 |
| pabelanger | AJaeger: is ubuntu-trusty on rax broken? | 15:40 |
| AJaeger | pabelanger: https://review.openstack.org/#/c/282878/ - it was broken - jeblair, clarkb, fungi, mordred : Do you know whether it's safe to enable again? | 15:41 |
| *** vgridnev has quit IRC | 15:41 | |
| pabelanger | AJaeger: Ah, ya. That is right | 15:42 |
| pabelanger | forgot about that | 15:42 |
| clarkb | not yet need shade release | 15:42 |
| anteaya | jeblair AJaeger thanks for clearing that up for me, at least I understand what is happening now | 15:42 |
| AJaeger | clarkb: I see - thanks | 15:42 |
| AJaeger | anteaya: I learned something as well ;) | 15:42 |
| odyssey4me | AJaeger ah, interesting - I see that http://docs.openstack.org/developer/openstack-ansible/12.0.7/ is published... so the only issue is that the master branch content isn't published | 15:42 |
| AJaeger | odyssey4me: the issue is that we copied wrongly to the "normal" location. | 15:43 |
| AJaeger | odyssey4me: I tried to fix it as referenced but failed ;( | 15:43 |
| AJaeger | odyssey4me: your branch is called just "liberty", not "stable/liberty", correct? | 15:44 |
| odyssey4me | AJaeger yes, correct - we have 'liberty' and 'kilo' | 15:45 |
| AJaeger | odyssey4me: So, my fix solved it for all projects that use stable - see " echo $ZUUL_REFNAME | grep stable/" | 15:45 |
| AJaeger | odyssey4me: Please tag stable/mitaka. | 15:45 |
| odyssey4me | AJaeger we plan to | 15:45 |
| AJaeger | odyssey4me: with your next merge to openstack-ansible the doc publishing will override your content, so merge something ;) | 15:46 |
| odyssey4me | AJaeger ok cool - we'll have some merges coming in soon enough - thanks | 15:46 |
| AJaeger | odyssey4me: changes still welcome ;) | 15:47 |
| AJaeger | anteaya: regarding 4 or 8: I don't know. I wanted to start slowly. Feedback is welcome on what values to choose here... | 15:47 |
| AJaeger | I considered doing it in one steps: One increase now, another later with lowering bare-trusty min-ready... | 15:48 |
| *** mtanino has joined #openstack-infra | 15:49 | |
| *** andreykurilin has left #openstack-infra | 15:49 | |
| *** sorantis has joined #openstack-infra | 15:49 | |
| fungi | around for a bit from a lobby | 15:49 |
| fungi | looks like the cut-over has not caused any pain yet? | 15:50 |
| AJaeger | fungi, manuals look fine. | 15:50 |
| AJaeger | pabelanger: what's the status for ansible? | 15:50 |
| pabelanger | AJaeger: live and working | 15:50 |
| AJaeger | fungi, project-config just merged. | 15:50 |
| odyssey4me | AJaeger I'm looking at how that all works, and I don't see where the problem lies. echo $ZUUL_REFNAME | sed 's/stable.// will work perfectly against our branch names (all you're doing is removing the 'stable/' from the branch ref... as we don't have one it remains unchanged | 15:51 |
| AJaeger | pabelanger: woot | 15:51 |
| *** kiran-r has quit IRC | 15:51 | |
| AJaeger | odyssey4me: check the complete file | 15:51 |
| anteaya | AJaeger: okay well I'm +2 on 4, the stated value in the patch but would also be okay with 8 | 15:51 |
| anteaya | AJaeger: and if anyone else shares feedback, I'm curious | 15:52 |
| *** Sukhdev has joined #openstack-infra | 15:52 | |
| *** esikachev has quit IRC | 15:52 | |
| AJaeger | anteaya: I'M as well. Fungi, what about merging https://review.openstack.org/288427? | 15:53 |
| *** x00350071 has quit IRC | 15:53 | |
| *** dims is now known as dimsum_ | 15:53 | |
| *** kiran-r has joined #openstack-infra | 15:53 | |
| *** x00350071 has joined #openstack-infra | 15:54 | |
| fungi | AJaeger: anteaya: 4 seems fine for now. it's really only a means of making sure we have enough kept ready when zuul is mostly idle so jobs don't need to wait for additional nodes to be booted | 15:55 |
| *** yamamoto has joined #openstack-infra | 15:55 | |
| *** mriedem is now known as mriedem_meeting | 15:55 | |
| fungi | we can tune it later as we see job volume for u-t increase and b-t decreate | 15:55 |
| fungi | s/decreate/decrease/ | 15:55 |
| anteaya | fair enough | 15:56 |
| anteaya | thanks | 15:56 |
| fungi | adjusting it isn't terribly urgent, but also won't really hurt as long as we're using them | 15:56 |
| anteaya | okay | 15:56 |
| AJaeger | anteaya, fungi: could either of you review 288446,please - that changes api-jobs | 15:57 |
| pabelanger | Are we having issues with bluebox? http://grafana.openstack.org/dashboard/db/nodepool-blue-box | 15:57 |
| pabelanger | don't see many nodes in use | 15:57 |
| pabelanger | out of floating IPs? | 15:57 |
| fungi | pabelanger: probably floating ip leak again. i'll check | 15:57 |
| anteaya | jenkins 06 just deleted 37 nodes | 15:58 |
| anteaya | that is the only outlier I see on grafana at the moment | 15:58 |
| anteaya | that is the only jenkins outlier I see at the moment | 15:58 |
| *** nwkarsten has joined #openstack-infra | 15:59 | |
| fungi | jenkins06 may be unhappy, looking now | 15:59 |
| *** yamahata has joined #openstack-infra | 15:59 | |
| fungi | my charger wasn't working in the car for some reason, so i'm on limited battery life too, just fyi... could disappear sooner than expected | 16:00 |
| *** thorst has quit IRC | 16:00 | |
| *** thorst has joined #openstack-infra | 16:01 | |
| anteaya | fungi: yup | 16:02 |
| *** amitgandhinz has joined #openstack-infra | 16:02 | |
| anteaya | fungi: thanks, not critical just looked odd to me | 16:02 |
| *** jsavak has joined #openstack-infra | 16:03 | |
| fungi | jenkins06 does indeed seem unhealthy. i'm going to put it into prepare or shutdown for now and once it's idle i can clean it up and restart it the next time i'm around unless another infra-root beats me to it | 16:03 |
| *** nwkarsten has quit IRC | 16:03 | |
| clarkb | fungi: maybe just run the playbook against it in screen? | 16:04 |
| clarkb | if the api isn't too far gone it should do all the work for you | 16:04 |
| fungi | clarkb: i'm hesitant to use the restart playbook against a jenkins which is already struggling | 16:04 |
| fungi | given the much higher incidence of it not working | 16:04 |
| clarkb | worst case you just have to back and manually cleanup | 16:04 |
| clarkb | which is true either way | 16:05 |
| dguitarbite | AJaeger: fungi: Issue with dependency 'zip' for CI jobs: https://review.openstack.org/#/c/288549/ https://jenkins04.openstack.org/job/gate-training-labs-scripts/3/ | 16:05 |
| *** yamamoto has quit IRC | 16:05 | |
| clarkb | dguitarbite: I mentioned yesterday I think that you should use gzip | 16:05 |
| dguitarbite | AJaeger: fungi AFAIK this should not happen since ZIP should be pre-installed with the image | 16:05 |
| fungi | yeah, actual worst case is it goes idle and the cron job restarts it tonight since that's the weekly schedule | 16:05 |
| dguitarbite | clarkb: ok, my bad I missed your comment. I will update it. Thanks a lot. | 16:05 |
| fungi | yeah, "zip" (infozip/pkzip) is not really a common tool in unix circles. that's more of an msdos thing | 16:06 |
| *** rossella_s has quit IRC | 16:07 | |
| AJaeger | fungi: zip is listed in bindep-fallback | 16:07 |
| clarkb | if osic gets the new cert rolled out today my plan is to go ahead and make a mirror there and add it to nodepool. Will push up some wip changes for that nowish | 16:07 |
| fungi | AJaeger: yep, some of our jobs do use it | 16:07 |
| anteaya | clarkb: yay! | 16:08 |
| fungi | clarkb: pretty sure the playbook would have already failed on 06, i had to try a couple of times to hit the quietdown url successfully due to api errors | 16:08 |
| clarkb | fungi: fun | 16:08 |
| *** annegentle has joined #openstack-infra | 16:08 | |
| clarkb | we can run the playbook continuously so that we are always one jenkins down :/ | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config: Increase min-ready for ubuntu-trusty https://review.openstack.org/288427 | 16:08 |
| fungi | but it's in prepare for shutdown now, finally | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config: Use ubuntu-trusty for api-jobs https://review.openstack.org/288446 | 16:09 |
| *** mikelk has quit IRC | 16:09 | |
| AJaeger | first project-config jobs running on ubuntu-tursty | 16:09 |
| anteaya | yay! | 16:10 |
| fungi | pabelanger: good guess. we were at 50 disassociated floating ips, which i expect is our quota (as it is, nodepool is configured to boot no more than 39 instances there). i'm deleting them now | 16:10 |
| clarkb | it is odd that bluebox seems to suffer that far more than hpcloud did | 16:10 |
| clarkb | osic uses fips and will give us more data too | 16:11 |
| pabelanger | fungi: Based on what grafana was reporting, it looked like something related to floating IPs | 16:11 |
| pabelanger | which was cool grafana displayed it | 16:11 |
| *** jsavak has quit IRC | 16:12 | |
| *** e0ne has quit IRC | 16:12 | |
| fungi | indeed | 16:12 |
| *** jsavak has joined #openstack-infra | 16:12 | |
| mnaser | small question: regarding 80gb local disk for provided resources, where does that come from? | 16:12 |
| tristanC | would be nice to get https://review.openstack.org/#/c/235381/ , it reduce the number of floating-ip leaks | 16:13 |
| fungi | clarkb: yes, would be awesome if someone had time to get to the bottom of it. i wonder if there's a particular boot or delete failure mode we're experiencing in bluebox that causes nodepool not to realize it needs to delete the associated fip | 16:13 |
| mnaser | we're creating some new flavors which are more cpu power oriented, but low disk space. something like 40gb local ssd + 8gb memory + 8vcpus, but that doesnt seem to "fit" the requirements listed, so im just curious | 16:13 |
| fungi | mnaser: that's roughly how much space a grenade job needs to set up two copies of openstack at different versions and switch between them | 16:14 |
| mnaser | ah i see | 16:14 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Bump python-neutronclient to 4.1.1 https://review.openstack.org/288428 | 16:14 |
| mordred | fungi: otoh - if I can ever finish the shade-nodepool patch, the failure path we have now for fips goes away | 16:14 |
| fungi | we tried running at smaller disk sizes in a recently added provider and then grenade was no longer able to work due to insufficient space | 16:15 |
| mordred | fungi: it might be replaced by a different failure path, of course | 16:15 |
| mnaser | fungi: i see, in that case we probably could do with a private/tenant-specific flavor | 16:15 |
| *** thorst is now known as thorst_afk | 16:15 | |
| clarkb | ya its basically two sets of git repo checkouts + cinder volumes + swift glance things + VM images + swap | 16:15 |
| fungi | mnaser: though we do have the ability to stitch in an ephemeral disk in devstack-gate to accommodate instances with a small rootfs but available ephemeral disk | 16:16 |
| AJaeger | fungi, clarkb, mordred: What's the best time for switching over all python jobs? So far it looks green. Do we want to do this asap to get notified early about issues? The zuul load is light enough that we could do it now | 16:16 |
| AJaeger | (I mean: switching to ubuntu-trusty) | 16:16 |
| AJaeger | If we switch them over, I expect a manual JJB run... | 16:16 |
| mnaser | fungi: i mean cinder volumes are always available but i assume this is not a path that's usually taken | 16:17 |
| fungi | AJaeger: i don't mind not being around for it as long as there are others around who can babysit jjb and approve a revert (and babysit jjb again) if something goes terribly wrong | 16:17 |
| AJaeger | fungi: I agree - that's why I'm asking for help and opinions ;) | 16:17 |
| mnaser | fungi: shouldn't be too difficult... boot from volume with preserve flag set to 0 | 16:17 |
| *** david_lyle__ is now known as david_lyle | 16:17 | |
| *** kmartin__ has quit IRC | 16:17 | |
| mnaser | besides finding the resources to do it and test it :p | 16:17 |
| fungi | mnaser: right, nodepool lacks cinder support, though that's something we've discussed adding in the past | 16:17 |
| *** nwkarsten has joined #openstack-infra | 16:18 | |
| fungi | or boot-from-volume support for that matter | 16:18 |
| *** nwkarsten has quit IRC | 16:18 | |
| *** annegentle has quit IRC | 16:18 | |
| pabelanger | fungi: cool, blue box launching nodes again | 16:18 |
| odyssey4me | AJaeger I think I see the bug - I'll patch it shortly. | 16:19 |
| *** annegentle has joined #openstack-infra | 16:19 | |
| *** nwkarsten has joined #openstack-infra | 16:19 | |
| clarkb | AJaeger: I mostly intend on being around today | 16:19 |
| clarkb | yesterday was an unexpected absence from compuer | 16:19 |
| *** achanda has joined #openstack-infra | 16:19 | |
| clarkb | AJaeger: so go for it and I can make sure jjb does its thing | 16:19 |
| mnaser | fungi: once we get the new flavors, i'll try to propose a change. we'll be able to give more infra based on this setup and the way i have it in mind, it shouldn't be *too* complicated | 16:19 |
| AJaeger | clarkb: then let me approve fungi's patch. THANKS! | 16:20 |
| mnaser | + using ceph means cow deploys, so WAAAAY faster! | 16:20 |
| *** achanda has quit IRC | 16:20 | |
| mnaser | than downloading an image, etc.. but i'll do some checking and get back :) | 16:20 |
| *** arxcruz has quit IRC | 16:21 | |
| lucasagomes | hi all, if you have time can you please take a look at https://review.openstack.org/#/c/284036/ ? This patch is the final piece that enables the ipmi drivers in ironic to be tested in gate. It already has a +2 and the patch is very small (3LOC), thank you | 16:21 |
| lucasagomes | mordred, ^ mind taking a look if you have a sec later on today? | 16:21 |
| *** x00350071_ has joined #openstack-infra | 16:22 | |
| *** x00350071 has quit IRC | 16:22 | |
| *** Qiming has quit IRC | 16:22 | |
| AJaeger | lucasagomes: better mention that it's devstack-gate change ;) | 16:22 |
| lucasagomes | yeah it's devstack-gate change :-) sorry | 16:22 |
| clarkb | lucasagomes: what sets DEVSTACK_GATE_IRONIC_DEPLOY_DRIVER? | 16:23 |
| lucasagomes | clarkb, it's set in project-config | 16:23 |
| *** kzaitsev_mb has joined #openstack-infra | 16:23 | |
| lucasagomes | 1 sec will get the piece for you | 16:23 |
| dimsum_ | anteaya : hound is still broke :( | 16:23 |
| *** jamesmcarthur has joined #openstack-infra | 16:23 | |
| clarkb | lucasagomes: it should have a default in -wrap.sh | 16:23 |
| anteaya | dimsum_: :( | 16:24 |
| fungi | dimsum_: yep, someone needs to propose a patch to up its max-files or fix the open file leak if there is one | 16:24 |
| lucasagomes | clarkb, https://github.com/openstack-infra/project-config/blob/0aed404938373938840c990b7b5168b6f583cf23/jenkins/jobs/devstack-gate.yaml#L617 | 16:24 |
| *** jaosorior has quit IRC | 16:24 | |
| lucasagomes | clarkb, all jobs do set the driver, it would be hard to come up with a default tho... | 16:24 |
| clarkb | agent_ssh :) | 16:25 |
| anteaya | fungi: I'll see if I can find that setting in the go code | 16:25 |
| lucasagomes | yeah or pxe_ssh... | 16:25 |
| lucasagomes | clarkb, ok I can put any, but it's a bit bogus IMHO | 16:26 |
| clarkb | lucasagomes: why is it bogus? | 16:26 |
| clarkb | will ironic work with that default value? I think yes | 16:26 |
| *** tongli has joined #openstack-infra | 16:26 | |
| clarkb | so not bogus | 16:26 |
| fungi | anteaya: it's probably going to be a matter of adding a ulimit call into the initscript to set a higher non-default value for max open files | 16:26 |
| *** thorst_afk has quit IRC | 16:26 | |
| clarkb | (I don't actually care what the default is just that there is one that is functional) | 16:27 |
| *** andymaier_ has quit IRC | 16:27 | |
| lucasagomes | clarkb, right... I just said bogus cause all jobs will actually set what driver it will test. But yeah, setting {pxe,agent}_ssh will work | 16:27 |
| *** piet has joined #openstack-infra | 16:27 | |
| lucasagomes | clarkb, ok updating the patch | 16:27 |
| anteaya | oh so something we would do on our end | 16:28 |
| *** nwkarsten has quit IRC | 16:28 | |
| *** thorst_afk has joined #openstack-infra | 16:29 | |
| *** nwkarsten has joined #openstack-infra | 16:29 | |
| *** sputnik13 has joined #openstack-infra | 16:29 | |
| *** armax has quit IRC | 16:29 | |
| lucasagomes | clarkb, it's already set actually | 16:29 |
| *** armax has joined #openstack-infra | 16:29 | |
| lucasagomes | https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate-wrap.sh#L185 | 16:30 |
| *** kzaitsev_mb has quit IRC | 16:30 | |
| clarkb | mordred: jeblair with bluebox the region is RegionOne but the cloud name is bluebox-sjc1 | 16:30 |
| *** daemontool__ has joined #openstack-infra | 16:30 | |
| clarkb | mordred: jeblair: should I not call osic region1 osic-region1? | 16:30 |
| *** pblaho has quit IRC | 16:30 | |
| mordred | clarkb: yah. I think that's a better name | 16:31 |
| clarkb | lucasagomes: huh I failed at grepping | 16:31 |
| jeblair | clarkb: ++ | 16:31 |
| mordred | oh - well, | 16:31 |
| mordred | hang on | 16:31 |
| AJaeger | clarkb: the python job just merged, hope our bot will notice as well - so let's wait for the JJB timeout ;) | 16:31 |
| lucasagomes | clarkb, happens | 16:31 |
| jeblair | clarkb: call it 'osic-something' :) | 16:31 |
| mordred | clarkb, jeblair: bluebox sjc is called sjc because there is the possibility of having more than one bluebox cloud | 16:31 |
| mordred | there is only one osic cloud | 16:32 |
| jeblair | clarkb: that way if there's another osic we have room | 16:32 |
| mordred | so I think its cloud name should be "osic" | 16:32 |
| mordred | jeblair: there might be another osic? | 16:32 |
| clarkb | yes cloud name is osic and done | 16:32 |
| jeblair | mordred: well, osic is all about spinning up clouds... | 16:32 |
| jeblair | theoretically there could be another one, right? | 16:32 |
| clarkb | jeblair: yes I think that is possible | 16:32 |
| clarkb | in fact this one is at cloud1.osic.org | 16:32 |
| clarkb | so maybe osic-cloud1? | 16:32 |
| jeblair | osic-cloud1 lgtm | 16:33 |
| *** nwkarsten has quit IRC | 16:33 | |
| mordred | ++ | 16:33 |
| *** daemontool_ has quit IRC | 16:34 | |
| *** daemontool__ is now known as daemontool | 16:34 | |
| *** thorst_afk is now known as thorst | 16:35 | |
| max_lobur | Hi Folks! Could someone please help me to get myself added to these groups https://review.openstack.org/#/admin/groups/?filter=bareon-ironic ? They are new and we need to fill the up | 16:36 |
| jroll | mordred | there is only one osic cloud <- this may not always be true | 16:36 |
| zaro | morning | 16:36 |
| * jroll finishes reading scrollback and sees that was concluded anyway | 16:36 | |
| anteaya | morning zaro | 16:36 |
| mordred | jroll: yah - I think "may not always be true" is the thing I have learned this monring | 16:36 |
| *** ifarkas has quit IRC | 16:37 | |
| jroll | yeah | 16:37 |
| jeblair | in https://developer.rackspace.com/blog/openstack-innovation-center-updates-january-2016/ it says "The next consumers slated to come on board is the OpenStack Foundation Infrastructure team." | 16:38 |
| jeblair | i appreciate the shout-out, but i wonder if anyone knows someone at rackspace where we could correct that by removing the word "foundation", at least in future communications? | 16:39 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert infra* jobs to ubuntu-trusty https://review.openstack.org/288572 | 16:39 |
| jeblair | i mean, we don't say "OpenStack Foundation Nova Team" :) | 16:39 |
| jeblair | some of us don't work at the foundation and like to feel included | 16:40 |
| *** esikachev has joined #openstack-infra | 16:40 | |
| *** mriedem_meeting is now known as mriedem | 16:40 | |
| *** roxanaghe has joined #openstack-infra | 16:40 | |
| *** jsavak has quit IRC | 16:40 | |
| anteaya | good point | 16:40 |
| *** vgridnev has joined #openstack-infra | 16:41 | |
| *** jaypipes has joined #openstack-infra | 16:41 | |
| *** jsavak has joined #openstack-infra | 16:41 | |
| *** jaypipes is now known as leakypipes | 16:42 | |
| clarkb | my comcast connection is very unhappy today | 16:44 |
| clarkb | apparently much of portland is affected? | 16:44 |
| *** yamahata has quit IRC | 16:44 | |
| jeblair | jroll, clarkb: when the second osic module comes online in oakland, we should see if we can get a cloud there :) | 16:44 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Verify SSL with OSIC https://review.openstack.org/288582 | 16:45 |
| jeblair | i can wave hi to it | 16:45 |
| clarkb | I will WIP ^ until dns and cert are actually ready | 16:45 |
| jroll | jeblair: heh | 16:45 |
| * mordred agrees with jeblair | 16:45 | |
| jroll | jeblair: also, I will get that post fixed | 16:45 |
| * mordred has so far never worked for the Foundation :) | 16:45 | |
| *** auggy is now known as frauggy | 16:45 | |
| *** yamahata has joined #openstack-infra | 16:45 | |
| jeblair | jroll: aw, thanks! | 16:46 |
| * mordred hands jroll a puppy | 16:46 | |
| fungi | jeblair: yeah, i expect they're confused because the initial communication about approving the openstack infra team's use of osic resources went through the openstack foundation executive director, and presumably people at intel/rackspace's marketing machine don't realize the community and the foundation are not all the same thing | 16:46 |
| jroll | jeblair: it's all on github, makes it easy :) https://github.com/rackerlabs/docs-developer-blog/blob/master/_posts/2016-01-22-openstack-innovation-center-updates-january-2016.md | 16:46 |
| jeblair | mordred: you just direct it :) | 16:46 |
| yolanda | max_lobur, done | 16:47 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use OSIC with nodepool https://review.openstack.org/288584 | 16:47 |
| fungi | battery is about to die, so don't expect more out of me until probably 2100-2200 utc | 16:47 |
| clarkb | reviews on both of those appreciated then I can just approve once it is ready | 16:47 |
| *** bpokorny has joined #openstack-infra | 16:47 | |
| jeblair | fungi: your battery or the asus? ;) | 16:48 |
| fungi | hah | 16:48 |
| fungi | we are one biomechanical organism | 16:48 |
| *** _nadya_ has joined #openstack-infra | 16:48 | |
| jeblair | that's what the "fungi" laptop category meant | 16:48 |
| fungi | luckily it's just the battery which powers my interface to the internet | 16:48 |
| fungi | the battery powering my other important functions is fueled through different means | 16:49 |
| *** apoorvad has joined #openstack-infra | 16:49 | |
| *** kencjohnston has joined #openstack-infra | 16:49 | |
| timrc | It's well known robots require beer to function properly. | 16:49 |
| clarkb | source: Futurama | 16:50 |
| nibalizer | the vexxhost mirror is now puppeting | 16:50 |
| nibalizer | http://puppetboard.openstack.org/node/mirror.ca-ymq-1.vexxhost.openstack.org | 16:50 |
| nibalizer | so yea it was the ssh-key accept issue that was making ansible not work | 16:50 |
| clarkb | nibalizer: oh! did I not accept the ssh key? | 16:50 |
| nibalizer | http://puppetboard.openstack.org/node/subunit-worker02.openstack.org still is not running | 16:50 |
| jroll | jeblair: fyi, https://github.com/rackerlabs/docs-developer-blog/pull/181 | 16:50 |
| nibalizer | clarkb: i accepted the key late last night, so i guess not | 16:51 |
| nibalizer | i'd love to see launch-node.py do better | 16:51 |
| nibalizer | tbh im not sure launch-node.py needs to run puppet agent | 16:51 |
| *** rcernin has quit IRC | 16:51 | |
| nibalizer | like couldn't it just create the server and ansible will get to it when it gets to it? | 16:51 |
| jeblair | nibalizer: yeah, mordred has some wip to do that i think? | 16:51 |
| clarkb | nibalizer: jeblair I would prefer it didn't do that | 16:52 |
| clarkb | because dns | 16:52 |
| clarkb | it is nice to be able to get everything up and running before you have to commit to it | 16:52 |
| fungi | nibalizer: though one of our most frequent discoveries of bootstrapping race issues in our puppet manifests is through seeing new servers fail to stand up completely because of puppet errors | 16:52 |
| *** jcoufal has quit IRC | 16:53 | |
| fungi | and also waiting for puppet to get around to applying on the node you just stood up means you context switch to working on something else for a bit and end-to-end it takes longer | 16:53 |
| clarkb | to clarify updating ssh known hosts is fine | 16:53 |
| *** asettle has joined #openstack-infra | 16:53 | |
| clarkb | but not running puppet would make me sad | 16:53 |
| mordred | yah. I think running puppet is good | 16:54 |
| nibalizer | ok | 16:54 |
| jeblair | oh sorry i hadn't read nibalizer's last msg when i wrote that | 16:54 |
| mordred | obvs running puppet agent is not | 16:54 |
| jeblair | i thought the conversation was about puppet agent/apply | 16:54 |
| jeblair | i agree we should run puppet something :) | 16:54 |
| *** asettle has quit IRC | 16:54 | |
| anteaya | clarkb: I think from channel conversation yesterday we agreed to skip 2 node on new providers until we test that 2 node works | 16:54 |
| nibalizer | our new subunit worker still isn't puppeting | 16:55 |
| clarkb | anteaya: yup I did | 16:55 |
| nibalizer | http://puppetboard.openstack.org/node/subunit-worker02.openstack.org | 16:55 |
| *** kzaitsev_mb has joined #openstack-infra | 16:55 | |
| anteaya | clarkb: I saw 2 node in your patch | 16:55 |
| clarkb | anteaya: it is only added to ubuntu-trusty 2 -node which isn't used anywhere but testing | 16:55 |
| *** scheuran has quit IRC | 16:55 | |
| clarkb | anteaya: whcih will allow us to test that it generally works before committing to it | 16:55 |
| anteaya | oh I'm sorry I didn't realize ubuntu-trusty 2 node is only for testing | 16:55 |
| anteaya | okay great, my mis-understanding | 16:55 |
| clarkb | anteaya: we don't use ubuntu-trusty for devstack things yet | 16:56 |
| clarkb | so that flavor is there to check that the image works with devstack but should also be good test of the overlay networking | 16:56 |
| anteaya | clarkb: we are about to, if I understand the way the wind is blowing today | 16:57 |
| anteaya | AJaeger: did I get that right? | 16:57 |
| clarkb | anteaya: I think its just for python-jobs | 16:57 |
| anteaya | oh is it? | 16:57 |
| AJaeger | anteaya: we move from bare-trusty to ubuntu-trusty | 16:57 |
| anteaya | I haven't been able to keep track | 16:57 |
| AJaeger | anteaya: nothing changes with devstack-trusty | 16:57 |
| *** sc68cal has joined #openstack-infra | 16:57 | |
| nibalizer | fatal: [subunit-worker02.openstack.org]: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply --deta | 16:57 |
| nibalizer | iled-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: -9", "rc": -9, "stderr": "", "stdout": "", "stdout_lines": []} | 16:57 |
| AJaeger | anteaya: at least for now | 16:57 |
| anteaya | to be honest that doesn't aliviate my confusion | 16:57 |
| anteaya | accurate though that maybe | 16:58 |
| *** ihrachys has quit IRC | 16:58 | |
| anteaya | if AJaeger and clarkb are in agreement I guess that will just have to be good enough for me | 16:58 |
| clarkb | anteaya: we have long had two sets of images per distro release. the "bare" and the "devstack" | 16:58 |
| clarkb | anteaya: bare ran all of the non devstack stuff typically nd devstack ran the devstack stuff | 16:58 |
| clarkb | we are wrking to move everything to a single distro release image across the board but are starting with bare which is what is happening today | 16:59 |
| anteaya | thank you | 16:59 |
| clarkb | at some point in the (near I hope) future we will move devstack | 16:59 |
| anteaya | that helps | 16:59 |
| nibalizer | http://paste.openstack.org/show/489361/ would seem to be the errors | 16:59 |
| *** skolekonov is now known as _skolekonov | 16:59 | |
| mtreinish | nibalizer: hmm | 17:00 |
| clarkb | ok time to check on jenkins jobbing | 17:00 |
| nibalizer | also i guess we're not posting | 17:00 |
| *** angdraug has joined #openstack-infra | 17:00 | |
| *** esikachev has quit IRC | 17:00 | |
| clarkb | nibalizer: we don't post on fail iirc | 17:00 |
| clarkb | which sort of makes puppetboard :/ | 17:01 |
| mtreinish | nibalizer: I think that's a bug in puppet-subunit2sql, it should probably be root | 17:01 |
| AJaeger | Is jenkins06 restarted? | 17:01 |
| mtreinish | nibalizer: also I'm not sure why that dir is named logstash, that seems kinda confusing :) | 17:02 |
| max_lobur | yolanda: thanks! | 17:02 |
| *** kmartin has joined #openstack-infra | 17:02 | |
| *** ashleighfarnham has joined #openstack-infra | 17:02 | |
| clarkb | AJaeger: jjb is updating on 01 which should pick up the node change, I will watch it and manualy restart if it times out | 17:02 |
| nibalizer | clarkb: we should be posting on everything | 17:02 |
| *** matrohon has quit IRC | 17:02 | |
| *** jsavak has quit IRC | 17:02 | |
| AJaeger | clarkb: thanks for the update | 17:02 |
| *** sdake_ has joined #openstack-infra | 17:02 | |
| nibalizer | like even if puppet catalog fails, ansible can post that happening to puppetboard | 17:03 |
| *** Jeffrey4l has quit IRC | 17:03 | |
| nibalizer | but in this case it should generate a catalog with some fails then send em in | 17:03 |
| *** cfarquhar has quit IRC | 17:03 | |
| clarkb | nibalizer: it should, but I don't think it does | 17:03 |
| *** sdake has quit IRC | 17:03 | |
| *** amotoki has joined #openstack-infra | 17:03 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Change /etc/logstash dir ownership to root https://review.openstack.org/288592 | 17:03 |
| mtreinish | nibalizer: ^^^ | 17:03 |
| anteaya | AJaeger: yes jenkins06 looks odd in grafana | 17:05 |
| anteaya | AJaeger: fungi said he was powering it off, but I lost what state it was in when he ran out of battery | 17:05 |
| *** yamamoto has joined #openstack-infra | 17:05 | |
| clarkb | he said he got it in shutdown mode | 17:06 |
| *** annegentle has quit IRC | 17:06 | |
| anteaya | ah okay | 17:06 |
| anteaya | so ready to be powered off but not yet powered off | 17:06 |
| *** annegentle has joined #openstack-infra | 17:06 | |
| clarkb | you have to wait for any running jobs to complete first | 17:07 |
| clarkb | but no new jobs will be scheduled there | 17:07 |
| *** jed56 has joined #openstack-infra | 17:07 | |
| *** sdake_ is now known as sdake | 17:07 | |
| anteaya | okay thanks | 17:07 |
| *** jlanoux has quit IRC | 17:07 | |
| anteaya | currently it has 8 nodes in use | 17:08 |
| *** kiran-r has quit IRC | 17:08 | |
| *** bastelfreak has quit IRC | 17:08 | |
| AJaeger | another change to ubuntu-trusty is https://review.openstack.org/288572 - project-config cores and pabelanger , what do you think? | 17:09 |
| *** bastelfreak has joined #openstack-infra | 17:09 | |
| *** vincentll has quit IRC | 17:09 | |
| *** hdd has joined #openstack-infra | 17:10 | |
| *** Sukhdev has quit IRC | 17:10 | |
| clarkb | AJaeger: +2 | 17:10 |
| AJaeger | thanks, clarkb | 17:10 |
| *** nwkarsten has joined #openstack-infra | 17:11 | |
| *** annegentle has quit IRC | 17:11 | |
| clarkb | and jjb is still running on 01 and 07 happy so far | 17:11 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Convert specs-jobs to ubuntu-trusty https://review.openstack.org/288596 | 17:13 |
| anteaya | AJaeger: I'm not sure where we are in the workflow so you can self approve when you are ready | 17:14 |
| *** dtardivel has joined #openstack-infra | 17:14 | |
| *** yamamoto has quit IRC | 17:15 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Display a modal when a card is clicked on https://review.openstack.org/284280 | 17:15 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal to confirm archiving cards https://review.openstack.org/285417 | 17:15 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Add Due Dates to boards https://review.openstack.org/284279 | 17:15 |
| *** degorenko is now known as _degorenko|afk | 17:17 | |
| *** bpokorny has quit IRC | 17:17 | |
| *** sridhar_ram has joined #openstack-infra | 17:17 | |
| clarkb | AJaeger: jjb is done on 01 and 07 now, I am rerunning on 01 really quickly to see if it needs to be redone | 17:18 |
| AJaeger | clarkb: wow! | 17:18 |
| *** bpokorny has joined #openstack-infra | 17:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Training-labs: Rework jobs https://review.openstack.org/288597 | 17:18 |
| AJaeger | anteaya: we can merge anytime unless it causes a problem for clarkb with jenkins... | 17:19 |
| * AJaeger will be back in an hour or so | 17:19 | |
| clarkb | go for it | 17:19 |
| *** bpokorny has quit IRC | 17:19 | |
| clarkb | I think jjb updated just fine, but doing my best to confirm | 17:19 |
| pabelanger | AJaeger: LGTM | 17:21 |
| dimsum_ | fungi : anteaya : looks like there's a fix! https://github.com/etsy/hound/issues/113 | 17:21 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Convert windmill jobs to bindep https://review.openstack.org/288600 | 17:22 |
| *** nwkarsten has quit IRC | 17:23 | |
| clarkb | I just checked gate-zaqar-python27 and it is not updated so I will be starting some manual updates now | 17:24 |
| nibalizer | mordred: so in ansible-puppet it kinda looks like if we fail puppet, we don't even try to post logs | 17:24 |
| clarkb | unless we are getting that other change in I can wait for it to merge | 17:24 |
| nibalizer | i can't see any obvious reason for this unless its ansibles way to stop a playbook after something fails | 17:24 |
| openstackgerrit | Khai Do proposed openstack-infra/git-review: my Fix one typo on git-review documentation https://review.openstack.org/288601 | 17:24 |
| clarkb | nibalizer: yes that is why | 17:24 |
| nibalizer | well... is there a way around that? | 17:25 |
| *** ashleighfarnham has quit IRC | 17:25 | |
| clarkb | lets hold off on approving 288596 so I can restart jjb and get these other jobs updated | 17:25 |
| clarkb | nibalizer: I think you can set a thing that says fail is ok | 17:25 |
| clarkb | I did it in the jenkins restart playbook iirc | 17:25 |
| anteaya | dimsum_: well the bug was filed | 17:25 |
| clarkb | (for jenkins stop) | 17:25 |
| anteaya | dimsum_: the fix seems to be The right answer here, I think, is to actually throttle indexing to ensure that no more than N searchers are reindexing at a time. | 17:25 |
| anteaya | I'm not sure how well throttling indexing will work for us | 17:26 |
| dimsum_ | anteaya : scroll down to the 2 merged PR's | 17:26 |
| bkero | nibalizer: ignore_errors: yes as a metaparam to resources | 17:26 |
| jamespage | if anyone has a moment I could do with a second review of https://review.openstack.org/#/c/287676/ ontop of yolandas ... | 17:26 |
| nibalizer | it looks like we can ignore_errors: True | 17:26 |
| *** bpokorny has joined #openstack-infra | 17:27 | |
| nibalizer | but that wuold mean we wouldn't see in the ansible logs when puppet failed | 17:27 |
| *** Sukhdev has joined #openstack-infra | 17:27 | |
| *** gnuoy_ has joined #openstack-infra | 17:27 | |
| odyssey4me | AJaeger how does the ZUUL_REFNAME get constructed? If I have a review, how do I determine the ZUUL_REFNAME - I'm trying to follow how the conditionals in https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-docs.sh flow | 17:28 |
| anteaya | dimsum_: okay so the fix appears to have merged a year ago | 17:29 |
| clarkb | odyssey4me: the only way to know what it is is to get it from zuul, its not deterministic and is a random uuid iirc | 17:29 |
| *** gnuoy_ has quit IRC | 17:31 | |
| *** dtantsur is now known as dtantsur|afk | 17:32 | |
| odyssey4me | clarkb blast :/ | 17:33 |
| *** harlowja_at_home has joined #openstack-infra | 17:33 | |
| *** annegentle has joined #openstack-infra | 17:33 | |
| *** yamahata has quit IRC | 17:33 | |
| odyssey4me | clarkb under what conditions would ZUUL_REFNAME have 'refs/tags/' in it? is the ZUUL_REFNAME the same as build_ref in logstash? | 17:34 |
| *** _nadya_ has quit IRC | 17:34 | |
| clarkb | oh wait its REFNAME you were asking for not REF | 17:35 |
| clarkb | ZUUL_REF is not deterministic but REFNAME may be | 17:35 |
| clarkb | clearly I need more caffeine | 17:35 |
| clarkb | odyssey4me: the REFNAME should be the branch name (master, stable/kilo, etc) wen in check or gate and I think it is that in post as well | 17:36 |
| clarkb | in release it is the tag that triggered the build | 17:36 |
| odyssey4me | clarkb so if I push a tag to a branch, does it trigger a docs build? | 17:37 |
| *** daemontool_ has joined #openstack-infra | 17:37 | |
| clarkb | odyssey4me: if that project has a docs job in the release pipeline | 17:37 |
| * odyssey4me hands clarkb some coffee | 17:37 | |
| *** bpokorny has quit IRC | 17:37 | |
| odyssey4me | :) | 17:37 |
| *** austin81 has joined #openstack-infra | 17:37 | |
| clarkb | for most projects I think this is the case | 17:37 |
| odyssey4me | clarkb ok, by release pipeline do you mean the release jobs managed by the release team - or a specific review-based process? | 17:38 |
| odyssey4me | OpenStack-Ansible is not using that process at this point - we use the 'git tag, git push' mechanism | 17:39 |
| clarkb | odyssey4me: in zuul you configure explicitly what you want to happen on different events. check runs on new patchset, gate on approval, post on merge, release on tag pushed | 17:39 |
| *** esikachev has joined #openstack-infra | 17:39 | |
| openstackgerrit | Merged openstack/requirements: Bump openstacksdk version to 0.8.1 https://review.openstack.org/285599 | 17:40 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Training-labs: Use ubuntu-trusty https://review.openstack.org/288597 | 17:40 |
| *** kzaitsev_mb has quit IRC | 17:40 | |
| odyssey4me | clarkb ok, we're using the template jobs for docs so I suspect we're inheriting the release docs job when a tag is pushed | 17:40 |
| *** Guest63919 has quit IRC | 17:41 | |
| *** daemontool has quit IRC | 17:41 | |
| *** ccrouch1 has joined #openstack-infra | 17:41 | |
| odyssey4me | clarkb perhaps you can confirm? https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L7498-L7512 | 17:42 |
| *** ccrouch has quit IRC | 17:43 | |
| odyssey4me | it would seem so: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L411-L415 | 17:43 |
| *** bpokorny has joined #openstack-infra | 17:44 | |
| clarkb | odyssey4me: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L411 yes that | 17:44 |
| *** fabio_ has quit IRC | 17:44 | |
| odyssey4me | ok, so it would seem then that when I tagged earlier today, we hit this conditional: https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-docs.sh#L32 | 17:45 |
| *** Daisy has joined #openstack-infra | 17:45 | |
| *** bpokorny has quit IRC | 17:45 | |
| odyssey4me | and this would have been hit, as the job is a tags only job for a tag release: https://github.com/openstack-infra/project-config/blob/master/jenkins/scripts/run-docs.sh#L48 | 17:45 |
| *** bpokorny has joined #openstack-infra | 17:45 | |
| odyssey4me | clarkb to clue you in, I'm troubleshooting an issue where the tag release replaced the published docs for the master branch with the docs from the tagged build | 17:46 |
| clarkb | that script is set -x so your job log should confirm | 17:46 |
| *** jistr has quit IRC | 17:46 | |
| clarkb | I am going to guess it incorrectly determined the latest tag? | 17:46 |
| odyssey4me | ie http://docs.openstack.org/developer/openstack-ansible/ which should contain https://github.com/openstack/openstack-ansible/blob/master/doc/source/index.rst ended up with the content destined for http://docs.openstack.org/developer/openstack-ansible/12.0.7/ | 17:47 |
| odyssey4me | ok, where do I find that job log - that would be most helpful! | 17:47 |
| clarkb | odyssey4me: http://logs.openstack.org/first2oftagsha1/tagsha1 | 17:48 |
| openstackgerrit | Merged openstack-infra/project-config: Convert infra* jobs to ubuntu-trusty https://review.openstack.org/288572 | 17:48 |
| *** Thelo has quit IRC | 17:49 | |
| clarkb | it looks like 12.0.7 is the latest tag | 17:49 |
| clarkb | so the script did what it is supposed to | 17:49 |
| odyssey4me | clarkb so therefore http://logs.openstack.org/5b/5b09e4fcc6fca2a6a7887d2522126b695eb5ba9e/ | 17:49 |
| *** Daisy has quit IRC | 17:50 | |
| odyssey4me | yeah, it did publish the tag - but also overwrote the master content | 17:50 |
| *** Thelo has joined #openstack-infra | 17:50 | |
| *** ccrouch1 has quit IRC | 17:50 | |
| clarkb | yes that is what it is supposed to do, see the comment above that last line you linked | 17:50 |
| *** ccrouch has joined #openstack-infra | 17:50 | |
| *** esikachev has quit IRC | 17:51 | |
| odyssey4me | ah, this is a bad assumption - or we're making a bad assumption | 17:51 |
| *** kencjohnston has quit IRC | 17:52 | |
| clarkb | the intent is that you tag as you go so the biggest number tag is always recent and should go to / | 17:52 |
| *** ccrouch has quit IRC | 17:52 | |
| *** salv-orlando has joined #openstack-infra | 17:52 | |
| odyssey4me | yeah, whereas we want the master content in / and are happy to have tagged content in the respective sub-URL | 17:52 |
| *** jpr has quit IRC | 17:53 | |
| *** yamamoto has joined #openstack-infra | 17:53 | |
| odyssey4me | I'm not quite sure what the use-case would be for not having the current HEAD in / - thoughts? | 17:53 |
| *** lucasagomes is now known as lucas-beer | 17:54 | |
| *** lezbar has quit IRC | 17:54 | |
| clarkb | odyssey4me: its for projects that don't publish current head | 17:54 |
| clarkb | like libraries that are intended to be consumed via releaess | 17:54 |
| odyssey4me | hmm, fair enough | 17:55 |
| *** salv-orl_ has quit IRC | 17:55 | |
| *** gyee has joined #openstack-infra | 17:55 | |
| *** ccrouch has joined #openstack-infra | 17:55 | |
| odyssey4me | so now the trick is to figure out how to do this a little differently - to cater for both use cases | 17:55 |
| clarkb | I think its already an option you just have to change your job? | 17:55 |
| clarkb | maybe? | 17:55 |
| odyssey4me | happy to do that - just need to figure out how | 17:56 |
| *** leakypipes has quit IRC | 17:56 | |
| odyssey4me | I guess I just need to disable the 'tags-only' bit | 17:56 |
| clarkb | odyssey4me: the trivial option is replace that job template with just a post: '{name}-docs entry then all updates to all branches will get publishes as you go | 17:57 |
| odyssey4me | so do I set a release job to override https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L411-L415 ? | 17:57 |
| clarkb | odyssey4me: instead of /12.0.7 you will have /stable/foo | 17:57 |
| *** dimtruck is now known as zz_dimtruck | 17:58 | |
| odyssey4me | so I'm suggesting that I just add release: '{name}-docs' to https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L7498-L7512 | 17:58 |
| odyssey4me | will that do the trick I wonder? | 17:58 |
| odyssey4me | hmm, I think we're both saying the same thing? | 17:59 |
| clarkb | ya I think so | 17:59 |
| *** Swami_ has joined #openstack-infra | 17:59 | |
| *** Swami has joined #openstack-infra | 17:59 | |
| mordred | clarkb: any chance you can +A this https://review.openstack.org/#/c/288110/ ? | 18:01 |
| anteaya | dimsum_: I'm trying to bring up a vm with puppet-hound on it | 18:01 |
| clarkb | mordred: I am still trying t ofigure out why we need the special case for unittesting | 18:01 |
| clarkb | mordred: won't Munch(obj) return a thing that is useable? | 18:01 |
| anteaya | dimsum_: I don't have a lot of experience working with puppet modules, so have low expectations | 18:01 |
| *** ashleighfarnham has joined #openstack-infra | 18:02 | |
| anteaya | dimsum_: I'm trying to see if we have that patch and how to set it as the readme suggests | 18:02 |
| *** geekinutah has joined #openstack-infra | 18:02 | |
| *** esker has quit IRC | 18:02 | |
| mordred | clarkb: I fix that in the two follow up patches | 18:02 |
| *** electrofelix has quit IRC | 18:03 | |
| mordred | clarkb: I left them as follow up patches becuse I didn't wnat this one to be too big | 18:03 |
| clarkb | mordred: I think you just remove that code and you are done? | 18:03 |
| clarkb | its less code not more | 18:03 |
| mordred | clarkb: https://review.openstack.org/#/c/288502 and https://review.openstack.org/#/c/288503 | 18:03 |
| mordred | clarkb: no - it's there because we had some places in the test code where we were mocking glance objects wrong | 18:04 |
| mordred | so they were traversing that function incorrectly | 18:04 |
| mordred | the unittest special case was a get-out-of-jail care to skip having to fix the unit tests | 18:04 |
| mordred | but then I fixed the unit tests | 18:04 |
| *** yamahata has joined #openstack-infra | 18:04 | |
| clarkb | ok I am going to review them in opposite order :) | 18:08 |
| *** jpr has joined #openstack-infra | 18:08 | |
| *** derekh has quit IRC | 18:11 | |
| *** notnownikki has quit IRC | 18:12 | |
| *** Keedya_4 has joined #openstack-infra | 18:13 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Make OpenStack-Ansible publish master docs https://review.openstack.org/288621 | 18:13 |
| odyssey4me | clarkb ^ does that do what I think it does? | 18:14 |
| clarkb | odyssey4me: close, let me comment | 18:14 |
| *** yamamoto has quit IRC | 18:14 | |
| *** yamamoto has joined #openstack-infra | 18:15 | |
| clarkb | odyssey4me: posted | 18:15 |
| clarkb | mordred: stack approved | 18:16 |
| *** Keedya_4 has quit IRC | 18:16 | |
| *** zz_dimtruck is now known as dimtruck | 18:17 | |
| *** harlowja_at_home is now known as hollaja | 18:17 | |
| mordred | clarkb: thanks | 18:17 |
| mordred | clarkb: once those land, we'll cut a new shade | 18:17 |
| *** sarob has joined #openstack-infra | 18:19 | |
| *** piet has quit IRC | 18:19 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job https://review.openstack.org/273743 | 18:20 |
| sdague | hmmm... vexxhost is failing single node grenade a lot | 18:21 |
| *** tongli has quit IRC | 18:21 | |
| *** piet has joined #openstack-infra | 18:21 | |
| sdague | clarkb: logstash seems to be missing a bunch of data and the queue is this crazy roller coaster, any idea what' | 18:23 |
| sdague | s going up? | 18:23 |
| clarkb | no | 18:23 |
| *** nwkarsten has joined #openstack-infra | 18:24 | |
| clarkb | what data is missing? | 18:25 |
| *** yamamoto has quit IRC | 18:25 | |
| *** jordanP has quit IRC | 18:25 | |
| sdague | it looks like part of the issue is it's 3+ hours behind again | 18:26 |
| clarkb | roller coaster is often a side effect of trying to index very large files | 18:26 |
| sdague | I know mriedem wasn't able to find some fail runs | 18:26 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Make OpenStack-Ansible publish master docs https://review.openstack.org/288621 | 18:26 |
| sdague | ok, well do we have any idea what those are? | 18:26 |
| clarkb | they get into queue and back it up then once past everything jumps ahead again | 18:27 |
| odyssey4me | thanks clarkb - feedback incorporated | 18:27 |
| clarkb | sdague: you can query for it | 18:27 |
| sdague | clarkb: how? | 18:27 |
| clarkb | sdague: by grouping on job uuid and file name and counting results | 18:27 |
| sdague | ok, I need to write a custom tool? | 18:28 |
| *** amotoki has quit IRC | 18:28 | |
| clarkb | not necessarily, I think the api will give you that | 18:28 |
| odyssey4me | oh, that's weird - since earlier today I'm seeing blank console logs - for example: http://logs.openstack.org/06/288006/5/check/gate-openstack-ansible-lxc_hosts-ansible-func-ubuntu-trusty/d12d13f/console.html | 18:28 |
| sdague | but a custom tool against that API | 18:28 |
| odyssey4me | pabelanger ^ | 18:28 |
| *** nwkarsten has quit IRC | 18:28 | |
| clarkb | I think its one request piped to soet | 18:28 |
| AJaeger | clarkb: reading backscroll, I'm confused about the docs publishing. | 18:28 |
| clarkb | *sort | 18:28 |
| *** kencjohnston has joined #openstack-infra | 18:30 | |
| sdague | that does assume that every line is the same cost. Is there a way to log it on the server so we can know real cost. Like the ansible runs that put everything in one line? | 18:30 |
| clarkb | sdague: we can look at time stamps in the server logs, but the way logstash works is num lines is basically cost | 18:31 |
| AJaeger | clarkb: https://review.openstack.org/#/c/279238/ should have fixed the publishing... | 18:31 |
| clarkb | since each line is processed individually | 18:31 |
| pabelanger | odyssey4me: AJaeger: I wonder if that is related to curl | 18:31 |
| clarkb | AJaeger: I thibk it did except that openstack-ansible doesnt tag master | 18:31 |
| sdague | clarkb: ok, well it's beyond scope for what I can bite off atm. | 18:33 |
| AJaeger | clarkb: let me check the log file, this confuses me... | 18:33 |
| *** Sukhdev has quit IRC | 18:33 | |
| *** zeih_ has joined #openstack-infra | 18:33 | |
| *** ccrouch has quit IRC | 18:33 | |
| AJaeger | pabelanger: you mean we do not have curl on the ubuntu-trusty? | 18:33 |
| pabelanger | AJaeger: Ya, I noted that yesterday | 18:34 |
| clarkb | https://www.elastic.co/guide/en/elasticsearch/reference/1.7/search-aggregations.html | 18:34 |
| clarkb | should work with ehat we allow through the api | 18:34 |
| clarkb | since it is under the search api | 18:34 |
| AJaeger | pabelanger: but it's in jenkins/data/bindep-fallback.txt, isn't it? | 18:35 |
| AJaeger | So, why is it not installed? | 18:35 |
| pabelanger | AJaeger: So, the issue is with other-requirements.txt | 18:35 |
| *** eharney has quit IRC | 18:35 | |
| AJaeger | pabelanger: so, odyssey4me needs to add it, doesn't he? | 18:35 |
| pabelanger | AJaeger: So, in this case is is JJB that needs curl, not ansible | 18:35 |
| clarkb | oh if we use it un the job then we should ship it | 18:35 |
| pabelanger | so, we need to figure out a good way to add JJB dependencies otherwise each project will need to add them into other-requirements.txt | 18:36 |
| * AJaeger cannot think straight right now, have patience with me | 18:36 | |
| clarkb | just install at job build | 18:36 |
| clarkb | er imsge build | 18:36 |
| odyssey4me | :) | 18:36 |
| pabelanger | clarkb: Yup, just finding a place to install it | 18:37 |
| *** bpokorny has quit IRC | 18:37 | |
| anteaya | AJaeger: wow, if you have a moment of confusion | 18:37 |
| anteaya | the rest of us poor mortals don't have a chance | 18:37 |
| *** zeih_ has quit IRC | 18:38 | |
| *** treaki has joined #openstack-infra | 18:38 | |
| *** piet has quit IRC | 18:38 | |
| AJaeger | anteaya: two problems here plus fun with my kids - and the end of the week... | 18:38 |
| anteaya | yup | 18:39 |
| *** vgridnev has quit IRC | 18:39 | |
| anteaya | I hear that | 18:39 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Ensure curl is installed for jenkins slaves https://review.openstack.org/288634 | 18:40 |
| pabelanger | odyssey4me: AJaeger: clarkb ^ | 18:40 |
| *** sbalukoff has quit IRC | 18:40 | |
| *** ccrouch has joined #openstack-infra | 18:40 | |
| AJaeger | clarkb: regarding docs publishing, I wonder whether we mixed the cases. | 18:41 |
| *** pcaruana has quit IRC | 18:41 | |
| AJaeger | clarkb: For server projects: We publish after each commit and also each tag - but tags should go only to a subdir | 18:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Ensure curl is installed for jenkins slaves https://review.openstack.org/288634 | 18:41 |
| *** piet has joined #openstack-infra | 18:41 | |
| clarkb | AJaeger: oh if that is the intent then I guess we can fix it directly in the script | 18:41 |
| clarkb | by not having any of the special sauce | 18:42 |
| openstackgerrit | Merged openstack-infra/shade: Fixes for latest cinder and neutron clients https://review.openstack.org/288110 | 18:42 |
| AJaeger | clarkb, odyssey4me: For client projects: We publish only with releases/tags - and should publish to the main dir in general and a sub dir | 18:42 |
| clarkb | oh right | 18:42 |
| AJaeger | clarkb: see my last change here https://review.openstack.org/#/c/279238/4/jenkins/scripts/run-docs.sh | 18:42 |
| *** bpokorny has joined #openstack-infra | 18:42 | |
| clarkb | so maybe what we need to do is update the template dto do what odyssey4me's change does | 18:42 |
| clarkb | for not clients | 18:42 |
| mriedem | sdague: clarkb: the cinder console logs were pretty massive for unit test runs until this week, but not sure if that would tip things over | 18:43 |
| AJaeger | I think we need to move the tags job from server to client in https://review.openstack.org/#/c/279238/4/zuul/layout.yaml | 18:43 |
| *** jamesmcarthur has quit IRC | 18:43 | |
| clarkb | AJaeger: yes I agree | 18:43 |
| clarkb | odyssey4me: this may be a larger issue that we fix for you :) | 18:43 |
| clarkb | mriedem: it really can | 18:43 |
| clarkb | mriedem: basically what seems to happen is expensive files come in and back up because they don't flush quickly | 18:44 |
| *** achanda has joined #openstack-infra | 18:44 | |
| clarkb | then once they flush everything is quick until the next high cost files back it up again | 18:44 |
| AJaeger | odyssey4me, clarkb: who of us will fix this? | 18:44 |
| *** ccrouch has quit IRC | 18:45 | |
| clarkb | AJaeger: I am trying to make sure jjb updates happen so probably best if it isn't me | 18:45 |
| *** cznewt has quit IRC | 18:45 | |
| clarkb | pabelanger: I +2'd and can approve if no one else reviews that soon | 18:45 |
| pabelanger | clarkb: sure, just waiting for jenkins to +1 the syntax | 18:46 |
| clarkb | mriedem: sdague es says cluster is red so now trying to figure out why | 18:47 |
| *** mriedem has quit IRC | 18:47 | |
| clarkb | which is also likely not helping | 18:47 |
| geekinutah | fungi, anteaya: for context, the Mike Wilson AndyU mentions is me | 18:47 |
| anteaya | geekinutah: ah wonderful, thank you for context | 18:47 |
| anteaya | geekinutah: so I think that from yesterday's chat if we could start with one node we can run tests and give you feedback | 18:48 |
| anteaya | and then you would have something more to go on | 18:48 |
| geekinutah | yeah, I am working on that | 18:48 |
| anteaya | when you have a node | 18:48 |
| anteaya | cool, thank you | 18:48 |
| geekinutah | but I seriously doubt there will be any problems | 18:48 |
| *** mriedem has joined #openstack-infra | 18:48 | |
| clarkb | mriedem: sdague curl -XGET 'http://logstash.openstack.org/elasticsearch/_cluster/health?pretty=true' if yo uwant to check that yourselves | 18:48 |
| odyssey4me | AJaeger clarkb if it's simply switching the templates around, I'm happy to submit the patch | 18:48 |
| pleia2 | argh, devstack | 18:48 |
| anteaya | I afirm you doubt there will be problems | 18:48 |
| clarkb | mriedem: sdague maybe we can add that to the e-r banner? | 18:48 |
| AJaeger | odyssey4me: go for it ;) | 18:48 |
| mriedem | clarkb: i'll run it | 18:49 |
| anteaya | geekinutah: for context we have never had a cloud that is problem free | 18:49 |
| anteaya | geekinutah: but who knows yours might be the first | 18:49 |
| clarkb | odyssey4me: I think you need to update th eexisting template to use {name}-docs instead of the tags only one | 18:49 |
| anteaya | pleia2: hey there, feeling better today? | 18:49 |
| clarkb | odyssey4me: and update the other template for clients to use tags only | 18:49 |
| pleia2 | anteaya: hanging in there | 18:49 |
| geekinutah | anteaya: of course, what I mean is OpenStack problem free | 18:49 |
| anteaya | pleia2: glad to hear it | 18:49 |
| geekinutah | meaning it will work with some tinkering :-) | 18:49 |
| anteaya | geekinutah: ah | 18:49 |
| pleia2 | I'm going to need some help with https://review.openstack.org/#/c/274701/ at some point (this is for the i18n-checksite spec) | 18:50 |
| anteaya | your expectations are coming into convergence with my experience | 18:50 |
| anteaya | in so much as watching others work counts as experience | 18:50 |
| *** doug-fis_ is now known as doug-fish | 18:50 | |
| pleia2 | running through this puppet module again in testing, but we have some infra-side problems to solve that I'm not sure about | 18:50 |
| pleia2 | like the fact that our nodes don't know their IP address, and devstack wants to know its ip address | 18:51 |
| openstackgerrit | Merged openstack-infra/shade: Use warlock in the glance v2 tests https://review.openstack.org/288502 | 18:51 |
| pleia2 | s/IP/public IP | 18:51 |
| nibalizer | can another core look at https://review.openstack.org/#/c/288592/1 please? | 18:51 |
| armax | AJaeger: ping | 18:51 |
| pleia2 | hm, actually, the rackspace ones do | 18:51 |
| *** hollaja has quit IRC | 18:52 | |
| anteaya | pleia2: I am no help to you unfortunately | 18:52 |
| clarkb | nibalizer: mtreinish approved but see comment | 18:52 |
| *** cznewt has joined #openstack-infra | 18:52 | |
| *** harlowja_at_home has joined #openstack-infra | 18:52 | |
| anteaya | I'm trying to figure out how to get the puppet-hound module to run | 18:52 |
| nibalizer | clarkb: ya | 18:53 |
| anteaya | so far puppet apply ../init.pp aren't the magic words | 18:53 |
| clarkb | pleia2: "infra-side problems" is code for "cloud design problems"? :) | 18:53 |
| mriedem | clarkb: so that says status is red but i'm not sure how to interpret why | 18:53 |
| clarkb | mriedem: I think it is because of the 3 unassigned shards | 18:53 |
| pleia2 | anteaya: you probably want to create a local.pp that has info from our codesearch.pp in system-config | 18:53 |
| clarkb | mriedem: but my local networking si very poor today and making it hard to follow up on that | 18:53 |
| pleia2 | anteaya: then follow instructions here: http://docs.openstack.org/infra/system-config/sysadmin.html#making-a-change-in-puppet | 18:54 |
| anteaya | pleia2: oh, I can try that | 18:54 |
| anteaya | oh look, documentation | 18:54 |
| anteaya | pleia2: thank you | 18:54 |
| pleia2 | clarkb: heh, yes, "this puppet module works fine in testing" :) | 18:54 |
| AJaeger | what's up armax ? | 18:54 |
| mriedem | ah http://stackoverflow.com/questions/19967472/elasticsearch-unassigned-shards-how-to-fix | 18:54 |
| clarkb | mriedem: it looks like elasticsearch07 might have gone away and is now recovering those shards | 18:54 |
| clarkb | they belong to two indexes | 18:54 |
| clarkb | mriedem: you can get all this in really ugly json fro mthe api :/ | 18:55 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Set tags-only docs publishing for client jobs, not server jobs https://review.openstack.org/288644 | 18:55 |
| armax | AJaeger: I wonder if it’s safe to pull the trigger on the chain of patches pertaining the constraints suffix drop on neutron jobs | 18:55 |
| armax | how long are you gonna be around? | 18:56 |
| odyssey4me | clarkb AJaeger please see if https://review.openstack.org/288644 is right? | 18:56 |
| jesusaur | clarkb: you may already know this, but i recently learned that you can do http://logstash.openstack.org/elasticsearch/_cluster/health?pretty&level=indices | 18:56 |
| AJaeger | armax: another hour - let me review them in a few minutes | 18:56 |
| armax | AJaeger: ok | 18:56 |
| clarkb | jesusaur: I didn not know that | 18:57 |
| clarkb | looks like the last time es07 logged anything was february 19th | 18:57 |
| clarkb | I may try bouncing it and seeing if that unsticks things | 18:58 |
| AJaeger | could anothoer project config core help getting rid of constraints, please? https://review.openstack.org/#/c/286777/ and https://review.openstack.org/#/c/288506/2 - I'll review the stack as well after looking at odyssey4me's change | 18:58 |
| clarkb | (this is a really interesting fail mode becausees07 is primary for those shards but it is derping so there is no replica for it either) | 18:58 |
| *** ccrouch has joined #openstack-infra | 18:58 | |
| anteaya | AJaeger: those patches are both waiting on your neutron patch to merge | 19:00 |
| clarkb | wow and of course when you restart an unhappy server all of the indexes it knew about that have since been deleted show back up again | 19:01 |
| clarkb | this is unfun but I will delete those old thing sthat have been deleted once already | 19:02 |
| anteaya | that is interesting | 19:02 |
| *** nwkarsten has joined #openstack-infra | 19:02 | |
| jesusaur | clarkb: yeah, a manual run of the curator cron will take care of that though | 19:02 |
| clarkb | jesusaur: yup just did that | 19:03 |
| openstackgerrit | Merged openstack-infra/puppet-subunit2sql: Change /etc/logstash dir ownership to root https://review.openstack.org/288592 | 19:03 |
| clarkb | its just an interesting way to rejoin a cluster | 19:03 |
| clarkb | "hey master you don't know anything about these indexes but I do therefore you are going to learn about them as fast as I can teach you | 19:03 |
| clarkb | rather than "hey master node what should I care about" | 19:03 |
| *** annegent_ has joined #openstack-infra | 19:04 | |
| *** yaume has quit IRC | 19:04 | |
| AJaeger | anteaya: my original neutron change, it has been changed significantly... | 19:04 |
| odyssey4me | pabelanger AJaeger so the workaround at this stage for the blank console file (due to 'curl' not being present on the d-i-b image) is to include 'curl' as a package in 'other-requirements.txt'? | 19:04 |
| anteaya | I'm sorry I never have gotten my head around constraints | 19:04 |
| odyssey4me | can I submit a patch somewhere to ensure that curl is included in the built image? | 19:05 |
| anteaya | so I can look at your patches but I have a hard time understanding what is happening | 19:05 |
| anteaya | sorry for being the slow person in the room | 19:05 |
| pabelanger | odyssey4me: yes | 19:05 |
| clarkb | mriedem: sdague cluster is green now, though it is rebalancing | 19:05 |
| clarkb | lets see if this gets it happier | 19:06 |
| jesusaur | clarkb: yeah, but that same mode is what lets us do a full cluster restart without losing data | 19:06 |
| clarkb | jesusaur: in theory the "minimum data nodes" param should pretect against that | 19:07 |
| *** annegentle has quit IRC | 19:07 | |
| *** ddieterly has joined #openstack-infra | 19:07 | |
| clarkb | jesusaur: I have it set to n-1 bceause we have n+1 redundancy | 19:07 |
| mriedem | clarkb: cool | 19:08 |
| clarkb | I don't see anything in dmesg or the es log that would indicate what went wrong | 19:08 |
| clarkb | I am guessing some es process internal state went wonky | 19:09 |
| *** isq has joined #openstack-infra | 19:09 | |
| *** isq has quit IRC | 19:09 | |
| jesusaur | disks aren't full or anything like that? | 19:09 |
| clarkb | jesusaur: nope I checked that, had 80% free on the 1TB volume | 19:09 |
| jesusaur | it looks like the mounted volume isnt in cacti | 19:09 |
| *** isq has joined #openstack-infra | 19:09 | |
| AJaeger | armax: is ihrachsys still around? If not, I'll fix his change... | 19:09 |
| *** ihrachys has joined #openstack-infra | 19:10 | |
| armax | I think he’s out | 19:10 |
| clarkb | jesusaur: huh it is mounted | 19:10 |
| armax | AJaeger: even though he’s still lurking around | 19:10 |
| armax | ihrachys: ^ | 19:10 |
| clarkb | pabelanger: AJaeger odyssey4me do we need to revert the openstack-ansible jobs use ubuntu-trusty change due to the lack of curl? | 19:12 |
| clarkb | then merge that once we have updated image swith curl? | 19:12 |
| pabelanger | clarkb: I'll let odyssey4me make the call for ansible-jobs | 19:12 |
| clarkb | pabelanger: ok | 19:12 |
| AJaeger | clarkb: at which point is puppet applied for these images? | 19:13 |
| AJaeger | build time - or run time? | 19:13 |
| *** noorul has joined #openstack-infra | 19:13 | |
| pabelanger | build | 19:13 |
| *** esikachev has joined #openstack-infra | 19:13 | |
| mtreinish | clarkb: sure, I can push a change to change /etc/logstash to /etc/subunit2sql or something | 19:13 |
| clarkb | nibalizer: mordred pleia2 other infra-root https://review.openstack.org/#/c/288634/2 needs to get in for curl on images | 19:13 |
| *** ociuhandu has quit IRC | 19:13 | |
| clarkb | AJaeger: pabelanger yup build, so my plan is to manually rebuild that image once ^ merges | 19:13 |
| pabelanger | Ah | 19:13 |
| pabelanger | okay, so my vote is land the patch and rebuild | 19:13 |
| *** eharney has joined #openstack-infra | 19:13 | |
| *** pvaneck has joined #openstack-infra | 19:14 | |
| *** kencjohnston has quit IRC | 19:14 | |
| odyssey4me | pabelanger no, we'll get curl added in to other-requirements.txt for now cc clarkb AJaeger | 19:14 |
| AJaeger | armax, ihrachys : updated - the neutron change looks fine to me now. | 19:14 |
| armax | AJaeger: ok, let’s get this deferred until first thing monday morning | 19:14 |
| clarkb | odyssey4me: ok, you should be able to pull that out by monday (I definitely want the fixes to be in place before the weekend) | 19:15 |
| odyssey4me | I do think that seeing as Jenkins requires curl to be there, it should be in the image - if you can point me to where the base package list is defined, I'll patch it up | 19:15 |
| AJaeger | armax: sure | 19:15 |
| clarkb | odyssey4me: the patch is written, just need to get it merged and rebuild the image | 19:15 |
| odyssey4me | clarkb do you have a review link - I'd like to reference it | 19:15 |
| *** yamamoto has joined #openstack-infra | 19:16 | |
| clarkb | odyssey4me: https://review.openstack.org/#/c/288634/2 | 19:16 |
| odyssey4me | thanks | 19:16 |
| clarkb | pabelanger: you should also remove the curl packages from thick_slave.pp in a follow up | 19:16 |
| AJaeger | armax: 286777 has two +2s and a +A - once the neutron change is in, we need to toggle the +A to merge - should be quickly done... | 19:16 |
| clarkb | pabelanger: so that they bare-* builds don't break while we transition | 19:16 |
| armax | AJaeger: ack | 19:16 |
| *** noorul has quit IRC | 19:17 | |
| mordred | Shrews: is this similar to the races you saw recently? http://logs.openstack.org/03/288503/2/gate/gate-shade-dsvm-ansible/1d74222/console.html | 19:17 |
| *** hashar has joined #openstack-infra | 19:17 | |
| pabelanger | clarkb: Oh, missed that | 19:17 |
| pabelanger | checking now | 19:17 |
| *** dtardivel has quit IRC | 19:18 | |
| *** rcarrillocruz has left #openstack-infra | 19:18 | |
| *** ccrouch1 has joined #openstack-infra | 19:18 | |
| Shrews | mordred: nope. that just looks like a timeout | 19:18 |
| ihrachys | AJaeger: thanks for respin. I will also kill venv-constraints target completely in that patch, it is probably already orphan and not used anywhere anyway. | 19:19 |
| *** rcarrillocruz has joined #openstack-infra | 19:19 | |
| *** _nadya_ has joined #openstack-infra | 19:20 | |
| *** ccrouch has quit IRC | 19:20 | |
| AJaeger | ihrachys: kill it in the cleanup patch once the infra changes are in... | 19:20 |
| *** yamamoto has quit IRC | 19:21 | |
| ihrachys | AJaeger: ack, let's leave it that way | 19:21 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches https://review.openstack.org/288506 | 19:21 |
| AJaeger | ihrachys: I just updated this project-config change ^ | 19:21 |
| AJaeger | ihrachys: should be fine now - I hope ;) | 19:21 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 19:21 |
| AJaeger | ihrachys: thanks for splitting these up and reworking! | 19:21 |
| *** apoorvad has quit IRC | 19:22 | |
| ihrachys | AJaeger: yeah, I still don't trust myself, thanks for checking all that. we'll need to be avail when it lands master ;) | 19:22 |
| ihrachys | in case something goes on fire | 19:22 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Remove curl from thick_slave https://review.openstack.org/288676 | 19:23 |
| AJaeger | ihrachys: I'm off most off Monday but others should be around | 19:23 |
| *** jed56 has quit IRC | 19:23 | |
| *** austin81 has left #openstack-infra | 19:23 | |
| ihrachys | AJaeger: ok, I guess I will find someone from infra side to look at 'oh my, neutron gate is totally broken' report | 19:23 |
| *** gyee has quit IRC | 19:23 | |
| pabelanger | clarkb: Ah, you are right. I have to squash both patches | 19:24 |
| clarkb | pabelanger: no just do two chagnes | 19:24 |
| clarkb | pabelanger: it will be fine because we don't build the images constantly | 19:25 |
| pabelanger | clarkb: 288634 will fail the gate | 19:25 |
| pabelanger | see zuul.o.o | 19:25 |
| clarkb | oh gotcha | 19:25 |
| *** apoorvad has joined #openstack-infra | 19:26 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Move curl to jenkins_common.pp https://review.openstack.org/288634 | 19:28 |
| *** _nadya_ has quit IRC | 19:28 | |
| pabelanger | okay, that should be better ^ | 19:28 |
| pabelanger | we'll likely need some sort of JJB-requirements.txt file for bindep moving forward | 19:29 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove experimental-bindep-jobs from repos https://review.openstack.org/288694 | 19:29 |
| pabelanger | but that requires some discussion with fungi | 19:29 |
| *** frauggy is now known as auggy | 19:30 | |
| clarkb | I think we just keep baking those things into the image | 19:30 |
| *** gyee has joined #openstack-infra | 19:31 | |
| SamYaple | just checking in, have we gotten mirror signing for packages done yet? if not is there a bp/ongoing work to do that? | 19:31 |
| *** ociuhandu has joined #openstack-infra | 19:31 | |
| mordred | no, I think it's on the backburner | 19:31 |
| *** sdague has quit IRC | 19:32 | |
| openstackgerrit | Merged openstack/requirements: Add Freezer modules and projects for Mitaka release https://review.openstack.org/271072 | 19:32 |
| *** kzaitsev_mb has joined #openstack-infra | 19:32 | |
| *** tqtran-afk has joined #openstack-infra | 19:34 | |
| *** piet has quit IRC | 19:35 | |
| *** sdake has quit IRC | 19:36 | |
| clarkb | mordred: did the change to allow multiple reprepros merge in puppet or do we still need to rewrite it as a defined type? for ceph testing we need to be adding the ceph repo | 19:36 |
| *** piet has joined #openstack-infra | 19:36 | |
| openstackgerrit | Kaitlin Farr proposed openstack/requirements: Bump Castellan to 0.4.0 https://review.openstack.org/288700 | 19:36 |
| mtreinish | nibalizer: are we manually running puppet on subunit-worker02.o.o? I only see the 1 run in puppetboard | 19:36 |
| *** jcoufal has joined #openstack-infra | 19:36 | |
| mtreinish | oh, I probably should have refreshed, now there are 2 | 19:37 |
| *** sdake has joined #openstack-infra | 19:37 | |
| *** ihrachys has quit IRC | 19:38 | |
| *** jpr has quit IRC | 19:38 | |
| AJaeger | dougwig: regarding https://review.openstack.org/#/c/286777/5/zuul/layout.yaml - all of this is removed in a follow-on change, so I would not go for style. Or do you see a real bug there? | 19:39 |
| dougwig | AJaeger: you're right, i can live with it in that case. | 19:40 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Need a new minimum for oslo.cache in Mitaka https://review.openstack.org/288703 | 19:41 |
| AJaeger | dougwig: 288506 is the followup | 19:41 |
| nibalizer | mtreinish: yea unfortunately when we fail puppet we don't tell puppetboard about it | 19:41 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Need a new minimum for oslo.cache in Mitaka https://review.openstack.org/288703 | 19:42 |
| *** dizquierdo has quit IRC | 19:47 | |
| *** kzaitsev_mb has quit IRC | 19:47 | |
| *** _nadya_ has joined #openstack-infra | 19:47 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:47 | |
| anteaya | jenkins06 doesn't have any jobs running on it anymore | 19:48 |
| *** Sukhdev has joined #openstack-infra | 19:49 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Switch /etc/logstash to /etc/subunit2sql https://review.openstack.org/288706 | 19:49 |
| *** annegent_ has quit IRC | 19:49 | |
| mtreinish | clarkb: ^^^ | 19:49 |
| mtreinish | nibalizer: really, ok :( | 19:49 |
| *** sbalukoff has joined #openstack-infra | 19:50 | |
| *** alivigni|gone is now known as ari | 19:50 | |
| *** ari is now known as ari- | 19:51 | |
| nibalizer | mtreinish: ya so i guess ansible runs with -e basically, so if it fails puppet it doesn't do the submit task | 19:51 |
| clarkb | I am tethering on my phone now because networking is so bad locally, going to see if I can make it better | 19:51 |
| nibalizer | so we can mark the puppet task to pass even if it fails, but thats not really what we want | 19:51 |
| nibalizer | what we want is some kind of a 'finally' task | 19:51 |
| nibalizer | that runs no matter what happened | 19:51 |
| nibalizer | so it look like http://docs.ansible.com/ansible/playbooks_error_handling.html we can do something clever where we report what happened into a variable, pass the puppet run, then do some stuff, then inspect the puppet run for failure and fail if we want to | 19:52 |
| odyssey4me | pabelanger clarkb thanks for keeping the review to https://review.openstack.org/288634 as I'm using it as a reference point for follow-up | 19:52 |
| *** fawadkhaliq has joined #openstack-infra | 19:52 | |
| *** tqtran-afk is now known as tqtran | 19:54 | |
| *** annegentle has joined #openstack-infra | 19:54 | |
| bkero | clarkb: is paste.openstack.org supposed to be working on v6? It has a v6 address, but doesn't seem to be answering | 19:56 |
| bkero | $ curl -v6 http://paste.openstack.org/ | 19:56 |
| bkero | * Trying 2001:4800:780e:510:3bc3:d7f6:ff04:fb20... | 19:56 |
| clarkb | I think it should listen on v6 | 19:56 |
| bkero | Ah, it returned after about 28 seconds into a 30 second timeout | 19:57 |
| clarkb | tcp6 0 0 :::80 :::* LISTEN 1820/apache2 | 19:57 |
| bkero | Maybe Portland comcast >_> | 19:57 |
| clarkb | bkero: you might be suffering the comcast derp | 19:57 |
| clarkb | ya | 19:57 |
| * bkero shakes fist | 19:57 | |
| clarkb | I am tmobiling now | 19:57 |
| *** ddieterly has quit IRC | 19:57 | |
| bkero | comcast seems to derp a lot more on v6 than v4 | 19:57 |
| clarkb | yes by far | 19:57 |
| bkero | Some sites don't load at all, some are super slow | 19:57 |
| *** asettle has joined #openstack-infra | 19:57 | |
| *** weshay has quit IRC | 19:57 | |
| clarkb | they seem to drop packets in seattle and in the denver area a lot | 19:58 |
| *** flepied has quit IRC | 19:58 | |
| bkero | oh yeah | 19:58 |
| bkero | there's totally some big segments of packet loss, v6 and v4 | 19:58 |
| bkero | I can't ping paste on v6 or v4 right now | 19:59 |
| bkero | I was pinging it, then the packets just...stopped. | 19:59 |
| *** flepied has joined #openstack-infra | 20:00 | |
| *** elmiko is now known as _elmiko | 20:00 | |
| AJaeger | clarkb: how many jenkins are updated now? | 20:00 |
| clarkb | AJaeger: I started with 01-03 and 07 | 20:00 |
| *** ddieterly has joined #openstack-infra | 20:01 | |
| clarkb | AJaeger: none are complete yet :/ | 20:01 |
| AJaeger | fungi, pabelanger, clarkb : Seems that jobs are running fine now - if they run on ubuntu-trusty... | 20:01 |
| clarkb | 07 and 03 are mostly done | 20:01 |
| clarkb | 01 and 02 are slower | 20:01 |
| AJaeger | clarkb: thanks for the update... | 20:01 |
| clarkb | and I saw some 502s go by so I will likely run twice on each master to make sure it is complete | 20:02 |
| *** asettle has quit IRC | 20:02 | |
| clarkb | we have a lot of xstatic projects | 20:02 |
| *** Sukhdev has quit IRC | 20:02 | |
| pabelanger | AJaeger: Ya, the ubuntu-trusty stuff I switched this morning seems to be working fine.... minus curl | 20:02 |
| clarkb | I am listening to the radio on my tv, there is something wrong about that | 20:02 |
| anteaya | I like listening to the radio | 20:03 |
| AJaeger | pabelanger, clarkb : I checked a few job that recently run and those looked fine | 20:04 |
| Clint | i've listened to tv on a radio | 20:04 |
| *** jamesmcarthur has joined #openstack-infra | 20:04 | |
| clarkb | AJaeger: great | 20:04 |
| pabelanger | AJaeger: Ya, I think we are in good shape | 20:06 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 20:06 |
| openstackgerrit | James Slagle proposed openstack-infra/tripleo-ci: WIP: Revert "Use swapfile environment in CI" https://review.openstack.org/288716 | 20:06 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add examples directory https://review.openstack.org/283794 | 20:06 |
| *** derekh has joined #openstack-infra | 20:06 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run cacti's node generation from cron https://review.openstack.org/284466 | 20:09 |
| *** ccrouch1 has quit IRC | 20:10 | |
| *** piet has quit IRC | 20:11 | |
| *** ccrouch has joined #openstack-infra | 20:11 | |
| nibalizer | mtreinish: at any rate | 20:12 |
| nibalizer | i think we've run puppet now | 20:12 |
| *** weshay has joined #openstack-infra | 20:12 | |
| anteaya | so I'm trying to get puppet-hound running on a vm I created | 20:13 |
| anteaya | and currently am seeing this error | 20:13 |
| anteaya | Failed to fetch ec2 uri http://169.254.169.254/latest/meta-data/ami-manifest-path: 500 Internal Server Error | 20:13 |
| anteaya | as part of the output when I run puppet apply | 20:13 |
| anteaya | the catalog run reports that it finishes | 20:13 |
| nibalizer | oh boo | 20:13 |
| mtreinish | nibalizer: ok, cool. Yeah it looks that way | 20:13 |
| anteaya | however there doesn't seem to be a process called hound | 20:14 |
| nibalizer | chown: invalid user: ‘logstash’ | 20:14 |
| nibalizer | start-stop-daemon: user 'logstash' not found | 20:14 |
| mtreinish | nibalizer: although I don't see it in cacti | 20:14 |
| mtreinish | nibalizer: oh, more logstash refs... | 20:14 |
| nibalizer | i can't see where in the init script its being called | 20:14 |
| mtreinish | nibalizer: I know where it is | 20:14 |
| clarkb | grep for USER | 20:15 |
| mtreinish | I fixed it here: https://review.openstack.org/#/c/288706/ | 20:15 |
| nibalizer | the user oh there it is derp | 20:15 |
| *** kzaitsev_mb has quit IRC | 20:15 | |
| *** yamamoto has joined #openstack-infra | 20:16 | |
| *** bodepd_ has quit IRC | 20:16 | |
| *** roxanaghe has quit IRC | 20:16 | |
| *** regXboi has quit IRC | 20:16 | |
| nibalizer | mtreinish: do we want to run it as root? | 20:16 |
| nibalizer | or some subunit user or w/e ? | 20:16 |
| *** annegentle has quit IRC | 20:17 | |
| anteaya | nor is go installed, nor is /home/hound created | 20:17 |
| anteaya | so puppet apply doesn't seem to be doing much that I can see | 20:17 |
| mtreinish | I'm not sure, but I'd have to create a subunit user too | 20:17 |
| krotscheck | Which is the shade channel? | 20:17 |
| *** bodepd has joined #openstack-infra | 20:17 | |
| nibalizer | anteaya: this is possible a facter bug | 20:18 |
| nibalizer | what puppet version do you have | 20:18 |
| *** derekh has quit IRC | 20:18 | |
| *** ddieterly has quit IRC | 20:18 | |
| jeblair | krotscheck: this one | 20:18 |
| krotscheck | jeblair: Oh. Neat! | 20:18 |
| anteaya | nibalizer: how do I find out? | 20:18 |
| nibalizer | anteaya: puppet --version | 20:18 |
| anteaya | ubuntu@hound:~$ puppet --version | 20:18 |
| anteaya | 3.8.6 | 20:18 |
| nibalizer | ok so run 'puppet apply -e 'notify { "derp": } ' | 20:19 |
| krotscheck | I just did a pip install --upgrade shade, and suddenly os_keypair is barfing on me. | 20:19 |
| anteaya | nibalizer: I will | 20:19 |
| *** jcoufal has quit IRC | 20:19 | |
| anteaya | nibalizer: I'm in a console? | 20:20 |
| nibalizer | oh god | 20:20 |
| nibalizer | run | 20:20 |
| nibalizer | :) | 20:20 |
| nibalizer | so puppet apply -e 'notify { "derp": }' | 20:21 |
| anteaya | best advice I've had all day | 20:21 |
| *** yamamoto has quit IRC | 20:21 | |
| nibalizer | exactly like that | 20:21 |
| * anteaya copy pastes | 20:21 | |
| mtreinish | nibalizer: do you want me to respin that patch to add a subunit user and use that for the init script? | 20:22 |
| anteaya | http://paste.openstack.org/show/489378/ | 20:22 |
| nibalizer | mtreinish: idunno | 20:22 |
| nibalizer | fungi: clarkb do you care? | 20:22 |
| clarkb | 07 just finished and I restarted it to redo the things that errored | 20:22 |
| clarkb | nibalizer: about? | 20:22 |
| anteaya | clarkb: just generally | 20:22 |
| anteaya | do you care? | 20:22 |
| anteaya | I've always considered you a caring person | 20:23 |
| *** rfolco has quit IRC | 20:23 | |
| mtreinish | clarkb: about running the subunit worker as root | 20:23 |
| mtreinish | clarkb: https://review.openstack.org/#/c/288706/1/templates/jenkins-subunit-worker.init.erb | 20:24 |
| clarkb | mtreinish: nibalizer oh I would prefer we used a system user | 20:24 |
| *** rfolco has joined #openstack-infra | 20:24 | |
| mtreinish | clarkb: ok, then I'll respin the patch to add a subunit user and use that | 20:24 |
| nibalizer | mtreinish: thanks | 20:24 |
| clarkb | and I just restarted on 03 | 20:24 |
| nibalizer | sorry not trying to put a ton of hoops in your way | 20:24 |
| nibalizer | also im curious how subunit01 works heh | 20:24 |
| nibalizer | probably it as a logstash user for $REASONS | 20:24 |
| mtreinish | nibalizer: yeah, subunit-worker01 was death by a thousand patches getting it up and running | 20:25 |
| *** piet has joined #openstack-infra | 20:25 | |
| mtreinish | it was also > 1 yr ago when we started it | 20:25 |
| nibalizer | okay so maybe its earned itsefl a spot on the reload-block | 20:25 |
| nibalizer | if we can get 02 in a place where its happy | 20:26 |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 20:26 |
| mtreinish | nibalizer: sure, although we'll have to be careful when we do to make sure we don't drop any queued streams it's processing | 20:27 |
| anteaya | nibalizer: in case you missed it when your name wasn't in front of it: http://paste.openstack.org/show/489378/ | 20:27 |
| anteaya | no hurry | 20:27 |
| nibalizer | anteaya: so that is great then | 20:28 |
| nibalizer | not a puppet/facter issue | 20:28 |
| nibalizer | can you paste the full output of the puppet run you are doing to install hound? | 20:28 |
| nibalizer | and what cloud are you using? | 20:28 |
| *** asettle has joined #openstack-infra | 20:29 | |
| anteaya | yay that's great | 20:29 |
| anteaya | sure | 20:29 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Switch /etc/logstash to /etc/subunit2sql https://review.openstack.org/288706 | 20:29 |
| mnaser | hello #-infra -- we're investigating adding ipv6 by default for all instances (so all instances are dual stack for public network connectivity). we used to do this for specific tenants that required it but we'd like to open it up for everyone. i was wondering if there is any feedback as cloud consumers of "this is what i like with x ipv6 impl" "this is what i dont like" "this is what i would ideally like" | 20:29 |
| mtreinish | nibalizer: ^^^ | 20:29 |
| *** sc68cal has quit IRC | 20:29 | |
| mtreinish | I think I did that right, my puppet syntax is a bit rusty | 20:29 |
| AJaeger | nibalizer: could you review https://review.openstack.org/288634 so that we get curl on ubuntu-trusty,please? | 20:29 |
| clarkb | mnaser: we like that it means no NAT with neutron :) but you don't NAT by default anyways on v4 | 20:30 |
| anteaya | nibalizer: hpe's internal mce cloud: http://paste.openstack.org/show/489376/ | 20:30 |
| mnaser | there seems to be many ways of how ips are allocated to instances, SLAAC, dhcpv6, etc. i was just hoping to get some feedback about experiences / etc | 20:31 |
| clarkb | mnaser: currently the only cloud we use with ipv6 is rackspace and I am not sure how they assign those addrs | 20:32 |
| *** salv-orlando has quit IRC | 20:33 | |
| mnaser | i will guess that it is probably static with nova-agent | 20:33 |
| mordred | dreamhost also does ipv6, but we do not use it for anything | 20:33 |
| anteaya | nibalizer: I'm totally guessing as to what to give puppet to make it hound | 20:33 |
| *** salv-orlando has joined #openstack-infra | 20:33 | |
| *** jtomasek has quit IRC | 20:33 | |
| mordred | mnaser: markmcclain could probably tell you what they did | 20:33 |
| nibalizer | anteaya: sure | 20:33 |
| mnaser | would be good to hear | 20:33 |
| greghaynes | I have had a great experience doing slaac v6 with openstack fwiw | 20:34 |
| greghaynes | it has 'just worked' | 20:34 |
| *** asettle has quit IRC | 20:34 | |
| greghaynes | highly recommend :) | 20:34 |
| * mtreinish checks which of the ipv6 allocation we still run on every tempest run | 20:34 | |
| anteaya | mtreinish: actually that was part of the feedback from ops meetup | 20:35 |
| Shrews | krotscheck: plz to define "barf" | 20:35 |
| markmcclain | mnaser: sure happy to tell you what we did when dual stacking v6 | 20:35 |
| krotscheck | Shrews: Sorry- was trying to debug | 20:35 |
| krotscheck | Have a stack trace, sec. | 20:35 |
| anteaya | we seem to get ipv6 running and then something new comes along a breaks it | 20:35 |
| anteaya | and* | 20:35 |
| mtreinish | it looks like slaac and dhcpv6: https://github.com/openstack/tempest/blob/master/tempest/scenario/test_network_v6.py#L242-L252 | 20:35 |
| mtreinish | anteaya: sure, thats true for us in the gate too (despite having test coverage for some of it) | 20:36 |
| krotscheck | Shrews: http://paste.openstack.org/show/489379/ | 20:36 |
| mnaser | http://docs.openstack.org/liberty/networking-guide/adv_config_ipv6.html -- this seems like a very very useful document | 20:36 |
| mtreinish | anteaya: which is why we demoted it to periodic (with the slow tag) | 20:36 |
| anteaya | mtreinish: okay so we are in agreement on the current status, that's a start | 20:36 |
| mtreinish | because no one was looking at the failure | 20:36 |
| mtreinish | s | 20:36 |
| anteaya | mtreinish: ah :( | 20:36 |
| anteaya | oh okay | 20:36 |
| krotscheck | Running shade 1.4.0 and ansible 2.0.1.0 | 20:36 |
| anteaya | so ipv6 folks to qa | 20:36 |
| mtreinish | and those tests are really poorly constructed and very expensive to run | 20:37 |
| nibalizer | anteaya: that error looks totally benine to me | 20:37 |
| mtreinish | anteaya: it's all on the neutron side | 20:37 |
| * krotscheck is building out a venv right now with all the things. | 20:37 | |
| *** yamamoto has joined #openstack-infra | 20:37 | |
| anteaya | nibalizer: yeah the catalog finished running | 20:37 |
| Shrews | krotscheck: pull the latest version from ansible repo. Should fix it for you | 20:37 |
| *** yamamoto has quit IRC | 20:37 | |
| *** bpokorny has quit IRC | 20:37 | |
| anteaya | nibalizer: but no hound process, no /home/hound dir, no installation of go | 20:37 |
| anteaya | nibalizer: so not sure what apply actually did | 20:37 |
| mtreinish | anteaya: http://lists.openstack.org/pipermail/openstack-dev/2016-February/085631.html that thread was some of the discussion | 20:38 |
| anteaya | mtreinish: what is? | 20:38 |
| anteaya | mtreinish: thanks | 20:38 |
| mordred | krotscheck: I generally prefer for os_keypair to not barf | 20:38 |
| Shrews | krotscheck: if not, will have a closer look in a bit | 20:38 |
| anteaya | mtreinish: not so very old | 20:38 |
| *** yamamoto has joined #openstack-infra | 20:39 | |
| *** yamamoto has quit IRC | 20:39 | |
| Shrews | krotscheck: mordred I think that's the ansible bug fix to default to str. Fixed with the latest modules | 20:39 |
| krotscheck | Shrews: Ok, i'll see if I can figure out how to do that. | 20:39 |
| *** sdake has quit IRC | 20:39 | |
| *** weshay has quit IRC | 20:39 | |
| openstackgerrit | Merged openstack-infra/system-config: Move curl to jenkins_common.pp https://review.openstack.org/288634 | 20:40 |
| *** cfarquhar has joined #openstack-infra | 20:40 | |
| *** cfarquhar has quit IRC | 20:40 | |
| *** cfarquhar has joined #openstack-infra | 20:40 | |
| pabelanger | clarkb: nibalizer thanks! ^ | 20:41 |
| *** annegentle has joined #openstack-infra | 20:41 | |
| clarkb | pabelanger: I will kick of an image rebuild in a few (lunch now) | 20:41 |
| anteaya | clarkb: enjoy lunch | 20:42 |
| Shrews | krotscheck: I can pull the PR for you later if you need it. afk atm | 20:42 |
| mtreinish | mordred, Shrews: btw, I meant to ask is this basic approach ok? https://review.openstack.org/285615 | 20:43 |
| mtreinish | I wanted to make sure it as ok before I start pushing it to completion | 20:43 |
| mtreinish | because it's far from working right now | 20:43 |
| openstackgerrit | Merged openstack-infra/shade: Mock glance v1 image with object not dict https://review.openstack.org/288503 | 20:43 |
| AJaeger | clarkb: enjoy lunch | 20:43 |
| *** bpokorny has joined #openstack-infra | 20:43 | |
| clarkb | AJaeger: I have job updates running everywhere but 06 now | 20:43 |
| clarkb | since 06 needs direct attention (which I will try to get to as well) | 20:44 |
| anteaya | thanks | 20:44 |
| *** esikachev has quit IRC | 20:44 | |
| *** nwkarsten has quit IRC | 20:44 | |
| anteaya | I had hoped at friday afternoon the rush would be quietening down | 20:44 |
| anteaya | but the gate is at 38 | 20:44 |
| *** ericksonsantos has quit IRC | 20:45 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations https://review.openstack.org/288715 | 20:45 |
| mordred | mtreinish: yes, looks good. I think you want self.list_images.invalidate instead of self._cache.invalidate | 20:45 |
| *** nwkarsten has joined #openstack-infra | 20:45 | |
| AJaeger | clarkb: thanks for taking care of all of this! | 20:45 |
| mtreinish | mordred: ok, cool. I'll do that when I respin it | 20:45 |
| *** Sukhdev has joined #openstack-infra | 20:46 | |
| clarkb | AJaeger: no problem, I wasn't able to help yesterday during feature freeze and :( | 20:47 |
| anteaya | mtreinish armax do either of you happen to know if any overture has been made on the operators list to see if we can recruit some folks who care about ipv6 to at least champion the bugs? | 20:48 |
| anteaya | referencing http://lists.openstack.org/pipermail/openstack-dev/2016-February/085631.html | 20:48 |
| *** nwkarste_ has joined #openstack-infra | 20:48 | |
| *** thorst_ has joined #openstack-infra | 20:48 | |
| mtreinish | anteaya: I haven't seen anything | 20:49 |
| anteaya | mtreinish: thank you | 20:49 |
| anteaya | I'm open to composing an email draft to them but didn't want to step on toes | 20:49 |
| *** nwkarsten has quit IRC | 20:49 | |
| *** _nadya_ has quit IRC | 20:49 | |
| krotscheck | Shrews, mordred: Well, that sortof worked, in that it's failing differently now "rror fetching keypair list (Inner Exception: Could not determine a suitable URL for the plugin)" | 20:50 |
| * krotscheck probably doesn't have a link to his cloudconfig in his venv | 20:51 | |
| anteaya | nibalizer: I'm at a loss as to what to do next to get puppet to hound, about to eat something but if you have any thoughts, I'm listening | 20:51 |
| *** asettle has joined #openstack-infra | 20:52 | |
| *** thorst has quit IRC | 20:52 | |
| mordred | krotscheck: 'could not determine suitable URL' is a keystone auth plugin discovery error | 20:52 |
| *** weshay has joined #openstack-infra | 20:53 | |
| mordred | krotscheck: it usually means that keystoneauth can't figure out from your arguments what auth plugin to use - although I imagine it might also be that if you don't have a linked config | 20:53 |
| krotscheck | mordred: Yeah, I'm macgyvering my system's ansible install right now to see if that's it. | 20:54 |
| *** geekinutah has quit IRC | 20:55 | |
| *** asettle has quit IRC | 20:56 | |
| anteaya | where are you applying the paper clip? | 20:56 |
| *** cloudtrainme has joined #openstack-infra | 20:56 | |
| anteaya | the paper clip does all things | 20:57 |
| jroll | anybody else seeing pillow install failures? 2016-03-04 20:48:12.367 | ValueError: jpeg is required unless explicitly disabled using --disable-jpeg, aborting | 20:57 |
| jroll | reference: https://jenkins02.openstack.org/job/gate-ironic-python34/1040/console | 20:57 |
| clarkb | jroll: do you have other-requirements.txt? | 20:57 |
| jroll | clarkb: nope | 20:57 |
| clarkb | ok so likely using the fallback list | 20:58 |
| jroll | clarkb: I don't know what that is :/ | 20:59 |
| *** Sukhdev has quit IRC | 20:59 | |
| krotscheck | mordred: The config is definitely linked - if I change it it yells at me, and adding an explicit version just yells at me more because /tokens does not exist. Are we expecting v3.0? | 20:59 |
| krotscheck | (keystone v3) | 20:59 |
| clarkb | jroll: those jobs are installing their deps via a bindep list now | 20:59 |
| pabelanger | clarkb: jroll: I can get a patch for that | 20:59 |
| clarkb | jroll: or optionally a project specific other-requirements.txt | 20:59 |
| clarkb | pabelanger: ty that will also need a new image since it is the fallback list | 21:00 |
| jroll | pabelanger: clarkb: I assume this is new? | 21:00 |
| jroll | pabelanger: also, thank you | 21:00 |
| pabelanger | jroll: Yup, gets us off bare-trusty images, which are pinned to rax | 21:01 |
| jroll | pabelanger: yay! | 21:01 |
| jroll | NobodyCam: ^ fyi, might be a bit before that's done as it requires new images | 21:01 |
| clarkb | and my networking is completely hosed | 21:03 |
| *** deray has joined #openstack-infra | 21:03 | |
| clarkb | I need to get a phone charger then I can tether more | 21:04 |
| mordred | krotscheck: can you paste your config without your password? | 21:04 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add libjpeg to bindep-fallback.txt https://review.openstack.org/288738 | 21:04 |
| pabelanger | clarkb: jroll: I think that will do the trick ^. Jenkins will confirm | 21:05 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: dump a release tracking dashboard https://review.openstack.org/288739 | 21:05 |
| *** sdake has joined #openstack-infra | 21:05 | |
| clarkb | pabelanger: if those are the package names from the old thick slave list then yup | 21:06 |
| AJaeger | pabelanger: apt is wrong for platform in your change, isn't it? | 21:06 |
| krotscheck | mordred: http://paste.openstack.org/show/489389/ | 21:06 |
| pabelanger | AJaeger: yes, thanks | 21:06 |
| clarkb | AJaeger: only if debian and other apt systems use a different name | 21:06 |
| AJaeger | pabelanger: shouldn't it be dpkg ? | 21:06 |
| clarkb | or is it ya that ok | 21:06 |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Add libjpeg to bindep-fallback.txt https://review.openstack.org/288738 | 21:06 |
| pabelanger | AJaeger: good eye | 21:06 |
| *** rguillebert has quit IRC | 21:07 | |
| pabelanger | I love how firefox is in bindep-fallback.txt | 21:07 |
| *** piet has quit IRC | 21:07 | |
| pabelanger | browser testing I guess? | 21:07 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: dump a release tracking dashboard https://review.openstack.org/288739 | 21:08 |
| rcarrillocruz | haha | 21:08 |
| rcarrillocruz | i guess | 21:08 |
| AJaeger | pabelanger: for horizon etc. | 21:08 |
| AJaeger | pabelanger: moving it to other-requirements would be a future cleanup... | 21:09 |
| pabelanger | AJaeger: indeed | 21:09 |
| AJaeger | but first let's get everything working ;) | 21:10 |
| *** apoorvad has quit IRC | 21:10 | |
| * AJaeger needs to call it a day now. | 21:10 | |
| anteaya | thanks AJaeger | 21:11 |
| pabelanger | AJaeger: enjoy | 21:11 |
| anteaya | enjoy the weekend | 21:11 |
| AJaeger | Thanks everybody to get us that far. Hope there aren't anymore surprises ;) | 21:11 |
| AJaeger | enjoy the weekend, anteaya, pabelanger and everybody ;) | 21:11 |
| clarkb | woot osic is ready to go | 21:14 |
| * clarkb removes wip from first change | 21:14 | |
| *** ldnunes has quit IRC | 21:14 | |
| *** rguillebert has joined #openstack-infra | 21:15 | |
| clarkb | infra-root please review https://review.openstack.org/#/c/288582/1 | 21:15 |
| clarkb | with that I can work on getting a mirror host deployed | 21:15 |
| openstackgerrit | Elizabeth K. Joseph proposed openstack-infra/system-config: Customize cgit max-repo-count and set it to 1500 https://review.openstack.org/288745 | 21:16 |
| pleia2 | clarkb: dns \o/ | 21:17 |
| *** dprince has quit IRC | 21:18 | |
| *** shardy has quit IRC | 21:18 | |
| anteaya | clarkb: yay | 21:19 |
| anteaya | mtreinish armax my first draft of an email to the operator's list recruiting help for ipv6: https://etherpad.openstack.org/p/ZEgSH1FjR5 | 21:19 |
| clarkb | jenkins 03 and 07 are all jjb'd. Every other jenkins is in progress (including 06), and a nodepool image build is happening but we will need a new one with new fallback bindep list | 21:20 |
| anteaya | mtreinish armax I'm thinking Monday would be a better day to post, rather than Friday afternoon | 21:20 |
| anteaya | or Saturday for some | 21:20 |
| anteaya | clarkb: yay! | 21:20 |
| pabelanger | clarkb: nice | 21:20 |
| clarkb | debating if I want to add the osic mirror build to the list or wait until a few things pop off :) | 21:20 |
| anteaya | dimsum_: so my status on getting a test hound instance up is that I can't figure out how to get puppet to tell hound to run | 21:21 |
| mnaser | anteaya: actually we're discussing and we're thinking of using slaac by our router | 21:21 |
| anteaya | dimsum_: sorry I don't have better news | 21:21 |
| mnaser | https://specs.openstack.org/openstack/neutron-specs/specs/juno/ipv6-provider-nets-slaac.html | 21:21 |
| dimsum_ | anteaya : no worries. thanks. /me does not know puppet either | 21:21 |
| anteaya | mnaser: is this in response to my etherpad? | 21:21 |
| mnaser | somewhat :-P | 21:21 |
| anteaya | dimsum_: were you able to get your work done? | 21:21 |
| anteaya | mnaser: okay great | 21:21 |
| mnaser | we have a big customer that is pushing on using it | 21:22 |
| mnaser | and they're usually nice about understanding that we're bringing in new-er things | 21:22 |
| mnaser | (aka things might break) | 21:22 |
| anteaya | mnaser: would this bug affect your usage of slaac? https://bugs.launchpad.net/openstack-gate/+bug/1540983 | 21:22 |
| openstack | Launchpad bug 1540983 in neutron "Gate failures for neutron in test_dualnet_multi_prefix_slaac" [Critical,Confirmed] | 21:22 |
| anteaya | mnaser: yay for nice customers! | 21:22 |
| mnaser | they use a ton of -infra projects like nodepool etc :) | 21:22 |
| mnaser | let me see | 21:22 |
| anteaya | cool, thanks | 21:22 |
| anteaya | mnaser: sounds like a great customer :) | 21:23 |
| * mnaser reads that test | 21:23 | |
| *** dmsimard has quit IRC | 21:23 | |
| *** roxanaghe has joined #openstack-infra | 21:24 | |
| mnaser | thats strange, i dont think i can imagine this affecting us | 21:24 |
| clarkb | gah there is a bug in my osic dns change | 21:24 |
| clarkb | womp womp | 21:24 |
| mnaser | 2 ports generally does cause troubles, ipv4 or ipv6 | 21:25 |
| anteaya | mnaser: can you add your perspective as a comment on the bug? | 21:25 |
| mnaser | good idea | 21:25 |
| anteaya | mnaser: that would be helpful | 21:25 |
| anteaya | thank you | 21:25 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Verify SSL with OSIC https://review.openstack.org/288582 | 21:25 |
| mnaser | let me read the initial commit | 21:25 |
| anteaya | clarkb: :( | 21:25 |
| mnaser | just to see the idea behind that test | 21:25 |
| anteaya | mnaser: yup, makes sense | 21:25 |
| clarkb | pleia2: do you mind rereviwing that really quick? | 21:25 |
| pleia2 | sure | 21:25 |
| armax | anteaya: let me reassess the severity of those issues, because I haven’t seen them burning the gate recently | 21:26 |
| armax | anteaya: | 21:26 |
| clarkb | I didn't actually remove the verify: false from one of the entries | 21:26 |
| armax | anteaya: that doesn’t mean they should not be addressed though | 21:26 |
| anteaya | armax: awesome thank you | 21:26 |
| armax | anteaya: thanks for reviving this up, I know haleyb volunteered at one point | 21:27 |
| anteaya | wonderful | 21:27 |
| armax | anteaya: but don’t count on it | 21:27 |
| anteaya | armax: I also want the ops to know we listen to their needs | 21:27 |
| anteaya | armax: yeah I know he is busy | 21:27 |
| anteaya | armax: feel free to edit what I wrote so it reflects reality | 21:27 |
| anteaya | armax: I was just making stuff up there | 21:27 |
| armax | let me double check the status of those bugs | 21:28 |
| anteaya | armax: https://etherpad.openstack.org/p/ZEgSH1FjR5 | 21:28 |
| anteaya | armax: wonderful thank you | 21:28 |
| clarkb | zuul looks like it is falling behind processing the results queue | 21:29 |
| clarkb | jeblair: ^ if this persists you may be interested | 21:30 |
| *** jpr has joined #openstack-infra | 21:30 | |
| odyssey4me | is the build up of events and results for zuul a known thing? it's been more than 10 mins and I don't see a review in the queue | 21:30 |
| dimsum_ | anteaya : no worries, it's not that bad | 21:30 |
| odyssey4me | lol clarkb great minds | 21:30 |
| anteaya | clarkb jeblair zuul memory usage is high: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=392&rra_id=all | 21:30 |
| mnaser | reading over the code, this comment here seems really relevent: https://bugs.launchpad.net/openstack-gate/+bug/1540983/comments/1 | 21:31 |
| openstack | Launchpad bug 1540983 in neutron "Gate failures for neutron in test_dualnet_multi_prefix_slaac" [Critical,Confirmed] | 21:31 |
| clarkb | I keep waiting for the number to fall but it seems ot only grow | 21:31 |
| odyssey4me | ++ | 21:31 |
| mnaser | i unfortunately don't have time to look into it even more but i believe that is largely unaffected | 21:31 |
| anteaya | mnaser: thanks for looking | 21:31 |
| mnaser | also, how often could we have logs regarding error'd instances from nodepool, we'd like to flatten it out to 0 error node launch attempts | 21:31 |
| anteaya | clarkb: only 2GB of memory available left for zuul | 21:32 |
| *** rfolco has quit IRC | 21:32 | |
| anteaya | clarkb: if jeblair doesn't show up soon, I would advocate for a restart of zuul | 21:32 |
| armax | anteaya: unfortunately with the current logstash status we’re sort of shootign blind | 21:32 |
| clarkb | anteaya: I'd like to see all of these completed jobs result in merges :( | 21:33 |
| anteaya | armax: I understand, it is an overture to work together on a common issue they have said is important | 21:33 |
| armax | anteaya: ack | 21:33 |
| anteaya | armax: I think a doorway to help is the important thing | 21:33 |
| anteaya | armax: if you can make the words true that will accomplish that I'd be grateful | 21:33 |
| anteaya | armax: going to send it monday, friday is poorly timed for an overture | 21:33 |
| anteaya | clarkb: I agree | 21:34 |
| armax | anteaya: right now logstash is not returning any result in the past 30 days | 21:34 |
| armax | anteaya: might be true, might be false, I simply don’t trust the tool any longer :( | 21:34 |
| anteaya | armax: yes, clarkb has been working on getting logstash happy again | 21:34 |
| clarkb | armax: we only ever record 10 days of info fwiw | 21:34 |
| armax | clarkb: I know | 21:34 |
| armax | but one can only hope ;) | 21:35 |
| anteaya | clarkb: look at that lovely stack of patches ready to merge | 21:35 |
| anteaya | c'mon zuul | 21:35 |
| *** apoorvad has joined #openstack-infra | 21:36 | |
| anteaya | armax: the amount of storage we would have to have for 30 days is beyond our resources | 21:36 |
| anteaya | armax: we have challenges keeping up with 10 days worth of logs | 21:36 |
| jeblair | hello | 21:36 |
| anteaya | hello | 21:37 |
| anteaya | jeblair: nice to see you | 21:37 |
| pleia2 | jeblair: it rained today :( | 21:37 |
| jeblair | pleia2: that's good, remember? | 21:37 |
| clarkb | jeblair: it looks like the zuul mainrun loop isn't looping | 21:37 |
| pleia2 | jeblair: oh yes, but now I don't want to go outside! | 21:37 |
| *** roxanaghe has quit IRC | 21:37 | |
| armax | I’d be happy with 7 days, if I knew that 0 results really mean 0 results | 21:38 |
| anteaya | jeblair: zuul is almost out of memory | 21:38 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: gerrit: add puppet-neutron-core https://review.openstack.org/288753 | 21:38 |
| armax | but that’s ok, I realize you guys are working to make this great again, if there’s anything I can do, user testing if nothing else | 21:38 |
| armax | please give me a shout | 21:38 |
| anteaya | armax: thank you | 21:38 |
| armax | I don’t mean to pres | 21:38 |
| armax | press | 21:38 |
| anteaya | I know | 21:39 |
| anteaya | we appreciate the support | 21:39 |
| anteaya | and that we have tools devs use | 21:39 |
| anteaya | and like on the good days | 21:39 |
| *** yamamoto has joined #openstack-infra | 21:39 | |
| clarkb | http://paste.openstack.org/show/489398/ but not sure if that is related | 21:40 |
| jeblair | i think that's harmless | 21:41 |
| anteaya | zuul eth0 out has peaked: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=403&rra_id=all | 21:42 |
| jeblair | now that's interesting... | 21:42 |
| anteaya | it doesn't feel healthy | 21:43 |
| clarkb | I wonder if that is due to jenkins job updates | 21:43 |
| clarkb | that inefficient gearman update | 21:43 |
| *** rlandy has quit IRC | 21:43 | |
| clarkb | I can halt some of them | 21:44 |
| *** |-paul-| has joined #openstack-infra | 21:44 | |
| *** rlandy has joined #openstack-infra | 21:44 | |
| jeblair | most outbound traffic is to graphite | 21:45 |
| jeblair | about 5mbps worth | 21:45 |
| anteaya | something is querying graphite? | 21:45 |
| jesusaur | fungi: are you still around? | 21:45 |
| anteaya | a lot? | 21:45 |
| anteaya | jesusaur: he is currently offline | 21:45 |
| jesusaur | anteaya: thanks | 21:45 |
| anteaya | jesusaur: he had to travel for a thing and ran out of battery | 21:45 |
| clarkb | anteaya: no zuul only sends data to graphite | 21:45 |
| anteaya | jesusaur: I'm sure he'll read backscroll | 21:46 |
| anteaya | jesusaur: welcome | 21:46 |
| anteaya | clarkb: okay, thanks | 21:46 |
| jeblair | clarkb: most of which are incrementers and timers for geard CAN_DO calls | 21:46 |
| clarkb | jeblair: ah that makes sense | 21:46 |
| clarkb | since we are updating them right now | 21:46 |
| fungi | okay, i'm back home. looks like jenkins06 has been cleaned up and restarted (thanks!) but isn't running any jobs at all? | 21:47 |
| anteaya | fungi: ah here you are | 21:47 |
| *** Douhet has quit IRC | 21:47 | |
| clarkb | fungi: zuul has stopped processing the results queue as quickly as we would like | 21:47 |
| clarkb | if at all | 21:47 |
| *** salv-orlando has quit IRC | 21:47 | |
| *** kgiusti has left #openstack-infra | 21:47 | |
| jeblair | that will be from geard; which shouldn't stop the zuul main loop, except indirectly (by being slow to respond) | 21:47 |
| *** yamamoto has quit IRC | 21:48 | |
| jesusaur | ahh, fungi a couple days ago you were talking with AJaeger about ZUUL_REF vs ZUUL_REFNAME, was there a decision to stop using ZUUL_REF in favor of ZUUL_REFNAME? | 21:48 |
| *** salv-orlando has joined #openstack-infra | 21:48 | |
| jeblair | clarkb: i see main loop activity at 21:44; last time i looked it was at 21:32 | 21:49 |
| jeblair | clarkb: so i think it's moving | 21:49 |
| clarkb | jeblair: just slow | 21:49 |
| fungi | jesusaur: i think prior consensus was that in jobs which will run in post and need to know the name of the branch to which the change merged, that both ZUUL_REF and ZUUL_REFNAME will contain the same values, but that using ZUUL_REFNAME is perhaps marginally more correct | 21:50 |
| jeblair | fungi: i would vote that way (ZUUL_REF is "the thing you should fetch") | 21:50 |
| *** hashar has quit IRC | 21:50 | |
| fungi | yep, and if you want a script to get the branch name in post but the tag name in tag-based queues, then ZUUL_REFNAME does that | 21:51 |
| jeblair | ZUUL_REFNAME is "the name of the ref that triggered a ref-updated event"... right. | 21:51 |
| *** salv-orlando has quit IRC | 21:51 | |
| clarkb | I am tethering off of my phone so debugging is hard | 21:51 |
| jesusaur | ah, well in that case we will need to teach zuul-cloner about ZUUL_REFNAME, it currently only has ZUUL_REF | 21:51 |
| anteaya | clarkb: :( | 21:52 |
| *** salv-orlando has joined #openstack-infra | 21:52 | |
| clarkb | it would not surprise me if somehow the jjb updates are adversely affecting zuul via gearman | 21:52 |
| *** ddieterly has joined #openstack-infra | 21:52 | |
| openstackgerrit | Sam Yaple proposed openstack-infra/shade: Add update_service() https://review.openstack.org/287504 | 21:52 |
| *** ddieterly has quit IRC | 21:53 | |
| jeblair | clarkb: i agree | 21:53 |
| *** aysyd has quit IRC | 21:53 | |
| jeblair | why are we updating everything? | 21:53 |
| anteaya | moving jobs from bare trusty to ubuntu trusty | 21:53 |
| *** roxanaghe has joined #openstack-infra | 21:54 | |
| jeblair | when did jjb start running? | 21:54 |
| clarkb | I started 01-03 and 07 several hours ago (~10:30am pst) | 21:55 |
| anteaya | jeblair: https://review.openstack.org/#/c/288446/ | 21:55 |
| clarkb | when 03 nad 07 completed I started the remainder | 21:55 |
| clarkb | about an hour ago | 21:55 |
| jeblair | the traffic pattern changed around 2045 | 21:56 |
| jeblair | which is about an hour ago | 21:56 |
| fungi | maybe gearman registration flood slamming zuul's geard? | 21:57 |
| clarkb | looks like 01 is done now too | 21:58 |
| jeblair | fungi: yeah, that's the theory, though it's interesting that the 4 clarkb started earlier would not have caused this, but the second 4 (plus 2 still running) would... | 21:58 |
| *** pcaruana has joined #openstack-infra | 21:58 | |
| cody-somerville | jeblair, fungi, anteaya, mtreinish, etc: FYI, here is the review for the Gertty breadcrumb footer bar, https://review.openstack.org/#/c/288250/ | 21:58 |
| clarkb | 02,04-06 are the ones currently running | 21:58 |
| jeblair | most gearman activity seems to be 2,4,5,6 | 21:58 |
| anteaya | jenkins06 was having an off day prior to the jjb update, might it still be off? | 21:58 |
| anteaya | cody-somerville: thank you | 21:59 |
| clarkb | anteaya: no, I brought it back and clenaed up its slaves | 21:59 |
| anteaya | clarkb: okay thanks | 21:59 |
| fungi | well, 01 and 02 are seriously slow, but 4/5/6 are a good deal faster | 21:59 |
| *** automagically has joined #openstack-infra | 21:59 | |
| clarkb | so maybe thats what it is? | 21:59 |
| clarkb | 04-06 can hammer the geard server harder? | 21:59 |
| fungi | could be they're able to flood better | 21:59 |
| jeblair | maybe | 21:59 |
| *** AndyU has joined #openstack-infra | 21:59 | |
| jeblair | clarkb: you suggested suspending some of them... i'm in favor of doing that now to see the effect | 22:00 |
| fungi | ruled out something spamming gerrit comments bogging down the event queue? | 22:00 |
| *** _sarob has joined #openstack-infra | 22:00 | |
| anteaya | fungi: nope | 22:00 |
| fungi | i'll take a peek in gerrit's logs | 22:01 |
| cody-somerville | jeblair: have to say it was pretty easy to implement thanks to good architecture on (assuming) your part. :) | 22:01 |
| anteaya | fungi: thanks | 22:01 |
| clarkb | you think just ^Z them? | 22:01 |
| jeblair | fungi: it's manifesting in zuul as a very slow main loop; doesn't look like a gerrit spam problem, but good to double check | 22:01 |
| *** deray has quit IRC | 22:01 | |
| jeblair | cody-somerville: thanks and thanks :) | 22:01 |
| jeblair | clarkb: yeah | 22:02 |
| AndyU | anteaya, pleia2, fungi, clarkb, etc....: Hi all, I'm back with another question. I was give this specific question "Does the community mandate any specific OpenStack components running and what access level do they require to those services?" | 22:02 |
| clarkb | I ^Z'd two of the 4 | 22:02 |
| clarkb | 04 and 05 are the suspended processes | 22:02 |
| jeblair | clarkb: i still see much tcp from them but will keep an eye out for that being reduced | 22:03 |
| fungi | very little comment activity, as befits late fridays | 22:03 |
| anteaya | AndyU: ah the only mandates regarding openstack componets come from refstack and <the name of the other thing> | 22:03 |
| jeblair | clarkb: i think it has reduced :) | 22:03 |
| anteaya | fungi: yay | 22:03 |
| jeblair | most gearman traffic is 02 and 06 now- | 22:03 |
| clarkb | I left 02 and 06 because they were further along so in theory will finish quicker | 22:03 |
| AndyU | I will wait patiently for your answer ;) | 22:03 |
| fungi | most active commenter during the 21:5.* timeframe was fuel-ci and it left a scant 8 comments in that 10-minute span | 22:03 |
| clarkb | AndyU: we do!, we need nova, keystone, glance. We can in theory work with nova net but all our clouds are neutron now so that may be an implicit requirement at this point. And we like it if we can use cinder | 22:03 |
| *** sarob has quit IRC | 22:04 | |
| anteaya | AndyU: regarding a cloud for infra testing, we just need the ability to create vms, upload images, reach the internet and upload logs | 22:04 |
| anteaya | AndyU: and then delete them | 22:04 |
| *** roxanaghe has quit IRC | 22:04 | |
| anteaya | so whatever you need to run so we can do that | 22:04 |
| *** rlandy has quit IRC | 22:04 | |
| jeblair | clarkb: main loop activity (as indicated by report events appears to be increasing | 22:04 |
| anteaya | AndyU: and thanks for your patience :) | 22:04 |
| *** _sarob has quit IRC | 22:04 | |
| *** thorst_ has quit IRC | 22:05 | |
| AndyU | I'll take as much info as you can give me. No hurry. I'll pass along all your comments. | 22:05 |
| anteaya | AndyU: so what clarkb said | 22:05 |
| *** roxanaghe has joined #openstack-infra | 22:05 | |
| *** dbrycen has joined #openstack-infra | 22:05 | |
| jeblair | clarkb: zuul is now transferring more data to gerrit than graphite | 22:05 |
| clarkb | AndyU: for perms we don't need anything fancy. We need to boot and delete instances with public IPs and wide open security groups. We need to upload disk images. and thats about it | 22:05 |
| anteaya | yay! | 22:05 |
| openstackgerrit | Merged openstack/requirements: Bumps os-win version to 0.2.3 https://review.openstack.org/288429 | 22:06 |
| AndyU | ok, got it. | 22:06 |
| clarkb | jeblair: 06 is done, shoul dI want before foregrounding one of 04 and 05? | 22:06 |
| clarkb | s/want/wait/ | 22:06 |
| fungi | zuul debug log shows it adding events pretty steadily, but also lots of job description setting in jenkins | 22:06 |
| fungi | most of the events look like ref-replicated | 22:06 |
| jeblair | clarkb: i think so, let's see if it can make forward progress | 22:06 |
| fungi | which it's going to ignore | 22:06 |
| *** ddieterly has joined #openstack-infra | 22:07 | |
| clarkb | the top of the gate is no longer there | 22:07 |
| AndyU | So, nova, keystone, glance neutron, cinder... | 22:07 |
| *** dimtruck is now known as zz_dimtruck | 22:07 | |
| *** thorst has joined #openstack-infra | 22:07 | |
| anteaya | yay! | 22:07 |
| clarkb | AndyU: yes, are there particular services you are worried about? | 22:08 |
| *** annegentle has quit IRC | 22:09 | |
| AndyU | I don't think so but the question came from my boss | 22:09 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Fix apache failure https://review.openstack.org/288763 | 22:09 |
| dbrycen | does anybody know how to get around PBR_VERSION conflicting with an explicit pbr>=x.x in a setup.py's setup_requires? | 22:09 |
| fungi | AndyU: to the extent that MOS is hopefully "MOStly OpenStack" i would be surprised if it's unsuitable | 22:09 |
| AndyU | Yeah, I think we're looking pretty good | 22:09 |
| *** moravec has quit IRC | 22:10 | |
| *** bgmccollum has joined #openstack-infra | 22:10 | |
| *** roxanaghe has quit IRC | 22:10 | |
| clarkb | separately I am uploading a new ubuntu-trusty image everywhere that should have the curl fix but not the libjpeg-dev fix | 22:10 |
| fungi | at least the things i've heard about it imply it's no less openstacky than rdo (and possibly more openstacky than some of the cloud providers already donating resources to us) | 22:10 |
| *** roxanaghe has joined #openstack-infra | 22:10 | |
| anteaya | clarkb: ack | 22:10 |
| clarkb | dbrycen: is the problme that you have an older version preinstalled and it complains that it is not new enough? | 22:11 |
| clarkb | dbrycen: your best bet is probably to upgrade pbr? | 22:11 |
| AndyU | all the projects that you mentioned are in play and many more | 22:11 |
| dbrycen | clarkb: well i'm just trying to do a "python setup.py sdist" and i dont have pbr installed | 22:11 |
| jeblair | clarkb: i think we may want to let it run one at a time... | 22:11 |
| *** e0ne has joined #openstack-infra | 22:11 | |
| clarkb | jeblair: ok | 22:11 |
| *** roxanagh_ has joined #openstack-infra | 22:11 | |
| jeblair | clarkb: is it in a screen we can join? | 22:11 |
| fungi | clarkb: were we installing curl and libjpeg-dev on bare-trusty or were those things that rackspace was probably preinstalling on their images and used by unusual jobs which our numerous experimental vetting attempts didn't uncover? | 22:11 |
| AndyU | Thanks all, I'll take back what you gave me. Have a great weekend. ;-) | 22:12 |
| clarkb | jeblair: yes all in screens (owned by me not root which makes it a little tricky) on 02, 04 and 05) | 22:12 |
| *** thorst has quit IRC | 22:12 | |
| clarkb | dbrycen: if you don't have pbr installed it should install the latest version for you | 22:12 |
| *** roxanaghe has quit IRC | 22:12 | |
| jeblair | clarkb: which one is running now? | 22:12 |
| clarkb | jeblair: 02 | 22:12 |
| fungi | sudo su - clarkb ; script /dev/null ; screen -x | 22:12 |
| clarkb | dbrycen: might help if you can paste the error output somewhere | 22:13 |
| *** weshay has quit IRC | 22:13 | |
| jeblair | okay, i believe this is making forward progress and will eventually catch up | 22:13 |
| fungi | dbrycen: where "somewhere" is probably http://paste.openstack.org/ | 22:13 |
| fungi | and preferably not in channel at least | 22:13 |
| jeblair | i'm going to go afk for a bit, but will check back later and fg jjb runs as needed | 22:14 |
| fungi | thanks jeblair! | 22:14 |
| dbrycen | clarkb: that's what i saw happening but it was making the version of pbr equal to what i set for PBR_VERSION and that version conflicts with what version of pbr is required by setup.py. It sounds funky but i'm using PBR_VERSION env var to make a custom version for my openstack project. | 22:14 |
| anteaya | AndyU: have a good weekend | 22:14 |
| *** baoli_ has quit IRC | 22:14 | |
| clarkb | dbrycen: that shouldn't affect the version of PBR, that affects the version that PBR calculates for the project being installed | 22:15 |
| clarkb | I think pasted output may help | 22:15 |
| dbrycen | ok let me do that | 22:16 |
| clarkb | jeblair: note that 02 may need to be restarted if it errored out along the way, the end of its output will tell you if it did | 22:16 |
| clarkb | I am watching it though and can likely get these done | 22:16 |
| *** moravec has joined #openstack-infra | 22:16 | |
| *** gordc has quit IRC | 22:17 | |
| clarkb | yup it did 502 so I have just restarted it | 22:17 |
| *** asettle has joined #openstack-infra | 22:18 | |
| dbrycen | clarkb: http://paste.openstack.org/show/489404/ | 22:19 |
| *** Sukhdev has joined #openstack-infra | 22:19 | |
| dbrycen | clarkb: note that if i change PBR_VERSION to be a version which cooperates with setup_requires, then the sdist creation is successful | 22:20 |
| *** Sukhdev has quit IRC | 22:20 | |
| clarkb | dbrycen: I don't think that is related to PBR_VERSION I think instead that you already have pbr installed there and it isn't new enough? or maybe if you use a pypi mirror you don't have a new enough version of pbr there | 22:20 |
| *** Sukhdev has joined #openstack-infra | 22:20 | |
| clarkb | dbrycen: is there no pbr egg in that dir? | 22:21 |
| AndyU | anteaya: thanks! | 22:21 |
| anteaya | :) | 22:21 |
| *** AndyU has quit IRC | 22:21 | |
| dbrycen | clarkb: nope, no pbr egg in the repo | 22:21 |
| clarkb | dbrycen: what about pip freeze | grep pbr? | 22:21 |
| dbrycen | clarkb: pip isnt installed and it's not in dpkg list either | 22:22 |
| clarkb | huh I wonder if that is realted, I think pbr requires pip | 22:23 |
| *** asettle has quit IRC | 22:23 | |
| dbrycen | hmm, i can try installing it to see if it helps | 22:23 |
| clarkb | what version of setuptools do you have? | 22:23 |
| *** thorst has joined #openstack-infra | 22:23 | |
| fungi | zuul has basically emptied its results queue now and is starting to burn down the ~16k events that generate | 22:23 |
| clarkb | since that is the thing attempting to install pbr in the first place | 22:23 |
| fungi | d | 22:23 |
| *** annegentle has joined #openstack-infra | 22:23 | |
| dbrycen | clarkb: setuptools is 3.3 | 22:24 |
| clarkb | fungi: https://review.openstack.org/#/c/288582/2 care to review that? osic is ready for us | 22:24 |
| clarkb | dbrycen: updating setuptools is the other thing I would try, that version is fairly old | 22:24 |
| * pabelanger loads http://grafana.openstack.org/dashboard/db/nodepool to watch the light show of 'in use' nodes | 22:25 | |
| dbrycen | clarkb: cool i will try that as well and report back | 22:25 |
| odyssey4me | alright, I'm out for the night - have a fantastic weekend everyone! | 22:25 |
| pabelanger | odyssey4me: later | 22:25 |
| clarkb | I like that we got our job count down to about 8k and a few weeks later its back to about 9k :) | 22:25 |
| *** akshai has quit IRC | 22:25 | |
| pabelanger | clarkb: where are you looking at that? | 22:26 |
| fungi | clarkb: hah, have you looked at the cert? i'm curious to see in 3 months whether they've punted on the auto-update process for that | 22:26 |
| *** abregman|afk is now known as abregman | 22:26 | |
| *** cloudtrainme has quit IRC | 22:26 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Fix on category groups expand https://review.openstack.org/288464 | 22:26 |
| *** tpsilva has quit IRC | 22:27 | |
| fungi | and now zuul's back to 0 events, 0 results | 22:27 |
| pabelanger | fungi: Yup, big spike in building nodes :) | 22:27 |
| pabelanger | 102 ATM | 22:27 |
| pabelanger | 144 :) | 22:27 |
| pabelanger | nice | 22:27 |
| fungi | so the jjb job updates reregistering in gearman triggering job description updates from zuul back to jenkins could very well have explained the performance roadblock | 22:27 |
| clarkb | fungi: yes you can pull it up in your web browser | 22:28 |
| *** annegentle has quit IRC | 22:28 | |
| clarkb | fungi: it looks all kosher | 22:28 |
| *** lascii is now known as alaski | 22:28 | |
| fungi | clarkb: oh, i know. i mean, i did | 22:28 |
| fungi | just curious if you looked at the issuer/expiration | 22:28 |
| fungi | not concerned, merely amused | 22:29 |
| *** [1]Thelo has joined #openstack-infra | 22:29 | |
| pabelanger | 385 nodes building, clouds must love us ATM | 22:30 |
| fungi | pabelanger: we are the definition of "thundering herd" | 22:30 |
| *** jamesmcarthur has quit IRC | 22:30 | |
| clarkb | fungi: oh you mean they went and did a free cert or something | 22:30 |
| pabelanger | fungi: indeed | 22:30 |
| fungi | clarkb: they've got it set up using letsencrypt | 22:31 |
| fungi | so 3-month expiration on the cert | 22:31 |
| *** asettle has joined #openstack-infra | 22:31 | |
| fungi | in theory they've got something set up to auto-update from le in a timely fashion | 22:31 |
| fungi | i'm just curious to see if it ends up working | 22:31 |
| *** Thelo has quit IRC | 22:31 | |
| *** [1]Thelo is now known as Thelo | 22:31 | |
| fungi | the update process i mean | 22:32 |
| *** asettle has quit IRC | 22:33 | |
| clarkb | fungi: once that is in I can attempt to make a mirror using my shade-launch-node script changes iwth unreleased shade | 22:33 |
| *** gildub has joined #openstack-infra | 22:33 | |
| dbrycen | clarkb: OK, what solved the issue was actually installing pbr. i guess i just have to pre-install it now but it was nice before when it would auto-install for me (when setup_requires just required "pbr" and didnt need a specific version) | 22:33 |
| *** thorst has quit IRC | 22:35 | |
| pabelanger | So, I just stumbled across: https://snapshot.raintank.io/dashboard/snapshot/28FzqeWPVMxiBWun9qkiH4zk0f5JPtdY could be a cool grafana dashboard for infracloud | 22:36 |
| clarkb | dbrycen: I think if your setuptools was newer the preinstall machinery would work | 22:36 |
| *** sorantis has quit IRC | 22:38 | |
| dbrycen | clarkb: i did try installing pip and upgrading setuptools and it still asked me to have the right version of pbr installed | 22:38 |
| dbrycen | clarkb: still happy that i can successfully create the tarball though :) | 22:38 |
| *** gildub has quit IRC | 22:39 | |
| *** amitgandhinz has quit IRC | 22:39 | |
| *** yonglihe has quit IRC | 22:42 | |
| *** piet has joined #openstack-infra | 22:42 | |
| *** rhallisey has quit IRC | 22:42 | |
| *** devananda has quit IRC | 22:45 | |
| *** mriedem has quit IRC | 22:46 | |
| *** erlon has quit IRC | 22:46 | |
| *** abregman is now known as abregman|afk | 22:46 | |
| fungi | clarkb: you're missing an amusing private e-mail thread unfortunately... in response to our request for an ipv4 /19 and ipv6 /48 for infra-cloud, someone (presumably in charge of provisioning) in hpe responded asking whether we were asking for private or public assignments for both of those | 22:49 |
| clarkb | private ipv6? what is this | 22:49 |
| clarkb | we could just run our own ULA if we wanted to ... | 22:49 |
| fungi | also, like... why would we be asking you to assign private addresses for us? wouldn't we just pick some if that's what we wanted? | 22:50 |
| *** jpr has quit IRC | 22:50 | |
| pleia2 | heh | 22:50 |
| *** zz_dimtruck is now known as dimtruck | 22:51 | |
| *** ari- has quit IRC | 22:51 | |
| clarkb | openstack-ansible has a job for everything | 22:51 |
| fungi | and for everything else, there's ansible-openstack | 22:52 |
| clarkb | I am guessing there is a different repo for all of these things | 22:52 |
| clarkb | like rsyslog | 22:52 |
| *** abregman|afk has quit IRC | 22:52 | |
| fungi | clarkb: on a positive note, i suppose that e-mail exchange does indicate someone in hpe is actually reviewing the request. that's faster than i could have hoped | 22:54 |
| pabelanger | So, as a recap for the week. wiki.o.o is looking pretty good right now. New spammers have stopped, and we are only getting 1 spammer account every few days now | 22:54 |
| pabelanger | https://wiki.openstack.org/w/index.php?title=Special:RecentChanges show valid contribs by the community | 22:54 |
| *** e0ne has quit IRC | 22:55 | |
| *** geekinutah has joined #openstack-infra | 22:55 | |
| pabelanger | So, next week I think we can discuss opening registrations again, and see what happens | 22:55 |
| pabelanger | and land some inflight patches giving admins more spamming tools | 22:55 |
| anteaya | pabelanger: right now new accounts are cut off? | 22:55 |
| pabelanger | anteaya: yes | 22:55 |
| anteaya | pabelanger: did the prior spam pages get deleted? | 22:55 |
| fungi | as of ~monday if memory serves | 22:55 |
| anteaya | or is that work to be done still | 22:56 |
| pabelanger | we should be able to fix openid accounts, but need to enable some settings | 22:56 |
| fungi | (the disablement of account creation i mean) | 22:56 |
| pleia2 | pabelanger: thanks for the update | 22:56 |
| *** Sukhdev has quit IRC | 22:56 | |
| pabelanger | anteaya: no, we still need to automate that. I believe 287232 will help with that | 22:56 |
| fungi | yes, i'm surprised this has stymied them | 22:56 |
| *** hdd has quit IRC | 22:56 | |
| anteaya | pabelanger: awesome, thank you | 22:57 |
| anteaya | and as pleia2 says, thanks for the update | 22:57 |
| pleia2 | btw, on the Ubuntu wiki, they created a team in launchpad of users authorized to edit the wiki and have restricted edits to that team | 22:57 |
| pleia2 | (did this yesterday) | 22:57 |
| pabelanger | I think if we set: $wgOpenIDUseEmailAsNickname for openid, new accounts will be created in mediawiki while having the createaccount flag disabled | 22:57 |
| pabelanger | but we need to test that | 22:57 |
| pleia2 | which makes me worry they aren't really tackling the "lp has tons of spam accounts" problem | 22:57 |
| fungi | wow, so ubuntu has basically closed down the public editing of their wiki without directly requesting permission? | 22:58 |
| anteaya | pleia2: well does lp have any developers left? | 22:58 |
| pabelanger | pleia2: Ya, I would like to look into migrating to openstackid.org, but not sure what is needed | 22:58 |
| *** tiswanso has quit IRC | 22:58 | |
| fungi | i hear they've re-grown a dev team, but that doesn't necessarily mean their openid policing is well-staffed | 22:58 |
| *** devananda has joined #openstack-infra | 22:58 | |
| pleia2 | anteaya: they moved some people over there | 22:58 |
| pleia2 | fungi: yeah | 22:58 |
| fungi | especially since login.launchpad.net and launchpad.net are entirely separate systems | 22:59 |
| anteaya | ah | 22:59 |
| anteaya | well yay dev team | 22:59 |
| pleia2 | yeah, IS handles operations | 22:59 |
| fungi | login.launchpad.net is an authentication/identification system which happens to have a domain name in common with launchpad.net | 22:59 |
| pleia2 | my very late lunch has arrived, so attending to that for a bit | 22:59 |
| anteaya | pleia2: enjoy lunch | 22:59 |
| fungi | or at least that's how i've heard it explained in the past | 23:00 |
| *** geekinut1h has joined #openstack-infra | 23:00 | |
| *** geekinutah has quit IRC | 23:00 | |
| anteaya | if we are doing updates, here's mine, I tried to create a hound test server and can't seem to get puppet to get hound to run: this is the command I try and the pp file: http://paste.openstack.org/show/489376/ | 23:01 |
| anteaya | the command completes but there is no hound proces, no /home/hound dir and no installation of go | 23:01 |
| openstackgerrit | Paul Belanger proposed openstack-infra/puppet-mediawiki: Set wgOpenIDUseEmailAsNickname True by default https://review.openstack.org/288785 | 23:02 |
| anteaya | I'm at a loss as to what to do next | 23:02 |
| pabelanger | pleia2: fungi: So, untested ^ but should allow openid account to properly connect on first time auth. | 23:02 |
| pabelanger | new users will have usernames based on email address | 23:02 |
| pabelanger | if openid is missing it | 23:02 |
| EmilienM | fungi: hi, I don't know if you've time to look at https://review.openstack.org/#/c/288244/ but just to make sure it's in your radar | 23:02 |
| *** Sukhdev has joined #openstack-infra | 23:02 | |
| *** geekinutah has joined #openstack-infra | 23:05 | |
| *** edmondsw has quit IRC | 23:05 | |
| *** nwkarste_ has quit IRC | 23:05 | |
| anteaya | I think I've had as much fun as I can take for the day | 23:05 |
| anteaya | I think I'll have an evening | 23:05 |
| anteaya | have a good weekend everyone | 23:06 |
| *** Sukhdev has quit IRC | 23:06 | |
| *** geekinut1h has quit IRC | 23:06 | |
| * mnaser is curious about the big uptick in test nodes all of a sudden | 23:07 | |
| clarkb | mnaser: zuul was taking a break while we ddos'd it with jenkins updates and now the ddos is over so it is spawning a bunch of work to catch back up again | 23:07 |
| mnaser | ah makes sense | 23:07 |
| mnaser | i thought it was something like | 23:08 |
| mnaser | ~5pm EST on a friday, lets spawn everything and break everything! :p | 23:08 |
| mordred | clarkb: hey - I had stepped away a bit - did we do python->ubuntu-trusty? | 23:08 |
| mnaser | i still see a bunch of error nodes still so whenever anyone is free to pass on the logs for these failures, id appreciate it | 23:08 |
| clarkb | mordred: yes, (though jenkins updates are still in progress) | 23:09 |
| mordred | cool. I'll go see what I can find for mnaser | 23:09 |
| clarkb | mnaser: let me see what I can do | 23:09 |
| *** bauwser is now known as bauzas | 23:09 | |
| mordred | oh. or clarkb can | 23:09 |
| mordred | and I can work on releasing shade | 23:09 |
| mnaser | i mean i was meaning to bug infra team every few days so we can investigate a few of them, instead of every single time one happens :-p | 23:09 |
| *** roxanagh_ has quit IRC | 23:09 | |
| fungi | mordred: i think the jenkins masters are having to get their job updates one at a time to keep from bogging down zuul's geard to the point where job reregistrations are all it's doing | 23:10 |
| mordred | mnaser: I believe we enjoy your proactive nature | 23:10 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Teach zuul-cloner about ZUUL_REFNAME https://review.openstack.org/288790 | 23:10 |
| mnaser | it'd be nice if these logs were accessible to us but I can imagine there's a few (good) reasons they're not | 23:10 |
| fungi | mnaser: mostly it's that there are still some libraries we don't trust to not leak random credentials into debug logs | 23:11 |
| fungi | so we tend to go over them carefully before handing them out, just to be safe | 23:11 |
| mnaser | that's completely understandable. better safe than sorry | 23:11 |
| mordred | fungi: I do believe we're actually safe now | 23:11 |
| fungi | the situation's improving though | 23:11 |
| fungi | i still have hopes we're not far from being able to stuff all our service logs into a central publishing location | 23:12 |
| openstackgerrit | Merged openstack-infra/project-config: Add libjpeg to bindep-fallback.txt https://review.openstack.org/288738 | 23:12 |
| clarkb | mnaser: http://paste.openstack.org/show/489408/ there is a few of them like that one | 23:12 |
| mordred | fungi: or, hrm. once we update to shade, I'll be comfortable with it | 23:12 |
| fungi | heh | 23:12 |
| clarkb | mnaser: ssh timeouts, I can get you all the uuids and tracebacks for the ssh timouts if you want | 23:12 |
| clarkb | mnaser: there is also another class of subnode failures I need to look at | 23:12 |
| mordred | fungi: because we'll be using keystoneauth Sesison at that point for everything, and that goes to great pains to not log secrets | 23:12 |
| pabelanger | clarkb: fungi: mordred so far I am only seeing libjpeg failures for ubuntu-trusty | 23:12 |
| mordred | that's great! | 23:13 |
| mordred | pabelanger: do I want to ask what we need libjpeg for? | 23:13 |
| pabelanger | but we have a patch in flight for that | 23:13 |
| mordred | pabelanger: I'm guessing somehting sphinx related? | 23:13 |
| mnaser | clarkb: new images were uploaded after the zuul ddos possibly? | 23:13 |
| pabelanger | mordred: Ya, pillow it seems | 23:13 |
| mordred | pabelanger: lovely | 23:13 |
| pabelanger | there are flags to disable it | 23:13 |
| pabelanger | but unsure who is using it | 23:13 |
| mordred | pabelanger: adding it seems safe enough | 23:14 |
| *** geekinutah has quit IRC | 23:14 | |
| clarkb | mnaser: yes we just did it | 23:14 |
| mnaser | im pretty sure we're ddosing our own glance-api | 23:15 |
| clarkb | nice | 23:15 |
| clarkb | I mean not relaly but cool | 23:15 |
| mnaser | because there's just so many nodes hitting it at once | 23:15 |
| *** e0ne has joined #openstack-infra | 23:15 | |
| clarkb | mnaser: http://paste.openstack.org/show/489409/ is the other class of issues which appears to be a bug in nodepool's scheduling | 23:15 |
| clarkb | we should count multinode envs as X hosts not 1 | 23:16 |
| mnaser | and glance-api is located at a single az, so it has to go between the local latency link between the two datacenters | 23:16 |
| *** angdraug has quit IRC | 23:16 | |
| mnaser | so basically with most nodes being deployed at az-1 and glance-api at az-2, the bottleneck is the link connecting both | 23:16 |
| clarkb | mnaser: ah, would it help if we deployed to az2 only? | 23:16 |
| mnaser | glance-api with caching is the solution in order for us to avoid going over the link multiple times | 23:16 |
| mnaser | probably not, overall capacity is smaller there so you might see more failures. but glance-api cache in az-1 will prevent this | 23:17 |
| mnaser | we'll have to pull it once instead of N times | 23:17 |
| pabelanger | so far octavia seems unhappy with libjpeg in the gate | 23:18 |
| clarkb | pabelanger: the change to fix that isn't on any images yet | 23:19 |
| clarkb | curl should be mostly out there though | 23:19 |
| pabelanger | Yup | 23:19 |
| clarkb | couple more regions for curl | 23:19 |
| pabelanger | just scrolling zuul.o.o/status looking at failures | 23:19 |
| clarkb | fungi: any idea how we missed libjpeg-dev? | 23:19 |
| clarkb | it will be neat to see projects switch over to other-requirements and explicitly declare their deps | 23:20 |
| pabelanger | likely small sample size, we did the TC approved projects first, plus ansible and puppet | 23:20 |
| pabelanger | none seemed to need it | 23:20 |
| clarkb | once the change to update the osic clouds.yaml hits the puppetmaster I will work on booting a mirror there | 23:21 |
| fungi | clarkb: my best guess is that rackspace preinstalled it onto their images so it wasn't in our puppet, but none of the hundred or so projects/branches/jobs we tested experimentally relied on that package being present | 23:21 |
| *** piet has quit IRC | 23:21 | |
| fungi | clarkb: the other possibility is that someone added it to our puppet after i created the bindep fallback list and reviewers missed asking the author to write an additional change to update the fallback | 23:22 |
| fungi | clarkb: looks like you added it for pillow in october https://review.openstack.org/230175 | 23:23 |
| fungi | i probably should have done a fresh pass through our puppet to try to catch any which slipped through the cracks in recent months | 23:24 |
| clarkb | ya doesn't surprise me it slipped through the cracks when packages were added more confused how octavia and ironic didn't catch it | 23:25 |
| *** sdake has quit IRC | 23:25 | |
| clarkb | but I guess we didn't test those? | 23:25 |
| openstackgerrit | Merged openstack-infra/system-config: Verify SSL with OSIC https://review.openstack.org/288582 | 23:25 |
| fungi | good question, i thought ironic was on my test list but perhaps not | 23:26 |
| fungi | also looking at git history, we're coming up on a year since i put together the first vestiges of that fallback list. wow (though i know i refreshed it sometime late last year, was obviously still before 230175 merged) | 23:27 |
| *** roxanaghe has joined #openstack-infra | 23:27 | |
| fungi | clarkb: that same change also introduces zlib1g-dev/zlib-devel which is not in the fallback list either | 23:28 |
| fungi | is it also needed? | 23:28 |
| clarkb | maybe? | 23:28 |
| clarkb | Idon't think we have seen jobs fail due to it missing | 23:28 |
| fungi | the change identifies both as build-time deps of pillow 3.0.0 | 23:29 |
| *** xyang1 has quit IRC | 23:29 | |
| clarkb | ah could be that we just fail on the first on which is jpeg | 23:29 |
| clarkb | and that once that is in zlib will be the fail | 23:29 |
| pabelanger | fungi: Ah, good catch | 23:30 |
| clarkb | fqdn for osic mirror will be mirror.regionone.osic-cloud1.openstack.org | 23:31 |
| *** Sukhdev has joined #openstack-infra | 23:32 | |
| clarkb | er thats not right | 23:33 |
| * clarkb has to figure out what value goes in for osic-cloud1 | 23:33 | |
| clarkb | NODEPOOL_CLOUD | 23:34 |
| clarkb | does that map to the cloud: foo value? | 23:34 |
| *** ccrouch1 has joined #openstack-infra | 23:35 | |
| clarkb | appears to | 23:35 |
| *** thorst has joined #openstack-infra | 23:35 | |
| clarkb | which means I need to update things | 23:35 |
| *** Sukhdev has quit IRC | 23:37 | |
| *** ccrouch has quit IRC | 23:37 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Use osic-cloud1 https://review.openstack.org/288793 | 23:38 |
| clarkb | thats part of it | 23:38 |
| *** fawadkhaliq has quit IRC | 23:38 | |
| *** fawadkhaliq has joined #openstack-infra | 23:39 | |
| *** annegentle has joined #openstack-infra | 23:39 | |
| jeblair | re | 23:39 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use OSIC with nodepool https://review.openstack.org/288584 | 23:39 |
| clarkb | and thats the other part | 23:39 |
| jeblair | clarkb: yes, it's the oscc "cloud" name, so whatever we name it in clouds.yaml | 23:40 |
| clarkb | I guess I need to update the depends on | 23:41 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Be more specific in logging timeout exceptions https://review.openstack.org/287835 | 23:42 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Use OSIC with nodepool https://review.openstack.org/288584 | 23:42 |
| mordred | that should fix ianw's comments and the test failures | 23:42 |
| clarkb | there we go | 23:42 |
| *** Jeffrey4l has joined #openstack-infra | 23:43 | |
| *** sridhar_ram has quit IRC | 23:43 | |
| *** thorst has quit IRC | 23:44 | |
| *** kingia_ has joined #openstack-infra | 23:44 | |
| mordred | clarkb: wow. pool: 'GATEWAY_NET' | 23:45 |
| NobodyCam | Hey clarkb how are you doing today. just checking in to see if it worth firing a recheck off. I've been working with jroll on the libjpeg fall back issue. https://review.openstack.org/#/c/288738 seems to have landed but jim sais something about a new image? | 23:46 |
| fungi | mordred: it's the network where you find your gateway! | 23:46 |
| NobodyCam | hey hey mordred :) happy friday | 23:46 |
| mordred | morning NobodyCam ! | 23:46 |
| NobodyCam | and to fungi too | 23:46 |
| NobodyCam | :) | 23:46 |
| jeblair | clarkb: are the osic ssl things fixed? | 23:47 |
| *** kingia has quit IRC | 23:47 | |
| fungi | NobodyCam: i think there were image updates in progress to address that? i wasn't at a computer when the conversation started and sort of skimmed so far, though it's possible zlib1g-dev may also be needed since it was added in the same patch which brought in libjpeg-dev to puppet | 23:47 |
| mordred | infra-root: eveybody ready for me to fire off a shade release? | 23:48 |
| clarkb | jeblair: yes they are | 23:48 |
| fungi | jeblair: yeah, they issued a new cert (valid) and updated the keystone catalog | 23:48 |
| clarkb | I am booting the mirror now | 23:48 |
| fungi | jeblair: though it's a letsencrypt cert, so only valid for 3 months... i'm curious to see if they set up the auto-updating for it correctly | 23:48 |
| jeblair | clarkb: what's the hostname you're using? | 23:49 |
| clarkb | jeblair: the one from above, mirror.regionone.osic-cloud1.openstack.org | 23:49 |
| jeblair | yeah that looks right | 23:49 |
| fungi | NobodyCam: i'm in the process of auditing recent updates to the packages list in puppet to see if there's anything else that might have slipped through the cracks besides those | 23:50 |
| NobodyCam | awesome thank you :) I'll hold off on the recheck for a while :) | 23:51 |
| *** x00350071_ has quit IRC | 23:51 | |
| *** x00350071 has joined #openstack-infra | 23:51 | |
| *** salv-orl_ has joined #openstack-infra | 23:52 | |
| *** annegentle has quit IRC | 23:52 | |
| clarkb | hrm might run into the same volume thing in osic where vdb is config drive but cinder and nova say it is the volume | 23:54 |
| clarkb | its ok we have a workaround, but it is annoying that config drive and cinder are basically incompatible | 23:55 |
| *** salv-orlando has quit IRC | 23:55 | |
| nibalizer | mordred: im wonder why we would release on friday afternoon | 23:57 |
| fungi | clarkb: admittedly this is a nova-side issue, it just happens that the information cinder is relying on from nova is a lie (or perhaps at best a terrible guess) | 23:58 |
| mordred | nibalizer: because FUN! | 23:58 |
| rcarrillocruz | yolo | 23:58 |
| nibalizer | for who? | 23:58 |
| mordred | nibalizer: for you | 23:58 |
| nibalizer | GG | 23:58 |
| mordred | mr. west coast | 23:58 |
| * mordred waves | 23:58 | |
| * nibalizer waves back | 23:58 | |
| *** annegentle has joined #openstack-infra | 23:58 | |
| bkero | You'd release because neither of you want to have a weekend :) | 23:59 |
| mordred | nibalizer: but also we're at slow gate anyway, so we can probably release, restart nodepool, watch for carnage, and rollback if needed in like 20 minutes | 23:59 |
| mordred | nibalizer: however, I could also wait | 23:59 |
| clarkb | fungi: yes but it is still cinder's api | 23:59 |
| clarkb | fungi: regardless of where hte info comes from if you have it built into your api well I think it hsould be correct | 23:59 |
| nibalizer | we are not the only users mordred | 23:59 |
| mordred | nibalizer: oh - so, there's this other thing | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!