Friday, 2016-03-04

*** rbrndt_ has quit IRC00:01
*** austin81 has left #openstack-infra00:01
crinkletcpdump also shows it going to the mirror so it's not the puppet pip provider doing weird things00:03
*** Sukhdev has joined #openstack-infra00:03
*** ashleighfarnham has quit IRC00:04
*** sputnik13 has quit IRC00:05
*** andymaier_ has quit IRC00:06
*** ayoung has quit IRC00:07
*** sarob has quit IRC00:08
docaedoianw: I use weechat running in a screen, but also connect to it with the weechat client on my phone and often glowing-bear web client too00:08
*** sarob has joined #openstack-infra00:08
docaedoianw: I love it00:09
*** sdake has quit IRC00:09
*** geekinutah has quit IRC00:09
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci: [DNM] Check pip.conf
*** annegentle has joined #openstack-infra00:12
*** annegentle has quit IRC00:13
*** zeih has joined #openstack-infra00:14
*** sigmavirus24 is now known as sigmavirus24_awa00:15
*** zeih has quit IRC00:18
*** Sukhdev has quit IRC00:21
*** ybathia has joined #openstack-infra00:22
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Use unversioned identity endpoints
*** notnownikki has quit IRC00:23
*** Sukhdev has joined #openstack-infra00:24
mordredcrinkle: ++ ^^00:26
*** Qiming has joined #openstack-infra00:26
*** ashleighfarnham has joined #openstack-infra00:27
*** Sukhdev has quit IRC00:27
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Let zuul-cloner work in post and release pipelines
*** andymaier_ has joined #openstack-infra00:32
*** ashleighfarnham has quit IRC00:33
*** annegentle has joined #openstack-infra00:33
*** zhurong has joined #openstack-infra00:35
*** bastelfreak has quit IRC00:35
nibalizeroh i think i know00:35
openstackgerritQiming Teng proposed openstack/requirements: Bump python-senlinclient version to 0.3.0
nibalizeri think puppets pip provder hits the pypi xmlrpc endpoint to determine if it has latest00:36
nibalizeris maybe that the issue?00:36
crinklenibalizer: i tcpdumped and it looked like it was talking to the mirror00:37
nibalizeroh hrm00:38
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci: [DNM] Check pip.conf
*** angdraug has quit IRC00:40
anteayamay I have the following four gerrit accounts disabled please? 14208, 14212, 14213, 15443:
*** tqtran is now known as tqtran-afk00:40
anteayathey are all posting failed to merge messages on patches00:40
anteayawhich is distracting noise for devs00:40
*** ybathia has quit IRC00:41
anteayamy attempt to get someone from cisco to turn them off from there end failed00:41
anteayathe person I was talking to disappeared without getting back to me00:41
anteayatheir end00:41
*** bastelfreak has joined #openstack-infra00:45
openstackgerritamrith proposed openstack/requirements: Update upper-constraints.txt for python-troveclient v 2.1.0
fungiokay, here's a new one i don't have time to dig into but the release time might love someone figuring out what we broke: seems (to me) to be hitting our default vhost on static.o.o which redirects to the system-config docs00:51
fungianteaya: i can disable those four ids, sure, just a sec00:51
anteayafungi: thanks00:51
openstackgerritamrith proposed openstack/requirements: Update upper-constraints.txt for python-troveclient v 2.1.0
*** ZZelle_ has quit IRC00:53
anteayaI see this when I navigate to releases.o.o:
*** rossella_s has quit IRC00:54
fungithanks anteaya. it looks like i probably have a cached 301 redirect in that case00:54
*** rossella_s has joined #openstack-infra00:54
*** ddieterly has joined #openstack-infra00:55
*** ddieterl_ has joined #openstack-infra00:56
*** jpr has quit IRC00:56
*** annegentle has quit IRC00:56
funginot sure how to clear this from the cache without blowing away the entire browser cache00:56
anteayaI have no suggestions00:57
anteayawell nothing I think will help00:57
anteayaI do have lots of unhelpful suggestions00:58
bswartzdocaedo: ping00:58
*** akshai has quit IRC00:58
fungianteaya: good news, i've disabled those four accounts, flushed gerrit's caches and disconnected the 8 connections they had open to it amongst them00:59
anteayafungi: wonderful thank you00:59
fungiso the annoyance should cease00:59
anteayaI will compose an email and go to bed00:59
anteayakevinbenton: you should soon cease to be annoyed00:59
anteayawell to the extent I have any say in the matter that is00:59
*** ddieterly has quit IRC01:00
docaedobswartz: howdy!01:00
bswartzdocaedo: I'm looking at now01:00
bswartzthis is first time I've seen it01:00
anteayakevinbenton: you might cease to be annoyed now01:00
*** zeih has joined #openstack-infra01:00
kevinbentonanteaya: yay :)01:00
fungibswartz: there was a pretty awesome demo of it at a keynote in vancouver01:00
* bswartz had to miss vancouver summit :-(01:00
fungiwhich puts it at right about a year old now01:01
fungior coming close01:01
docaedobswartz: fire away with any questions - it DOES aim to be the authoritative place for having glance images01:01
*** AndyU has joined #openstack-infra01:01
bswartzdocaedo: so our use case requires that we can pull images out of it for running check/gate jobs01:02
*** akshai has joined #openstack-infra01:02
bswartzand we need to be able to quickly push bugfixes to images as they're found, as they can affect manila01:02
bswartzso I guess I'm curious to know what is the upload process?01:02
docaedothat's kind of the sore spot right now01:02
bswartzcan it be automated? are there humans involved? what quality checks are enforced?01:02
fungiif the primary purpose is to use the images in ci jobs then continuing to stash them on tarballs.o.o still probably makes sense, but apps.o.o would probably still be a good authoritative place for docs to refer for released images01:03
bswartzfungi: that's what I was starting to think01:03
docaedoTo add an asset right now, a human checks out the repo, edits assets.yaml in it (and adds the info there), then submits a PR01:03
AndyUclarkb nibalizer jeblair fungi etc:  have a couple more quick questions about potential infra donation01:03
fungisince the earlier comment you'd made was that you needed somewhere "official" for the install docs to link01:03
docaedoif there's a big binary chunk as well, there's a human involved01:03
bswartzmost likely we don't want to expose HEAD commit of manila-image-elements to the whole world01:04
docaedowe're working to replace that by implementing glare at the back end, which would mean the entire process could be fully automated01:04
fungiAndyU: ask away01:04
*** jaypipes has quit IRC01:04
*** zeih has quit IRC01:05
AndyUThanks!  :-)01:05
bswartzfungi: our primary purpose is images for CI jobs, but a close second is customer use -- the main difference is that customers would use tagged builds and CI would use HEAD01:05
*** ayoung has joined #openstack-infra01:05
AndyUYou said that if we host a cloud for your use it need not be plain vanilla openstack, right?  Any concerns that it would contain some customizations that have been done on our end?01:06
*** max_lobur has quit IRC01:07
*** Swami has quit IRC01:07
bswartzdocaedo: so if we output images from manila-image-elements to tarballs.o.o, could we take the tagged ones can push them into the app-catalog just by writing some yaml with a URL points to the tarballs.o.o site?01:07
fungibswartz: yeah, so tarballs.o.o for a branch tip and versioned release images, but maybe also apps.o.o for the released images and have the install docs refer to the latter01:07
*** Swami has joined #openstack-infra01:07
*** piet has quit IRC01:07
AndyUIn other words would our reference architecture for our openstack clouds be acceptable? Do you need us to be on a specific release?01:07
bswartzfungi: That sounds very appealing01:07
docaedobswartz: yes that would work01:08
*** akshai has quit IRC01:08
*** Qiming has quit IRC01:08
bswartzdocaedo: what is the turnaround time on reviewing new image submissions? I'm asking because we're working on the install guide right now and in order to test it, the image actually has to exist where we say it does01:08
fungiAndyU: you've sort of just described the state of openstack across existing providers. there are very few who just have a vanilla deployment (we sometimes like to jokingly quota randy bias who said "there is no such thing as vanilla openstack" though a lot of us disagree on that point)01:09
bswartzso if we're going to use app-catalog I need to hurry up and get a build done, tagged, and submitted01:09
fungiAndyU: so it would sort of depend on what kind of "customizations" you're talking about01:09
anteayaAndyU: and what release01:10
fungiAndyU: for example, if it prevents us being able to upload images to glance, or being able to reach instances we've booted, or being able to communicate between those instances, those would be problematic for our use case01:10
docaedobswartz: can review tonight, turnaround is usually same day (within a few hours)01:10
AndyUFungi: hard to really describe but it's all based on MOS openstack.01:10
anteayaI don't know what mos openstack is01:11
bswartzokay well my project-config patch has to land first
AndyUfungi: I can have Mike Wilson connect with you on specifics to see if anything would raise any red flags.01:11
*** Swami has quit IRC01:11
anteayabswartz: I have reviewed that already01:11
bswartzonce that merges I can tag a commit in manila-image-elements and ....HOPEFULLY..... out will pop a qcow2 file on tarballs site01:11
bswartzanteaya: I saw, ty01:12
jheskethjeblair: ping01:12
docaedobswartz: I'm stepping away for a bit for family stuff but will be back later and am generally around - definitely ping me or someone else on #openstack-app-catalog if you have any more questions, or need to try to get something reviewed ASAP01:12
AndyUfungi: Assuming customization is not an issue, what about the version of openstack?  Any concerns if we're a couple releases behind current?01:12
bswartzdocaedo: thanks, it's fine if we don't get to this until tomorrow01:13
docaedobswartz: cool01:13
fungiAndyU: as in kilo? juno? we've used them, i don't think we do anything that demands a particularly new openstack environment01:13
fungiAndyU: is mos mirantis openstack or something else?01:14
AndyUfungi: yes, mirantis01:14
*** achanda has joined #openstack-infra01:14
AndyUand yes... juno, kilo  ;-)01:14
*** blogan is now known as blogan-bot01:14
*** blogan-bot is now known as blogan01:14
AndyUfungi: it's somewhat a matter of timing01:15
anteayawas hoping it was Mos Eisley openstack01:15
fungii don't know that we've explicitly used environments based on mirantis openstack but i have a feeling they're still pretty openstacky01:15
anteayathat would be fun01:15
fungimos eisley is dangerous. always carry a blaster01:15
anteayaso I hear01:15
*** sridhar_ram has quit IRC01:16
* anteaya will be cautious01:16
AndyUfungi: but I think we'd treat it like all our other clouds and periodically upgrade in an unending quest to catch up01:16
fungiAndyU: makes sense. we don't expect 100% uptime and our ci system is able to tolerate different clouds going offline for extended periods01:16
fungiit's one of the reasons we like to use multiple clouds. better resilience that way01:17
AndyUfungi: any upgrades would be done very efficiently... they have to be. And of course you'd have ample notice.01:17
*** dprince has joined #openstack-infra01:18
AndyUfungi: so what i hear from you is that In PRINCIPLE some customization is not a problem and being 2 or 3 releases of the trunk is not a problem.01:19
*** dimtruck is now known as zz_dimtruck01:19
fungiAndyU: yes, really we do very basic things, all via the api01:19
fungiupload images, boot instances from them, log into the instances over the internet, run stuff on them, delete the instances again01:20
fungiimage uploads are refreshed once a day when things are working correctly01:20
fungiaverage lifetime of an instance is probably somewhere around 30 minutes01:20
AndyUfungi: Correct me if I have any of that wrong. I'll ask Mike Wilson to connect with you to explore some of those potential concerns and see where we end up.01:21
AndyUfungi: ok, I type slow. You were already there ahead of me  ;-)01:21
fungithey're generally 8gb ram, hopefully with somewhere around 8 vcpus and at least 80gb disk each01:21
AndyUfungi: right01:22
fungiand we run as many in parallel as we can during peak load, though virtually none when things are quiet (we're very elastic)01:22
*** kzaitsev_mb has quit IRC01:22
fungiand that's basically it01:23
AndyUfungi I just was hit with a few questions and I wanted to double check with you before I responded. It's all in an exploratory stage at the moment so if he hit a bump, we'll adjust the plan01:23
anteayathings are quietening down, 30 in check, 20 in the gate01:23
fungicool, and thanks for wanting to pitch in!01:23
anteayaAndyU: once the cloud is ready we start with one node and troubleshoot from there01:24
AndyUfungi: thanks for the info!  My confidence in actually succeeding on this quest is growing  ;-)  I'll surely be in touch again.01:24
anteayaAndyU: then we up slowly and see what happens01:24
*** zz_dimtruck is now known as dimtruck01:25
fungiokay, i think i'm going to wrap up for the night01:25
AndyUfungi: Maybe we could get you access to a few tenants to just check things out. That would probably be a really good idea so that we don't get too far and find out it won't work.01:25
AndyUI'll look into it. Have a good night!01:26
anteayafungi: yup, have a good night01:26
fungiAndyU: sounds like that would be a useful exploration01:26
tonybfungi: do you have time for a quick election question?01:26
fungii'm also not around much tomorrow, on the road a bunch of the day, though i'll pop in as i can in case there are emergencies01:26
fungitonyb: sure, what's up?01:26
anteayafungi: hope you have good driving conditions01:27
tonybfungi: WRT generating the roles and having the correct email adress in gerrit01:27
fungiantqit's supposed to be rainy and windy, but at least nothing frozen01:27
fungier, anteaya ^01:27
anteayaoh good, no ice01:27
*** sdake has joined #openstack-infra01:27
fungitonyb: electoral rolls? what's the question about addresses?01:27
tonybfungi: that's only an issue for the generation of the roles.  I'm assuming that as long as they're valid at that time as opposed to the start of the election all things are good01:27
anteayawell in the past I communicated that the electorate should confirm their gerrit prefered address01:28
tonybfungi: Really I just need to pick a good date to say "make sure the email address in gerrit is right by $date"01:28
*** thorst_ has quit IRC01:28
tonybanteaya: right I want to set that date01:28
anteayaas that would be the address the poll was sent to01:28
fungitonyb: i suppose it's possible some will shift out from under us. when the rolls are generated they refledt the gerrit preferred e-mail addresses for each voter01:28
tonybthe current draft is today which is clearly uncool01:28
*** thorst has joined #openstack-infra01:29
anteayathe roll was created on the after the last day of eligibility01:29
anteayawhich in the past was the day prior to the start of the ptl nomination period01:29
bswartzoh god is it election time already?01:30
tonybanteaya: okay thanks.  That's what I was looking for.01:30
bswartzdid I miss a deadline?01:30
*** dingyichen has quit IRC01:30
tonybbswartz: soon01:30
anteayatonyb: yeah look for the archived emails about timelines01:30
anteayaand the wikipages01:30
fungiyeah, so basically if we have _too_ long of a delay between when rolls are generated and when the ballots are sent, it's possible some voters' addresses might change/become invalid01:30
anteayait should all be there01:30
anteayatonyb: I can dig this up for you tomorrow01:30
tonybfungi, anteaya: thanks.01:30
anteayaunless you are tight for time and need it tonight01:30
tonybanteaya: no need I'm pretty sure I have a solid plan.01:31
*** dingyichen has joined #openstack-infra01:31
anteayahow much time have you given yourself before you post an information email01:31
*** akshai has joined #openstack-infra01:31
anteayawell taht was what happened in the past01:31
anteayait was up to the electorate to check their email01:31
fungias for what anteaya is saying, the reasoning is 1. so that i have a day to be leisurely about roll generation without delaying the start of the election, and 2. so that i only have to do it once and you can use the rolls generated from that point in time for the tc election and for any ptl elections which come up01:31
anteayayes, what fungi said01:32
tonybanteaya, fungi: +101:32
anteayaI also used to give the electorate about 4 weeks to get themselves collected and confirm their email address was correct01:32
anteayayou always get some who can't find the email but it minimized that01:32
tonybAll the dates are public just not announced, so I'm going to set the "deadline" to PTL_start - 24 hours01:33
fungialso you need ttx, as the tc chair, to tag a point in time in the governance repo so that we can document the state of the reference material used to determine which project teams are involved and what deliverable repos count toward each of them01:33
anteayayes that01:33
tonybfungi: okay we'll line that up.01:34
fungiwhich also serves as a cut-off point for extra-atc additions counting toward electorate eligibility01:34
tonybfungi: makes sense, but I have to admit it didn't occur to me that extra-atc's get to vote.01:35
fungithough since those tend to only get merged on tc meeting days, that cutoff is potentially as much as week earlier than when the tag is pushed01:35
*** sdake has quit IRC01:35
tonybfungi: :)01:35
openstackgerritMerged openstack/requirements: Bump upper constraint django_openstack_auth
fungitonyb: the existence of extra-atcs tracking _primarily_ to enable them to vote01:35
anteayatonyb: it is in the by-laws01:35
fungithat they also get free admission to the conference and summit is secondary01:36
AndyUanteaya:  thanks for your input too. reading back I felt like I neglected to address you as well as fungi.  I kinda loved the Mos Eisley comment  ;)  Bye for now.01:36
tonybfungi: cool.  Thanks so much for letting me de-rail your plans to unplug.01:37
*** thorst has quit IRC01:37
*** mtanino has quit IRC01:37
anteayatonyb: 3.b.ii
fungianyway, i concur with getting your public communication to the electoral body going as soon as you can. as anteaya says, you need to warm them up and get them prepared01:37
*** yamahata has quit IRC01:37
tonybfungi: Yeah the plan is tomorrow.01:38
anteayaAndyU: thanks for coming back, yeah was feeling ignored, it happens sometimes, thanks for saying so01:38
anteayaAndyU: talk to you later01:38
tonybanteaya: Thanks.01:39
anteayatonyb: welcome01:39
fungiokay, night all01:39
*** akshai_ has joined #openstack-infra01:39
AndyUanteaya: My brain was all over paying your great attention... but my fingers failed to follow!  Sorry about that.01:39
anteayano worries it happens01:39
anteayajust thanks for saying it wasn't intentional01:40
anteayaattend a tc meeting sometime01:40
anteayanoone can keep up01:40
*** aeng has quit IRC01:40
AndyUyw... my bad01:40
anteayano worries01:40
anteayaI also have to offline01:40
*** ashleighfarnham has joined #openstack-infra01:41
anteayanight all01:41
*** _sarob has joined #openstack-infra01:42
*** akshai has quit IRC01:42
*** AndyU has quit IRC01:43
*** sam_wan has joined #openstack-infra01:44
openstackgerritJim Rollenhagen proposed openstack/requirements: Bump u-c for python-ironicclient 1.2.0
*** sarob has quit IRC01:45
*** _sarob has quit IRC01:46
*** Qiming has joined #openstack-infra01:47
openstackgerritMerged openstack/requirements: Raise upper-constraint on python-congressclient
*** sflanigan has quit IRC01:49
*** dmsimard has quit IRC01:50
*** dmsimard has joined #openstack-infra01:51
openstackgerritDoug Hellmann proposed openstack/requirements: Bump python-glanceclient to 2.0.0
openstackgerritMerged openstack/requirements: Raise manilaclient upper to 1.8.0
*** apoorvad has quit IRC01:53
openstackgerritFei Long Wang proposed openstack/requirements: Bump upper-constraints for python-zaqarclient 1.0.0
*** sflanigan has joined #openstack-infra01:53
*** yamamoto has joined #openstack-infra01:53
*** tqtran-afk has quit IRC01:54
*** Sukhdev has joined #openstack-infra01:54
*** yuanying has quit IRC01:58
*** yuanying has joined #openstack-infra01:59
*** zeih has joined #openstack-infra02:01
*** esker has joined #openstack-infra02:01
*** Sukhdev has quit IRC02:04
*** zeih has quit IRC02:05
*** esker has quit IRC02:07
*** pvaneck has quit IRC02:08
openstackgerritSam Yaple proposed openstack-infra/shade: Fixes for latest cinder and neutron clients
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
*** gyee has quit IRC02:15
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: WIP: IPv4 network isolation testing
*** bpokorny_ has joined #openstack-infra02:16
*** thorst has joined #openstack-infra02:16
*** bpokorny has quit IRC02:19
*** tiswanso has quit IRC02:19
*** fawadkhaliq has quit IRC02:20
*** tiswanso has joined #openstack-infra02:20
openstackgerritMerged openstack/requirements: bump upper constraints of openstackclient
*** bpokorny_ has quit IRC02:21
*** thorst has quit IRC02:23
*** yamamoto has quit IRC02:23
*** yamamoto has joined #openstack-infra02:23
clarkbok I am back again02:26
*** andymaier_ has quit IRC02:26
clarkbanything related to release/feature freeze need attention?02:26
*** yamamoto has quit IRC02:29
*** ccrouch has quit IRC02:32
*** ccrouch has joined #openstack-infra02:33
*** apoorvad has joined #openstack-infra02:36
funginot that i've seen02:36
*** sdake has joined #openstack-infra02:39
*** dprince has quit IRC02:40
*** esker has joined #openstack-infra02:43
*** dimtruck is now known as zz_dimtruck02:43
*** woodster_ has quit IRC02:47
openstackgerritMerged openstack/requirements: Bump upper-constraints for python-neutronclient 4.1.0
*** esker has quit IRC02:48
*** akshai_ has quit IRC02:51
*** sdake has quit IRC02:51
*** sdake has joined #openstack-infra02:52
*** sdake has quit IRC02:52
*** piet has joined #openstack-infra02:53
*** mgagne has left #openstack-infra02:56
*** akshai has joined #openstack-infra02:56
*** mgagne has joined #openstack-infra02:56
*** annegentle has joined #openstack-infra02:57
*** akshai has quit IRC03:00
*** links has joined #openstack-infra03:01
*** zeih has joined #openstack-infra03:02
openstackgerritMerged openstack/requirements: Add python-searchlightclient>=0.2.0
*** annegentle has quit IRC03:03
*** xiangxinyong has quit IRC03:04
*** kevinsho_ has quit IRC03:04
*** sflanigan is now known as sflanigan_afk03:04
*** sflanigan_afk is now known as sflanigan03:04
*** sflanigan is now known as sflanigan_afk03:05
*** lykinsbd has joined #openstack-infra03:05
*** lykinsbd has quit IRC03:06
*** zeih has quit IRC03:07
*** lykinsbd has joined #openstack-infra03:07
*** harlowja_at_home has quit IRC03:07
*** lykinsbd has quit IRC03:11
*** lykinsbd has joined #openstack-infra03:11
*** yamamoto has joined #openstack-infra03:11
*** lykinsbd has quit IRC03:12
*** kevinsho_ has joined #openstack-infra03:12
*** sflanigan_afk is now known as sflanigan03:13
*** esikachev has joined #openstack-infra03:13
*** piet has quit IRC03:13
*** piet has joined #openstack-infra03:14
*** amrith is now known as _amrith_03:15
*** yuanying has quit IRC03:16
*** esikachev has quit IRC03:18
*** esker has joined #openstack-infra03:18
*** lykinsbd has joined #openstack-infra03:19
*** thorst has joined #openstack-infra03:21
openstackgerritTim Buckley proposed openstack-infra/puppet-subunit2sql: Fix log_url parsing for new job types
*** rhallisey has quit IRC03:27
*** thorst has quit IRC03:28
*** aeng has joined #openstack-infra03:37
*** fawadkhaliq has joined #openstack-infra03:38
openstackgerritMerged openstack/requirements: Update upper-constraints.txt for python-troveclient v 2.1.0
*** baoli has joined #openstack-infra03:39
*** dims has quit IRC03:40
*** yamahata has joined #openstack-infra03:41
openstackgerritMerged openstack/requirements: Bump ironic-lib to 1.1.0
*** dims has joined #openstack-infra03:43
*** dims has quit IRC03:44
*** links has quit IRC03:44
*** apoorvad has quit IRC03:45
*** _ody_ has joined #openstack-infra03:47
*** esker has quit IRC03:49
*** _ody has quit IRC03:50
*** rguillebert has quit IRC03:57
*** jamesmcarthur has joined #openstack-infra04:00
*** asselin__ has joined #openstack-infra04:02
*** zeih has joined #openstack-infra04:02
*** zeih has quit IRC04:07
*** yuanying has joined #openstack-infra04:08
openstackgerritMerged openstack/requirements: Update upper-constraints for barbianclient 4.0.0
*** yuanying has quit IRC04:12
openstackgerritEmilien Macchi proposed openstack-infra/system-config: long.defs/redat: set SYS_UID_MIN and SYS_GID_MIN
EmilienMianw: hey! if you don't ming, please review ^04:12
*** jamesmcarthur has quit IRC04:15
*** jamesmcarthur has joined #openstack-infra04:20
*** Jeffrey4l has joined #openstack-infra04:22
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
*** tojuvone_ has joined #openstack-infra04:27
*** sam_wan has quit IRC04:27
*** zz_dimtruck is now known as dimtruck04:27
*** piet has quit IRC04:27
*** thorst has joined #openstack-infra04:27
*** esker has joined #openstack-infra04:27
*** links has joined #openstack-infra04:27
*** baoli_ has joined #openstack-infra04:27
*** jamesmcarthur_ has joined #openstack-infra04:27
*** openstack has joined #openstack-infra04:29
*** jamesmcarthur has quit IRC04:29
*** jamesmcarthur_ is now known as jamesmcarthur04:29
*** markmcclain has joined #openstack-infra04:29
*** alex_xu has joined #openstack-infra04:29
*** dingyichen has quit IRC04:29
*** ZZelle has quit IRC04:29
*** yuanying has joined #openstack-infra04:30
*** baoli_ has quit IRC04:30
*** ccrouch has quit IRC04:30
*** dtantsur|afk has quit IRC04:30
*** cznewt has quit IRC04:30
*** gildub has quit IRC04:30
*** cznewt has joined #openstack-infra04:30
*** openstack has quit IRC04:30
*** openstack has joined #openstack-infra04:32
*** jlk has joined #openstack-infra04:33
*** jlk has quit IRC04:33
*** jlk has joined #openstack-infra04:33
*** thorst has quit IRC04:33
*** dangers_away has quit IRC04:33
*** esker has quit IRC04:33
openstackgerritEmilien Macchi proposed openstack-infra/system-config: login.defs/redhat: set SYS_UID_MIN and SYS_GID_MIN
*** openstack has quit IRC04:34
*** openstack has joined #openstack-infra04:39
*** openstack has quit IRC04:39
*** openstack has joined #openstack-infra14:07
lucasagomeshi all, if you have time can you please take a look at ? This patch is the final piece that enables the ipmi drivers in ironic to be tested in gate14:07
lucasagomesit already has a +2 and the patch is very small (3LOC)14:07
*** amitgandhinz has joined #openstack-infra14:07
*** _nadya_ has quit IRC14:08
*** zeih has joined #openstack-infra14:10
openstackgerritMerged openstack/requirements: Bump dogpile.cache to 0.5.7
pabelangerAJaeger: going to use as topic14:10
*** _nadya_ has joined #openstack-infra14:11
AJaegeryeah, good idea. Can you change topics?14:11
* AJaeger suggests to merge and next - then project-config, then python jobs (with manual JJB run).14:12
openstackgerritMerged openstack/requirements: Add couchdb to global requirements
AJaegerfor python jobs I'd like to have some more infra-root around ;)14:12
openstackgerritMerged openstack/requirements: Bump django-compressor to 2.0
openstackgerritMerged openstack/requirements: Add oauth2client to requirements
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Fix on category groups expand
AJaegeryolanda, mordred : could you review 288446, please?14:13
*** mhu has quit IRC14:14
*** mhu has joined #openstack-infra14:14
*** mriedem has joined #openstack-infra14:15
*** piet has quit IRC14:16
pabelangerAJaeger: Yup, already on the list14:17
pabelangerAJaeger: I'm working ansible-jobs patch14:17
*** yamamoto_ has quit IRC14:17
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch to ansible-func-ubuntu-trusty for ansible jobs
openstackgerritAkihiro Motoki proposed openstack/requirements: Block neutronclient 4.1.0 to avoid gate breakage
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch to ansible-func-ubuntu-trusty for ansible jobs
*** amotoki has quit IRC14:21
*** Daisy has joined #openstack-infra14:22
*** dkranz has quit IRC14:22
*** ildikov has joined #openstack-infra14:24
pabelangerAJaeger: is there a bindep patch in flight ATM?14:25
pabelangerAJaeger: just to know what I should be watching for14:25
*** moravec has joined #openstack-infra14:28
*** dkranz has joined #openstack-infra14:29
*** dansmith is now known as superdan14:29
*** rbrndt has joined #openstack-infra14:34
*** kencjohnston has joined #openstack-infra14:34
*** piet has joined #openstack-infra14:36
*** openstackgerrit_ has quit IRC14:36
*** openstackgerrit_ has joined #openstack-infra14:37
*** sorantis has joined #openstack-infra14:37
*** ddieterly has joined #openstack-infra14:38
*** dkranz has quit IRC14:39
SamYaplewe use pypi mirrors thats not whats slowing down centos gate14:39
*** eharney has joined #openstack-infra14:39
SamYaplewe dont have package mirrors for rpm for centos14:39
sdakei recognize we dont have package mirrors for rpm14:40
sdakeclarkb said earlier we are lacking pypi wheel prebuilt binaries on centos as well14:40
sdakein the infrastructure14:40
sdakenot in our gating code14:40
sdakebut in openstack inra14:40
*** links has quit IRC14:41
*** icey has quit IRC14:42
*** icey has joined #openstack-infra14:43
AJaegerpabelanger: nothiing in flight right now - and nothing coming from me for the moment14:43
AJaegeryolanda, thanks for the review. Do you have time for a +2A for, please?14:43
AJaegerpabelanger: 288471 is nice in that reduces jobs ;)14:44
pabelangerAJaeger: Ya, I think 288471 should be safe to do14:44
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for neutron/master
pabelangerAJaeger: do we want to ask for another +2 from somebody?14:44
openstackgerritMerged openstack/requirements: Updated from generate-constraints
pabelangerI confirmed with odyssey4me that his team is ready for the flip to ubuntu-trusty14:45
*** asettle has joined #openstack-infra14:45
*** nwkarsten has joined #openstack-infra14:45
AJaegerpabelanger: yolanda just gave a +2.14:47
AJaegerpabelanger: thanks14:47
pabelangerAJaeger: yolanda: \o/14:48
pabelangerwill be testing sortly14:48
AJaegerpabelanger: that's what I did as well - run a couple of rechecks ;)14:49
*** zz_dimtruck is now known as dimtruck14:49
AJaegerI did this on the manual jobs we merged14:49
*** ddieterly has quit IRC14:49
pabelangerAJaeger: Ya odyssey4me already has patches up for other-requirements.txt so think we'll be good for rechecks14:50
*** asettle has quit IRC14:50
AJaegerpabelanger: do they really need other-requirements? Or is that an optimization?14:51
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for neutron/master
*** ddieterly has joined #openstack-infra14:52
pabelangerAJaeger: Ya, optimization. Since ansible installs packages by default, using empty other-requirements.txt will be a good way to ensure roles actually install dependencies14:52
*** abregman|afk is now known as abregman14:52
openstackgerritMonty Taylor proposed openstack-infra/shade: Use warlock in the glance v2 tests
openstackgerritMonty Taylor proposed openstack-infra/shade: Mock glance v1 image with object not dict
*** ddieterly has quit IRC14:53
*** _nadya_ has quit IRC14:53
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches
AJaegerpabelanger: ok, let ansible play with it ;)14:54
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for neutron/master
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches
*** piet has quit IRC14:56
AJaegerihrachys: is there one patch we should merge today?14:57
ihrachysAJaeger: not really, I still try to set all patches proper and pass gate. still in the middle of the process for the set.14:57
ihrachysI think we can wait till next week14:57
ihrachysas long as we clean it during the next week, we are good14:58
*** abregman is now known as abregman|afk14:58
AJaegerihrachys: I'm not much available early in the week but can review before - and help push it through if needed later14:58
*** mcarden has quit IRC14:59
*** jcoufal_ has quit IRC14:59
*** ccrouch has quit IRC15:00
*** mcarden has joined #openstack-infra15:00
*** tonyb has quit IRC15:01
*** tonyb has joined #openstack-infra15:01
openstackgerritMerged openstack-infra/project-config: Switch to ansible-func-ubuntu-trusty for ansible jobs
*** kgiusti has quit IRC15:02
AJaegerpabelanger: Wait 30mins before you recheck so that those jobs are updated ^15:03
*** woodster_ has joined #openstack-infra15:03
*** piet has joined #openstack-infra15:04
pabelangerAJaeger: ack15:04
AJaegeror wait: Enough to wait for next zuul reconfigure, you're not adding/changing jobs on jenkins15:05
openstackgerritDan Prince proposed openstack-infra/tripleo-ci: WIP: IPv4 network isolation testing
anteayasmcginnis: so patches to cinder's unit tests don't run any integrated tests in the gate?15:05
*** yamamoto has joined #openstack-infra15:05
*** yamamoto has quit IRC15:05
AJaegerSo, let's watch for "Last reconfigured"15:05
AJaegermorning, anteaya15:06
anteayaAJaeger: morning Andreas15:06
*** vgridnev has quit IRC15:06
smcginnisanteaya: There was a change recently to filter out some of the tests based on if it is a docs/reno/tests only change.15:06
*** yamamoto has joined #openstack-infra15:06
smcginnisAJaeger: That was you, right? ^^15:06
anteayaright I'm familiar with not running intregrated jobs in the gate on docs or reno tests15:07
anteayajust confirming what I am seeing15:07
AJaegeranteaya: we're currently moving jobs from bare-trusty to ubuntu-trusty, so if any questions about strange jobs show up, let's investigate. If you want to review, see
AJaegersmcginnis: let me check...15:07
*** zeih has quit IRC15:07
AJaegersmcginnis: I673db66cbfba573182abaa200e13546715881c82 - mriedem did it15:08
smcginnisAJaeger: Oh, right. Thanks. THere you go anteaya ^^15:08
anteayaso this behaviour is as intended?15:08
smcginnisanteaya: Does make the infra load a little lighter, so I do like it.15:08
openstackgerritMonty Taylor proposed openstack-infra/shade: Fixes for latest cinder and neutron clients
openstackgerritMonty Taylor proposed openstack-infra/shade: Use warlock in the glance v2 tests
openstackgerritMonty Taylor proposed openstack-infra/shade: Mock glance v1 image with object not dict
anteayasmcginnis: okey dokey15:09
anteayaAJaeger: can I ask a favour?15:09
AJaegeranteaya: sure15:09
*** kgiusti has joined #openstack-infra15:09
anteayado you mind sharing the patch number of a patch rather than the change id in channel?15:09
anteayawhen searching on a change id gerrit returns every reference for it15:09
anteayaincluding patches that reference that change id15:10
AJaegeranteaya: I can try - I was going from git ...15:10
AJaegeranteaya: 28618015:10
*** markvoelker has quit IRC15:10
*** markvoelker has joined #openstack-infra15:10
anteayaAJaeger: you are a bit of a trend setter15:11
AJaegerwhat have I done again? ;)15:11
anteayajust hoping the crowd wasn't follwing you into change ids :)15:11
AJaegeranteaya: I see...15:11
anteayano just explaining myself further15:11
*** ccrouch has joined #openstack-infra15:12
*** claudiub has joined #openstack-infra15:12
*** cznewt has quit IRC15:12
jeblairclarkb, mordred: yes, pabelanger and krotscheck were looking into centos afs... i think there was a wip patch15:12
krotscheckjeblair, mordred: I'm currently focused on fixing a mitaka release blocker.15:13
krotscheckAnd ironic things15:13
anteayaokay so here is my follow up argument, why do we need patches that don't run integrated tests in the integrated gate?15:13
*** annegentle has quit IRC15:14
anteayathey just line up with the others, risk resetting the gate and they aren't running integrated tests15:14
*** esker has joined #openstack-infra15:14
anteayaso what is the point?15:14
EmilienMfungi: when you have some time today, login.defs patch
jeblairclarkb, mordred, krotscheck, pabelanger: apparently there's an abandoned patch:
*** esker has quit IRC15:15
*** jswarren_ is now known as jswarren15:15
*** esker has joined #openstack-infra15:15
krotscheckI have un-abandoned that15:15
krotscheckpeople keep asking about it.15:16
*** amitgandhinz has quit IRC15:16
krotscheckBut I'm not certain I'm comfortable with the way I'm selecting the version15:16
morgankrotscheck: quick reabandon it to confuse people more! :P15:16
* morgan goes back under a rock.15:16
jeblairanteaya: when you search using a change id, how do you go about doing that?15:17
anteayaI copy the change id and paste it into the gerrit search text box15:17
anteayaI do believe old gerrit would just return the patch that had that change id as its change id15:18
jeblairanteaya: that should only return results that are that change id15:18
anteayanew gerrit gives you every patch that mentions it15:18
*** vgridnev has joined #openstack-infra15:18
anteayaI would prefer if it did that15:18
anteayaold gerrit used to15:18
jeblairanteaya: (which is usually just one, in which case it goes directly to the change)15:18
anteayathat is the old behaviour15:19
*** hashar has quit IRC15:19
anteayaand my preference15:19
mordredkrotscheck: fwiw, it's nowhere near the most terrible thing I've ever seen15:19
anteayain which case sharing change ids was efficient15:19
krotscheckHey, anyone want to collaborate on an April Fool's RFC? "TCP Packet engraving for ultra-long network delays"? Recommendations on materials on which your TCP packets can be etched based on how many years of latency you're expecting, and/or how long the government wants you to keep it.15:19
jeblairanteaya: ^15:20
*** vgridnev has quit IRC15:20
anteayajeblair: thank you15:20
ihrachysAJaeger: ok, I think I am set for the most part. it seems we really want to get in though it depends on another neutron patch, so I gotta chase for reviews there first.15:20
anteayajeblair: if you look at
*** jpr has joined #openstack-infra15:20
*** jcoufal has joined #openstack-infra15:21
jeblairanteaya: compare those two searches.  the first is just on the change id and it returns exactly one result.  the second is for the change id within the commit message and returns 315:21
anteayajeblair: if you click on the change id in the commit message after "this reverts commit"15:21
ihrachysAJaeger: there is another patch to switch stable/liberty to regular target names, but that will need to wait for a backport15:21
*** piet has quit IRC15:21
anteayajeblair: gerrit links to two patches15:21
*** sorantis has quit IRC15:21
jeblairanteaya:,I2316d13e90210d641541fdc18d9357d085971346,n,z  ?15:21
jeblairanteaya: that link returns 2 patches for you?15:22
mordredthat does15:22
anteayaclick on This reverts commit 6495027a8a2dac1870d39716c44a85348e8bffad.15:22
mordredthe commit link - not the depends-on reference15:22
anteayathis is new gerrit behaviour15:22
mordredyah - that's a search for a git sha, not a gerrit change-id15:22
AJaegerihrachys: I'll review later...15:22
anteayagerrit used to just take you do the patch that it is reverting15:22
*** jed56 has quit IRC15:23
anteayaokay well I'm gerriting wrong then15:23
anteayaI'm fine if this is my fault and I should be doing something else15:23
mordredanteaya: I don't thnk it's about fault - just wanting to make sure we understand which things does which thing15:24
AJaegerjeblair, mordred: As next step for going from bare-trusty to ubuntu-trusty I'd like us  to dogfoot project-config - . Are you around in case any problems arise since a revert might need a manual merge?15:24
anteayabut I do find 6 digits easier to track than shas15:24
*** cznewt has joined #openstack-infra15:25
anteayamordred: sure that's fine15:25
anteayavisually track them that is15:25
mordredAJaeger: yah15:25
jeblairanteaya: i don't know about that, but i think the old behavior of searching on a bare changeid still returns only changes that are that changeid.  on the other hand, searching on a commit sha does indeed search within messages.15:25
jeblairanteaya: i agree about the shorter numbers :)15:26
*** nwkarsten has quit IRC15:26
*** vgridnev has joined #openstack-infra15:26
*** kzaitsev_mb has quit IRC15:26
*** bhunter71 has quit IRC15:27
*** yamahata has joined #openstack-infra15:28
*** kencjohnston has quit IRC15:28
anteayaI didn't realize that fungi's commit message about the revert included the git sha rather than the change id15:29
anteayaI guess since I had always used or seen the change id in commit messages about reverts I just assumed change id usage was universal15:30
anteayaso that is my incorrect assumption15:30
AJaegerOk, then I +A it.15:30
*** vgridnev has quit IRC15:30
anteayaso perhaps I am mistaken about gerrit's behaviour15:31
AJaegerI think the changed behaviour is that gerrit inlcudes the git sha instead of the change ID in a revert message now.15:31
jeblairanteaya: you can tell a git commit sha from a gerrit change-id by whether it has a leading "I".  if it does, it's a gerrit change id.15:31
anteayaoh okay thanks, I didn't know that15:32
anteayathat helps15:32
openstackgerritClaudiu Belu proposed openstack/requirements: Bumps os-win version to 0.2.3
anteayaAJaeger: oh perhaps that is it15:32
pabelangerjeblair: mordred: Ya, working on it while bindep stuff merges now.  I might have an initial patch today15:33
jeblairanteaya, AJaeger: some folks may create a revert commit with git itself, rather than gerrit.  especially if they are creating it ahead of time.  i don't know gerrit's behavior when creating a revert (i almost never use gerrit to do that), but git will only leave the sha, not the change-id, in the message).15:33
*** Daisy has quit IRC15:33
*** jsavak has quit IRC15:34
AJaegerjeblair: was done from review.openstack.org15:34
*** david_lyle__ has joined #openstack-infra15:34
AJaegerand it shows the git SHA15:34
*** ZZelle has quit IRC15:34
mordredgotcha. so we'll always get a git SHA in revert messages no matter where they come from15:35
AJaegerjeblair: has the usual "this change got reverted" message15:35
*** dizquierdo has joined #openstack-infra15:35
openstackgerritMark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin
openstackgerritClaudiu Belu proposed openstack/requirements: Bumps os-win version to 0.2.3
*** vgridnev has joined #openstack-infra15:36
*** david-lyle has quit IRC15:37
*** vgridnev has quit IRC15:37
*** aysyd has quit IRC15:38
odyssey4meoh dear, something seems to have changed in the docs publishing - is now showing content from our liberty branch, instead of from the master branch15:38
* AJaeger likes to increase min-ready for ubuntu-trusty while we convert jobs - are these values fine? Please review
*** aysyd has joined #openstack-infra15:38
*** yamamoto has quit IRC15:39
pabelangerAJaeger: LGTM15:39
*** vgridnev has joined #openstack-infra15:39
AJaegerodyssey4me: is 12.0.7 your latest tag? When was it tagged?15:39
odyssey4meAJaeger today, earlier today15:40
*** yamahata has quit IRC15:40
openstackgerritMerged openstack-infra/project-config: Use ubuntu-trusty for project-config
*** regXboi has joined #openstack-infra15:40
AJaegerodyssey4me: So, did not solve the problem it should have solve - any project-config expert interested to take it to the drawing board again?15:40
pabelangerAJaeger: is ubuntu-trusty on rax broken?15:40
AJaegerpabelanger: - it was broken - jeblair, clarkb, fungi, mordred : Do you know whether it's safe to enable again?15:41
*** vgridnev has quit IRC15:41
pabelangerAJaeger: Ah, ya. That is right15:42
pabelangerforgot about that15:42
clarkbnot yet need shade release15:42
anteayajeblair AJaeger thanks for clearing that up for me, at least I understand what is happening now15:42
AJaegerclarkb: I see - thanks15:42
AJaegeranteaya: I learned something as well ;)15:42
odyssey4meAJaeger ah, interesting - I see that is published... so the only issue is that the master branch content isn't published15:42
AJaegerodyssey4me: the issue is that we copied wrongly to the "normal" location.15:43
AJaegerodyssey4me: I tried to fix it as referenced but failed ;(15:43
AJaegerodyssey4me: your branch is called just "liberty", not "stable/liberty", correct?15:44
odyssey4meAJaeger yes, correct - we have 'liberty' and 'kilo'15:45
AJaegerodyssey4me: So, my fix solved it for all projects that use stable - see " echo $ZUUL_REFNAME | grep stable/"15:45
AJaegerodyssey4me: Please tag stable/mitaka.15:45
odyssey4meAJaeger we plan to15:45
AJaegerodyssey4me: with your next merge to openstack-ansible the doc publishing will override your content, so merge something ;)15:46
odyssey4meAJaeger ok cool - we'll have some merges coming in soon enough - thanks15:46
AJaegerodyssey4me: changes still welcome ;)15:47
AJaegeranteaya: regarding 4 or 8: I don't know. I wanted to start slowly. Feedback is welcome on what values to choose here...15:47
AJaegerI considered doing it in one steps: One increase now, another later with lowering bare-trusty min-ready...15:48
*** mtanino has joined #openstack-infra15:49
*** andreykurilin has left #openstack-infra15:49
*** sorantis has joined #openstack-infra15:49
fungiaround for a bit from a lobby15:49
fungilooks like the cut-over has not caused any pain yet?15:50
AJaegerfungi, manuals look fine.15:50
AJaegerpabelanger: what's the status for ansible?15:50
pabelangerAJaeger: live and working15:50
AJaegerfungi, project-config just merged.15:50
odyssey4meAJaeger I'm looking at how that all works, and I don't see where the problem lies. echo $ZUUL_REFNAME | sed 's/stable.// will work perfectly against our branch names (all you're doing is removing the 'stable/' from the branch ref... as we don't have one it remains unchanged15:51
AJaegerpabelanger: woot15:51
*** kiran-r has quit IRC15:51
AJaegerodyssey4me: check the complete file15:51
anteayaAJaeger: okay well I'm +2 on 4, the stated value in the patch but would also be okay with 815:51
anteayaAJaeger: and if anyone else shares feedback, I'm curious15:52
*** Sukhdev has joined #openstack-infra15:52
*** esikachev has quit IRC15:52
AJaegeranteaya: I'M as well. Fungi, what about merging
*** x00350071 has quit IRC15:53
*** dims is now known as dimsum_15:53
*** kiran-r has joined #openstack-infra15:53
*** x00350071 has joined #openstack-infra15:54
fungiAJaeger: anteaya: 4 seems fine for now. it's really only a means of making sure we have enough kept ready when zuul is mostly idle so jobs don't need to wait for additional nodes to be booted15:55
*** yamamoto has joined #openstack-infra15:55
*** mriedem is now known as mriedem_meeting15:55
fungiwe can tune it later as we see job volume for u-t increase and b-t decreate15:55
anteayafair enough15:56
fungiadjusting it isn't terribly urgent, but also won't really hurt as long as we're using them15:56
AJaegeranteaya, fungi: could either of you review 288446,please - that changes api-jobs15:57
pabelangerAre we having issues with bluebox?
pabelangerdon't see many nodes in use15:57
pabelangerout of floating IPs?15:57
fungipabelanger: probably floating ip leak again. i'll check15:57
anteayajenkins 06 just deleted 37 nodes15:58
anteayathat is the only outlier I see on grafana at the moment15:58
anteayathat is the only jenkins outlier I see at the moment15:58
*** nwkarsten has joined #openstack-infra15:59
fungijenkins06 may be unhappy, looking now15:59
*** yamahata has joined #openstack-infra15:59
fungimy charger wasn't working in the car for some reason, so i'm on limited battery life too, just fyi... could disappear sooner than expected16:00
*** thorst has quit IRC16:00
*** thorst has joined #openstack-infra16:01
anteayafungi: yup16:02
*** amitgandhinz has joined #openstack-infra16:02
anteayafungi: thanks, not critical just looked odd to me16:02
*** jsavak has joined #openstack-infra16:03
fungijenkins06 does indeed seem unhealthy. i'm going to put it into prepare or shutdown for now and once it's idle i can clean it up and restart it the next time i'm around unless another infra-root beats me to it16:03
*** nwkarsten has quit IRC16:03
clarkbfungi: maybe just run the playbook against it in screen?16:04
clarkbif the api isn't too far gone it should do all the work for you16:04
fungiclarkb: i'm hesitant to use the restart playbook against a jenkins which is already struggling16:04
fungigiven the much higher incidence of it not working16:04
clarkbworst case you just have to back and manually cleanup16:04
clarkbwhich is true either way16:05
dguitarbiteAJaeger: fungi: Issue with dependency 'zip' for CI jobs:
*** yamamoto has quit IRC16:05
clarkbdguitarbite: I mentioned yesterday I think that you should use gzip16:05
dguitarbiteAJaeger: fungi AFAIK this should not happen since ZIP should be pre-installed with the image16:05
fungiyeah, actual worst case is it goes idle and the cron job restarts it tonight since that's the weekly schedule16:05
dguitarbiteclarkb: ok, my bad I missed your comment. I will update it. Thanks a lot.16:05
fungiyeah, "zip" (infozip/pkzip) is not really a common tool in unix circles. that's more of an msdos thing16:06
*** rossella_s has quit IRC16:07
AJaegerfungi: zip is listed in bindep-fallback16:07
clarkbif osic gets the new cert rolled out today my plan is to go ahead and make a mirror there and add it to nodepool. Will push up some wip changes for that nowish16:07
fungiAJaeger: yep, some of our jobs do use it16:07
anteayaclarkb: yay!16:08
fungiclarkb: pretty sure the playbook would have already failed on 06, i had to try a couple of times to hit the quietdown url successfully due to api errors16:08
clarkbfungi: fun16:08
*** annegentle has joined #openstack-infra16:08
clarkbwe can run the playbook continuously so that we are always one jenkins down :/16:08
openstackgerritMerged openstack-infra/project-config: Increase min-ready for ubuntu-trusty
fungibut it's in prepare for shutdown now, finally16:08
openstackgerritMerged openstack-infra/project-config: Use ubuntu-trusty for api-jobs
*** mikelk has quit IRC16:09
AJaegerfirst project-config jobs running on ubuntu-tursty16:09
fungipabelanger: good guess. we were at 50 disassociated floating ips, which i expect is our quota (as it is, nodepool is configured to boot no more than 39 instances there). i'm deleting them now16:10
clarkbit is odd that bluebox seems to suffer that far more than hpcloud did16:10
clarkbosic uses fips and will give us more data too16:11
pabelangerfungi: Based on what grafana was reporting, it looked like something related to floating IPs16:11
pabelangerwhich was cool grafana displayed it16:11
*** jsavak has quit IRC16:12
*** e0ne has quit IRC16:12
*** jsavak has joined #openstack-infra16:12
mnasersmall question:  regarding 80gb local disk for provided resources, where does that come from?16:12
tristanCwould be nice to get , it reduce the number of floating-ip leaks16:13
fungiclarkb: yes, would be awesome if someone had time to get to the bottom of it. i wonder if there's a particular boot or delete failure mode we're experiencing in bluebox that causes nodepool not to realize it needs to delete the associated fip16:13
mnaserwe're creating some new flavors which are more cpu power oriented, but low disk space.  something like 40gb local ssd + 8gb memory + 8vcpus, but that doesnt seem to "fit" the requirements listed, so im just curious16:13
fungimnaser: that's roughly how much space a grenade job needs to set up two copies of openstack at different versions and switch between them16:14
mnaserah i see16:14
openstackgerritDoug Hellmann proposed openstack/requirements: Bump python-neutronclient to 4.1.1
mordredfungi: otoh - if I can ever finish the shade-nodepool patch, the failure path we have now for fips goes away16:14
fungiwe tried running at smaller disk sizes in a recently added provider and then grenade was no longer able to work due to insufficient space16:15
mordredfungi: it might be replaced by a different failure path, of course16:15
mnaserfungi: i see, in that case we probably could do with a private/tenant-specific flavor16:15
*** thorst is now known as thorst_afk16:15
clarkbya its basically two sets of git repo checkouts + cinder volumes + swift glance things + VM images + swap16:15
fungimnaser: though we do have the ability to stitch in an ephemeral disk in devstack-gate to accommodate instances with a small rootfs but available ephemeral disk16:16
AJaegerfungi, clarkb, mordred: What's the best time for switching over all python jobs? So far it looks green. Do we want to do this asap to get notified early about issues? The zuul load is light enough that we could do it now16:16
AJaeger(I mean: switching to ubuntu-trusty)16:16
AJaegerIf we switch them over, I expect a manual JJB run...16:16
mnaserfungi: i mean cinder volumes are always available but i assume this is not a path that's usually taken16:17
fungiAJaeger: i don't mind not being around for it as long as there are others around who can babysit jjb and approve a revert (and babysit jjb again) if something goes terribly wrong16:17
AJaegerfungi: I agree - that's why I'm asking for help and opinions ;)16:17
mnaserfungi: shouldn't be too difficult... boot from volume with preserve flag set to 016:17
*** david_lyle__ is now known as david_lyle16:17
*** kmartin__ has quit IRC16:17
mnaserbesides finding the resources to do it and test it :p16:17
fungimnaser: right, nodepool lacks cinder support, though that's something we've discussed adding in the past16:17
*** nwkarsten has joined #openstack-infra16:18
fungior boot-from-volume support for that matter16:18
*** nwkarsten has quit IRC16:18
*** annegentle has quit IRC16:18
pabelangerfungi: cool, blue box launching nodes again16:18
odyssey4meAJaeger I think I see the bug - I'll patch it shortly.16:19
*** annegentle has joined #openstack-infra16:19
*** nwkarsten has joined #openstack-infra16:19
clarkbAJaeger: I mostly intend on being around today16:19
clarkbyesterday was an unexpected absence from compuer16:19
*** achanda has joined #openstack-infra16:19
clarkbAJaeger: so go for it and I can make sure jjb does its thing16:19
mnaserfungi: once we get the new flavors, i'll try to propose a change.  we'll be able to give more infra based on this setup and the way i have it in mind, it shouldn't be *too* complicated16:19
AJaegerclarkb: then let me approve fungi's patch. THANKS!16:20
mnaser+ using ceph means cow deploys, so WAAAAY faster!16:20
*** achanda has quit IRC16:20
mnaserthan downloading an image, etc.. but i'll do some checking and get back :)16:20
*** arxcruz has quit IRC16:21
lucasagomeshi all, if you have time can you please take a look at ? This patch is the final piece that enables the ipmi drivers in ironic to be tested in gate. It already has a +2 and the patch is very small (3LOC), thank you16:21
lucasagomesmordred, ^ mind taking a look if you have a sec later on today?16:21
*** x00350071_ has joined #openstack-infra16:22
*** x00350071 has quit IRC16:22
*** Qiming has quit IRC16:22
AJaegerlucasagomes: better mention that it's devstack-gate change ;)16:22
lucasagomesyeah it's devstack-gate change :-) sorry16:22
clarkblucasagomes: what sets DEVSTACK_GATE_IRONIC_DEPLOY_DRIVER?16:23
lucasagomesclarkb, it's set in project-config16:23
*** kzaitsev_mb has joined #openstack-infra16:23
lucasagomes1 sec will get the piece for you16:23
dimsum_anteaya : hound is still broke :(16:23
*** jamesmcarthur has joined #openstack-infra16:23
clarkblucasagomes: it should have a default in -wrap.sh16:23
anteayadimsum_: :(16:24
fungidimsum_: yep, someone needs to propose a patch to up its max-files or fix the open file leak if there is one16:24
*** jaosorior has quit IRC16:24
lucasagomesclarkb, all jobs do set the driver, it would be hard to come up with a default tho...16:24
clarkbagent_ssh :)16:25
anteayafungi: I'll see if I can find that setting in the go code16:25
lucasagomesyeah or pxe_ssh...16:25
lucasagomesclarkb, ok I can put any, but it's a bit bogus IMHO16:26
clarkblucasagomes: why is it bogus?16:26
clarkbwill ironic work with that default value? I think yes16:26
*** tongli has joined #openstack-infra16:26
clarkbso not bogus16:26
fungianteaya: it's probably going to be a matter of adding a ulimit call into the initscript to set a higher non-default value for max open files16:26
*** thorst_afk has quit IRC16:26
clarkb(I don't actually care what the default is just that there is one that is functional)16:27
*** andymaier_ has quit IRC16:27
lucasagomesclarkb, right... I just said bogus cause all jobs will actually set what driver it will test. But yeah, setting {pxe,agent}_ssh will work16:27
*** piet has joined #openstack-infra16:27
lucasagomesclarkb, ok updating the patch16:27
anteayaoh so something we would do on our end16:28
*** nwkarsten has quit IRC16:28
*** thorst_afk has joined #openstack-infra16:29
*** nwkarsten has joined #openstack-infra16:29
*** sputnik13 has joined #openstack-infra16:29
*** armax has quit IRC16:29
lucasagomesclarkb, it's already set actually16:29
*** armax has joined #openstack-infra16:29
*** kzaitsev_mb has quit IRC16:30
clarkbmordred: jeblair with bluebox the region is RegionOne but the cloud name is bluebox-sjc116:30
*** daemontool__ has joined #openstack-infra16:30
clarkbmordred: jeblair: should I not call osic region1 osic-region1?16:30
*** pblaho has quit IRC16:30
mordredclarkb: yah. I think that's a better name16:31
clarkblucasagomes: huh I failed at grepping16:31
jeblairclarkb: ++16:31
mordredoh - well,16:31
mordredhang on16:31
AJaegerclarkb: the python job just merged, hope our bot will notice as well - so let's wait for the JJB timeout ;)16:31
lucasagomesclarkb, happens16:31
jeblairclarkb: call it 'osic-something' :)16:31
mordredclarkb, jeblair: bluebox sjc is called sjc because there is the possibility of having more than one bluebox cloud16:31
mordredthere is only one osic cloud16:32
jeblairclarkb: that way if there's another osic we have room16:32
mordredso I think its cloud name should be "osic"16:32
mordredjeblair: there might be another osic?16:32
clarkbyes cloud name is osic and done16:32
jeblairmordred: well, osic is all about spinning up clouds...16:32
jeblairtheoretically there could be another one, right?16:32
clarkbjeblair: yes I think that is possible16:32
clarkbin fact this one is at cloud1.osic.org16:32
clarkbso maybe osic-cloud1?16:32
jeblairosic-cloud1 lgtm16:33
*** nwkarsten has quit IRC16:33
*** daemontool_ has quit IRC16:34
*** daemontool__ is now known as daemontool16:34
*** thorst_afk is now known as thorst16:35
max_loburHi Folks! Could someone please help me to get myself added to these groups ? They are new and we need to fill the up16:36
jrollmordred | there is only one osic cloud <- this may not always be true16:36
* jroll finishes reading scrollback and sees that was concluded anyway16:36
anteayamorning zaro16:36
mordredjroll: yah - I think "may not always be true" is the thing I have learned this monring16:36
*** ifarkas has quit IRC16:37
jeblairin it says "The next consumers slated to come on board is the OpenStack Foundation Infrastructure team."16:38
jeblairi appreciate the shout-out, but i wonder if anyone knows someone at rackspace where we could correct that by removing the word "foundation", at least in future communications?16:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert infra* jobs to ubuntu-trusty
jeblairi mean, we don't say "OpenStack Foundation Nova Team" :)16:39
jeblairsome of us don't work at the foundation and like to feel included16:40
*** esikachev has joined #openstack-infra16:40
*** mriedem_meeting is now known as mriedem16:40
*** roxanaghe has joined #openstack-infra16:40
*** jsavak has quit IRC16:40
anteayagood point16:40
*** vgridnev has joined #openstack-infra16:41
*** jaypipes has joined #openstack-infra16:41
*** jsavak has joined #openstack-infra16:41
*** jaypipes is now known as leakypipes16:42
clarkbmy comcast connection is very unhappy today16:44
clarkbapparently much of portland is affected?16:44
*** yamahata has quit IRC16:44
jeblairjroll, clarkb: when the second osic module comes online in oakland, we should see if we can get a cloud there :)16:44
openstackgerritClark Boylan proposed openstack-infra/system-config: Verify SSL with OSIC
jeblairi can wave hi to it16:45
clarkbI will WIP ^ until dns and cert are actually ready16:45
jrolljeblair: heh16:45
* mordred agrees with jeblair16:45
jrolljeblair: also, I will get that post fixed16:45
* mordred has so far never worked for the Foundation :)16:45
*** auggy is now known as frauggy16:45
*** yamahata has joined #openstack-infra16:45
jeblairjroll: aw, thanks!16:46
* mordred hands jroll a puppy16:46
fungijeblair: yeah, i expect they're confused because the initial communication about approving the openstack infra team's use of osic resources went through the openstack foundation executive director, and presumably people at intel/rackspace's marketing machine don't realize the community and the foundation are not all the same thing16:46
jrolljeblair: it's all on github, makes it easy :)
jeblairmordred: you just direct it :)16:46
yolandamax_lobur, done16:47
openstackgerritClark Boylan proposed openstack-infra/project-config: Use OSIC with nodepool
fungibattery is about to die, so don't expect more out of me until probably 2100-2200 utc16:47
clarkbreviews on both of those appreciated then I can just approve once it is ready16:47
*** bpokorny has joined #openstack-infra16:47
jeblairfungi: your battery or the asus? ;)16:48
fungiwe are one biomechanical organism16:48
*** _nadya_ has joined #openstack-infra16:48
jeblairthat's what the "fungi" laptop category meant16:48
fungiluckily it's just the battery which powers my interface to the internet16:48
fungithe battery powering my other important functions is fueled through different means16:49
*** apoorvad has joined #openstack-infra16:49
*** kencjohnston has joined #openstack-infra16:49
timrcIt's well known robots require beer to function properly.16:49
clarkbsource: Futurama16:50
nibalizerthe vexxhost mirror is now puppeting16:50
nibalizerso yea it was the ssh-key accept issue that was making ansible not work16:50
clarkbnibalizer: oh! did I not accept the ssh key?16:50
nibalizer still is not running16:50
jrolljeblair: fyi,
nibalizerclarkb: i accepted the key late last night, so i guess not16:51
nibalizeri'd love to see do better16:51
nibalizertbh im not sure needs to run puppet agent16:51
*** rcernin has quit IRC16:51
nibalizerlike couldn't it just create the server and ansible will get to it when it gets to it?16:51
jeblairnibalizer: yeah, mordred has some wip to do that i think?16:51
clarkbnibalizer: jeblair I would prefer it didn't do that16:52
clarkbbecause dns16:52
clarkbit is nice to be able to get everything up and running before you have to commit to it16:52
funginibalizer: though one of our most frequent discoveries of bootstrapping race issues in our puppet manifests is through seeing new servers fail to stand up completely because of puppet errors16:52
*** jcoufal has quit IRC16:53
fungiand also waiting for puppet to get around to applying on the node you just stood up means you context switch to working on something else for a bit and end-to-end it takes longer16:53
clarkbto clarify updating ssh known hosts is fine16:53
*** asettle has joined #openstack-infra16:53
clarkbbut not running puppet would make me sad16:53
mordredyah. I think running puppet is good16:54
jeblairoh sorry i hadn't read nibalizer's last msg when i wrote that16:54
mordredobvs running puppet agent is not16:54
jeblairi thought the conversation was about puppet agent/apply16:54
jeblairi agree we should run puppet something :)16:54
*** asettle has quit IRC16:54
anteayaclarkb: I think from channel conversation yesterday we agreed to skip 2 node on new providers until we test that 2 node works16:54
nibalizerour new subunit worker still isn't puppeting16:55
clarkbanteaya: yup I did16:55
*** kzaitsev_mb has joined #openstack-infra16:55
anteayaclarkb: I saw 2 node in your patch16:55
clarkbanteaya: it is only added to ubuntu-trusty 2 -node which isn't used anywhere but testing16:55
*** scheuran has quit IRC16:55
clarkbanteaya: whcih will allow us to test that it generally works before committing to it16:55
anteayaoh I'm sorry I didn't realize ubuntu-trusty 2 node is only for testing16:55
anteayaokay great, my mis-understanding16:55
clarkbanteaya: we don't use ubuntu-trusty for devstack things yet16:56
clarkbso that flavor is there to check that the image works with devstack but should also be good test of the overlay networking16:56
anteayaclarkb: we are about to, if I understand the way the wind is blowing today16:57
anteayaAJaeger: did I get that right?16:57
clarkbanteaya: I think its just for python-jobs16:57
anteayaoh is it?16:57
AJaegeranteaya: we move from bare-trusty to ubuntu-trusty16:57
anteayaI haven't been able to keep track16:57
AJaegeranteaya: nothing changes with devstack-trusty16:57
*** sc68cal has joined #openstack-infra16:57
nibalizer fatal: []: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply --deta16:57
nibalizeriled-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: -9", "rc": -9, "stderr": "", "stdout": "", "stdout_lines": []}16:57
AJaegeranteaya: at least for now16:57
anteayato be honest that doesn't aliviate my confusion16:57
anteayaaccurate though that maybe16:58
*** ihrachys has quit IRC16:58
anteayaif AJaeger and clarkb are in agreement I guess that will just have to be good enough for me16:58
clarkbanteaya: we have long had two sets of images per distro release. the "bare" and the "devstack"16:58
clarkbanteaya: bare ran all of the non devstack stuff typically nd devstack ran the devstack stuff16:58
clarkbwe are wrking to move everything to a single distro release image across the board but are starting with bare which is what is happening today16:59
anteayathank you16:59
clarkbat some point in the (near I hope) future we will move devstack16:59
anteayathat helps16:59
nibalizer would seem to be the errors16:59
*** skolekonov is now known as _skolekonov16:59
mtreinishnibalizer: hmm17:00
clarkbok time to check on jenkins jobbing17:00
nibalizeralso i guess we're not posting17:00
*** angdraug has joined #openstack-infra17:00
*** esikachev has quit IRC17:00
clarkbnibalizer: we don't post on fail iirc17:00
clarkbwhich sort of makes puppetboard :/17:01
mtreinishnibalizer: I think that's a bug in puppet-subunit2sql, it should probably be root17:01
AJaegerIs jenkins06 restarted?17:01
mtreinishnibalizer: also I'm not sure why that dir is named logstash, that seems kinda confusing :)17:02
max_loburyolanda: thanks!17:02
*** kmartin has joined #openstack-infra17:02
*** ashleighfarnham has joined #openstack-infra17:02
clarkbAJaeger: jjb is updating on 01 which should pick up the node change, I will watch it and manualy restart if it times out17:02
nibalizerclarkb: we should be posting on everything17:02
*** matrohon has quit IRC17:02
*** jsavak has quit IRC17:02
AJaegerclarkb: thanks for the update17:02
*** sdake_ has joined #openstack-infra17:02
nibalizerlike even if puppet catalog fails, ansible can post that happening to puppetboard17:03
*** Jeffrey4l has quit IRC17:03
nibalizerbut in this case it should generate a catalog with some fails then send em in17:03
*** cfarquhar has quit IRC17:03
clarkbnibalizer: it should, but I don't think it does17:03
*** sdake has quit IRC17:03
*** amotoki has joined #openstack-infra17:03
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Change /etc/logstash dir ownership to root
mtreinishnibalizer: ^^^17:03
anteayaAJaeger: yes jenkins06 looks odd in grafana17:05
anteayaAJaeger: fungi said he was powering it off, but I lost what state it was in when he ran out of battery17:05
*** yamamoto has joined #openstack-infra17:05
clarkbhe said he got it in shutdown mode17:06
*** annegentle has quit IRC17:06
anteayaah okay17:06
anteayaso ready to be powered off but not yet powered off17:06
*** annegentle has joined #openstack-infra17:06
clarkbyou have to wait for any running jobs to complete first17:07
clarkbbut no new jobs will be scheduled there17:07
*** jed56 has joined #openstack-infra17:07
*** sdake_ is now known as sdake17:07
anteayaokay thanks17:07
*** jlanoux has quit IRC17:07
anteayacurrently it has 8 nodes in use17:08
*** kiran-r has quit IRC17:08
*** bastelfreak has quit IRC17:08
AJaegeranother change to ubuntu-trusty is - project-config cores and pabelanger , what do you think?17:09
*** bastelfreak has joined #openstack-infra17:09
*** vincentll has quit IRC17:09
*** hdd has joined #openstack-infra17:10
*** Sukhdev has quit IRC17:10
clarkbAJaeger: +217:10
AJaegerthanks, clarkb17:10
*** nwkarsten has joined #openstack-infra17:11
*** annegentle has quit IRC17:11
clarkband jjb is still running on 01 and 07 happy so far17:11
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Convert specs-jobs to ubuntu-trusty
anteayaAJaeger: I'm not sure where we are in the workflow so you can self approve when you are ready17:14
*** dtardivel has joined #openstack-infra17:14
*** yamamoto has quit IRC17:15
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Display a modal when a card is clicked on
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal to confirm archiving cards
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add Due Dates to boards
*** degorenko is now known as _degorenko|afk17:17
*** bpokorny has quit IRC17:17
*** sridhar_ram has joined #openstack-infra17:17
clarkbAJaeger: jjb is done on 01 and 07 now, I am rerunning on 01 really quickly to see if it needs to be redone17:18
AJaegerclarkb: wow!17:18
*** bpokorny has joined #openstack-infra17:18
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Training-labs: Rework jobs
AJaegeranteaya: we can merge anytime unless it causes a problem for clarkb with jenkins...17:19
* AJaeger will be back in an hour or so17:19
clarkbgo for it17:19
*** bpokorny has quit IRC17:19
clarkbI think jjb updated just fine, but doing my best to confirm17:19
pabelangerAJaeger: LGTM17:21
dimsum_fungi : anteaya : looks like there's a fix!
openstackgerritPaul Belanger proposed openstack-infra/project-config: Convert windmill jobs to bindep
*** nwkarsten has quit IRC17:23
clarkbI just checked gate-zaqar-python27 and it is not updated so I will be starting some manual updates now17:24
nibalizermordred: so in ansible-puppet it kinda looks like if we fail puppet, we don't even try to post logs17:24
clarkbunless we are getting that other change in I can wait for it to merge17:24
nibalizeri can't see any obvious reason for this unless its ansibles way to stop a playbook after something fails17:24
openstackgerritKhai Do proposed openstack-infra/git-review: my Fix one typo on git-review documentation
clarkbnibalizer: yes that is why17:24
nibalizerwell... is there a way around that?17:25
*** ashleighfarnham has quit IRC17:25
clarkblets hold off on approving 288596 so I can restart jjb and get these other jobs updated17:25
clarkbnibalizer: I think you can set a thing that says fail is ok17:25
clarkbI did it in the jenkins restart playbook iirc17:25
anteayadimsum_: well the bug was filed17:25
clarkb(for jenkins stop)17:25
anteayadimsum_: the fix seems to be The right answer here, I think, is to actually throttle indexing to ensure that no more than N searchers are reindexing at a time.17:25
anteayaI'm not sure how well throttling indexing will work for us17:26
dimsum_anteaya : scroll down to the 2 merged PR's17:26
bkeronibalizer: ignore_errors: yes as a metaparam to resources17:26
jamespageif anyone has a moment I could do with a second review of ontop of yolandas ...17:26
nibalizerit looks like we can ignore_errors: True17:26
*** bpokorny has joined #openstack-infra17:27
nibalizerbut that wuold mean we wouldn't see in the ansible logs when puppet failed17:27
*** Sukhdev has joined #openstack-infra17:27
*** gnuoy_ has joined #openstack-infra17:27
odyssey4meAJaeger how does the ZUUL_REFNAME get constructed? If I have a review, how do I determine the ZUUL_REFNAME - I'm trying to follow how the conditionals in flow17:28
anteayadimsum_: okay so the fix appears to have merged a year ago17:29
clarkbodyssey4me: the only way to know what it is is to get it from zuul, its not deterministic and is a random uuid iirc17:29
*** gnuoy_ has quit IRC17:31
*** dtantsur is now known as dtantsur|afk17:32
odyssey4meclarkb blast :/17:33
*** harlowja_at_home has joined #openstack-infra17:33
*** annegentle has joined #openstack-infra17:33
*** yamahata has quit IRC17:33
odyssey4meclarkb under what conditions would ZUUL_REFNAME have 'refs/tags/' in it? is the ZUUL_REFNAME the same as build_ref in logstash?17:34
*** _nadya_ has quit IRC17:34
clarkboh wait its REFNAME you were asking for not REF17:35
clarkbZUUL_REF is not deterministic but REFNAME may be17:35
clarkbclearly I need more caffeine17:35
clarkbodyssey4me: the REFNAME should be the branch name (master, stable/kilo, etc) wen in check or gate and I think it is that in post as well17:36
clarkbin release it is the tag that triggered the build17:36
odyssey4meclarkb so if I push a tag to a branch, does it trigger a docs build?17:37
*** daemontool_ has joined #openstack-infra17:37
clarkbodyssey4me: if that project has a docs job in the release pipeline17:37
* odyssey4me hands clarkb some coffee17:37
*** bpokorny has quit IRC17:37
*** austin81 has joined #openstack-infra17:37
clarkbfor most projects I think this is the case17:37
odyssey4meclarkb ok, by release pipeline do you mean the release jobs managed by the release team - or a specific review-based process?17:38
odyssey4meOpenStack-Ansible is not using that process at this point - we use the 'git tag, git push' mechanism17:39
clarkbodyssey4me: in zuul you configure explicitly what you want to happen on different events. check runs on new patchset, gate on approval, post on merge, release on tag pushed17:39
*** esikachev has joined #openstack-infra17:39
openstackgerritMerged openstack/requirements: Bump openstacksdk version to 0.8.1
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Training-labs: Use ubuntu-trusty
*** kzaitsev_mb has quit IRC17:40
odyssey4meclarkb ok, we're using the template jobs for docs so I suspect we're inheriting the release docs job when a tag is pushed17:40
*** Guest63919 has quit IRC17:41
*** daemontool has quit IRC17:41
*** ccrouch1 has joined #openstack-infra17:41
odyssey4meclarkb perhaps you can confirm?
*** ccrouch has quit IRC17:43
odyssey4meit would seem so:
*** bpokorny has joined #openstack-infra17:44
clarkbodyssey4me: yes that17:44
*** fabio_ has quit IRC17:44
odyssey4meok, so it would seem then that when I tagged earlier today, we hit this conditional:
*** Daisy has joined #openstack-infra17:45
*** bpokorny has quit IRC17:45
odyssey4meand this would have been hit, as the job is a tags only job for a tag release:
*** bpokorny has joined #openstack-infra17:45
odyssey4meclarkb to clue you in, I'm troubleshooting an issue where the tag release replaced the published docs for the master branch with the docs from the tagged build17:46
clarkbthat script is set -x so your job log should confirm17:46
*** jistr has quit IRC17:46
clarkbI am going to guess it incorrectly determined the latest tag?17:46
odyssey4meie which should contain ended up with the content destined for
odyssey4meok, where do I find that job log - that would be most helpful!17:47
openstackgerritMerged openstack-infra/project-config: Convert infra* jobs to ubuntu-trusty
*** Thelo has quit IRC17:49
clarkbit looks like 12.0.7 is the latest tag17:49
clarkbso the script did what it is supposed to17:49
odyssey4meclarkb so therefore
*** Daisy has quit IRC17:50
odyssey4meyeah, it did publish the tag - but also overwrote the master content17:50
*** Thelo has joined #openstack-infra17:50
*** ccrouch1 has quit IRC17:50
clarkbyes that is what it is supposed to do, see the comment above that last line you linked17:50
*** ccrouch has joined #openstack-infra17:50
*** esikachev has quit IRC17:51
odyssey4meah, this is a bad assumption - or we're making a bad assumption17:51
*** kencjohnston has quit IRC17:52
clarkbthe intent is that you tag as you go so the biggest number tag is always recent and should go to /17:52
*** ccrouch has quit IRC17:52
*** salv-orlando has joined #openstack-infra17:52
odyssey4meyeah, whereas we want the master content in / and are happy to have tagged content in the respective sub-URL17:52
*** jpr has quit IRC17:53
*** yamamoto has joined #openstack-infra17:53
odyssey4meI'm not quite sure what the use-case would be for not having the current HEAD in / - thoughts?17:53
*** lucasagomes is now known as lucas-beer17:54
*** lezbar has quit IRC17:54
clarkbodyssey4me: its for projects that don't publish current head17:54
clarkblike libraries that are intended to be consumed via releaess17:54
odyssey4mehmm, fair enough17:55
*** salv-orl_ has quit IRC17:55
*** gyee has joined #openstack-infra17:55
*** ccrouch has joined #openstack-infra17:55
odyssey4meso now the trick is to figure out how to do this a little differently - to cater for both use cases17:55
clarkbI think its already an option you just have to change your job?17:55
odyssey4mehappy to do that - just need to figure out how17:56
*** leakypipes has quit IRC17:56
odyssey4meI guess I just need to disable the 'tags-only' bit17:56
clarkbodyssey4me: the trivial option is replace that job template with just a post: '{name}-docs entry then all updates to all branches will get publishes as you go17:57
odyssey4meso do I set a release job to override ?17:57
clarkbodyssey4me: instead of /12.0.7 you will have /stable/foo17:57
*** dimtruck is now known as zz_dimtruck17:58
odyssey4meso I'm suggesting that I just add release: '{name}-docs' to
odyssey4mewill that do the trick I wonder?17:58
odyssey4mehmm, I think we're both saying the same thing?17:59
clarkbya I think so17:59
*** Swami_ has joined #openstack-infra17:59
*** Swami has joined #openstack-infra17:59
mordredclarkb: any chance you can +A this ?18:01
anteayadimsum_: I'm trying to bring up a vm with puppet-hound on it18:01
clarkbmordred: I am still trying t ofigure out why we need the special case for unittesting18:01
clarkbmordred: won't Munch(obj) return a thing that is useable?18:01
anteayadimsum_: I don't have a lot of experience working with puppet modules, so have low expectations18:01
*** ashleighfarnham has joined #openstack-infra18:02
anteayadimsum_: I'm trying to see if we have that patch and how to set it as the readme suggests18:02
*** geekinutah has joined #openstack-infra18:02
*** esker has quit IRC18:02
mordredclarkb: I fix that in the two follow up patches18:02
*** electrofelix has quit IRC18:03
mordredclarkb: I left them as follow up patches becuse I didn't wnat this one to be too big18:03
clarkbmordred: I think you just remove that code and you are done?18:03
clarkbits less code not more18:03
mordredclarkb: and
mordredclarkb: no - it's there because we had some places in the test code where we were mocking glance objects wrong18:04
mordredso they were traversing that function incorrectly18:04
mordredthe unittest special case was a get-out-of-jail care to skip having to fix the unit tests18:04
mordredbut then I fixed the unit tests18:04
*** yamahata has joined #openstack-infra18:04
clarkbok I am going to review them in opposite order :)18:08
*** jpr has joined #openstack-infra18:08
*** derekh has quit IRC18:11
*** notnownikki has quit IRC18:12
*** Keedya_4 has joined #openstack-infra18:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Make OpenStack-Ansible publish master docs
odyssey4meclarkb ^ does that do what I think it does?18:14
clarkbodyssey4me: close, let me comment18:14
*** yamamoto has quit IRC18:14
*** yamamoto has joined #openstack-infra18:15
clarkbodyssey4me: posted18:15
clarkbmordred: stack approved18:16
*** Keedya_4 has quit IRC18:16
*** zz_dimtruck is now known as dimtruck18:17
*** harlowja_at_home is now known as hollaja18:17
mordredclarkb: thanks18:17
mordredclarkb: once those land, we'll cut a new shade18:17
*** sarob has joined #openstack-infra18:19
*** piet has quit IRC18:19
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Enable undercloud ssl on nonha job
sdaguehmmm... vexxhost is failing single node grenade a lot18:21
*** tongli has quit IRC18:21
*** piet has joined #openstack-infra18:21
sdagueclarkb: logstash seems to be missing a bunch of data and the queue is this crazy roller coaster, any idea what'18:23
sdagues going up?18:23
*** nwkarsten has joined #openstack-infra18:24
clarkbwhat data is missing?18:25
*** yamamoto has quit IRC18:25
*** jordanP has quit IRC18:25
sdagueit looks like part of the issue is it's 3+ hours behind again18:26
clarkbroller coaster is often a side effect of trying to index very large files18:26
sdagueI know mriedem wasn't able to find some fail runs18:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Make OpenStack-Ansible publish master docs
sdagueok, well do we have any idea what those are?18:26
clarkbthey get into queue and back it up then once past everything jumps ahead again18:27
odyssey4methanks clarkb - feedback incorporated18:27
clarkbsdague: you can query for it18:27
sdagueclarkb: how?18:27
clarkbsdague: by grouping on job uuid and file name and counting results18:27
sdagueok, I need to write a custom tool?18:28
*** amotoki has quit IRC18:28
clarkbnot necessarily, I think the api will give you that18:28
odyssey4meoh, that's weird - since earlier today I'm seeing blank console logs - for example:
sdaguebut a custom tool against that API18:28
odyssey4mepabelanger ^18:28
*** nwkarsten has quit IRC18:28
clarkbI think its one request piped to soet18:28
AJaegerclarkb: reading backscroll, I'm confused about the docs publishing.18:28
*** kencjohnston has joined #openstack-infra18:30
sdaguethat does assume that every line is the same cost. Is there a way to log it on the server so we can know real cost. Like the ansible runs that put everything in one line?18:30
clarkbsdague: we can look at time stamps in the server logs, but the way logstash works is num lines is basically cost18:31
AJaegerclarkb: should have fixed the publishing...18:31
clarkbsince each line is processed individually18:31
pabelangerodyssey4me: AJaeger: I wonder if that is related to curl18:31
clarkbAJaeger: I thibk it did except that openstack-ansible doesnt tag master18:31
sdagueclarkb: ok, well it's beyond scope for what I can bite off atm.18:33
AJaegerclarkb: let me check the log file, this confuses me...18:33
*** Sukhdev has quit IRC18:33
*** zeih_ has joined #openstack-infra18:33
*** ccrouch has quit IRC18:33
AJaegerpabelanger: you mean we do not have curl on the ubuntu-trusty?18:33
pabelangerAJaeger: Ya, I noted that yesterday18:34
clarkbshould work with ehat we allow through the api18:34
clarkbsince it is under the search api18:34
AJaegerpabelanger: but it's in jenkins/data/bindep-fallback.txt, isn't it?18:35
AJaegerSo, why is it not installed?18:35
pabelangerAJaeger: So, the issue is with other-requirements.txt18:35
*** eharney has quit IRC18:35
AJaegerpabelanger: so, odyssey4me needs to add it, doesn't he?18:35
pabelangerAJaeger: So, in this case is is JJB that needs curl, not ansible18:35
clarkboh if we use it un the job then we should ship it18:35
pabelangerso, we need to figure out a good way to add JJB dependencies otherwise each project will need to add them into other-requirements.txt18:36
* AJaeger cannot think straight right now, have patience with me18:36
clarkbjust install at job build18:36
clarkber imsge build18:36
pabelangerclarkb: Yup, just finding a place to install it18:37
*** bpokorny has quit IRC18:37
anteayaAJaeger: wow, if you have a moment of confusion18:37
anteayathe rest of us poor mortals don't have a chance18:37
*** zeih_ has quit IRC18:38
*** treaki has joined #openstack-infra18:38
*** piet has quit IRC18:38
AJaegeranteaya: two problems here plus fun with my kids - and the end of the week...18:38
*** vgridnev has quit IRC18:39
anteayaI hear that18:39
openstackgerritPaul Belanger proposed openstack-infra/system-config: Ensure curl is installed for jenkins slaves
pabelangerodyssey4me: AJaeger: clarkb ^18:40
*** sbalukoff has quit IRC18:40
*** ccrouch has joined #openstack-infra18:40
AJaegerclarkb: regarding docs publishing, I wonder whether we mixed the cases.18:41
*** pcaruana has quit IRC18:41
AJaegerclarkb: For server projects: We publish after each commit and also each tag - but tags should go only to a subdir18:41
openstackgerritPaul Belanger proposed openstack-infra/system-config: Ensure curl is installed for jenkins slaves
*** piet has joined #openstack-infra18:41
clarkbAJaeger: oh if that is the intent then I guess we can fix it directly in the script18:41
clarkbby not having any of the special sauce18:42
openstackgerritMerged openstack-infra/shade: Fixes for latest cinder and neutron clients
AJaegerclarkb, odyssey4me: For client projects: We publish only with releases/tags - and should publish to the main dir in general and a sub dir18:42
clarkboh right18:42
AJaegerclarkb: see my last change here
*** bpokorny has joined #openstack-infra18:42
clarkbso maybe what we need to do is update the template dto do what odyssey4me's change does18:42
clarkbfor not clients18:42
mriedemsdague: clarkb: the cinder console logs were pretty massive for unit test runs until this week, but not sure if that would tip things over18:43
AJaegerI think we need to move the tags job from server to client in
*** jamesmcarthur has quit IRC18:43
clarkbAJaeger: yes I agree18:43
clarkbodyssey4me: this may be a larger issue that we fix for you :)18:43
clarkbmriedem: it really can18:43
clarkbmriedem: basically what seems to happen is expensive files come in and back up because they don't flush quickly18:44
*** achanda has joined #openstack-infra18:44
clarkbthen once they flush everything is quick until the next high cost files back it up again18:44
AJaegerodyssey4me, clarkb: who of us will fix this?18:44
*** ccrouch has quit IRC18:45
clarkbAJaeger: I am trying to make sure jjb updates happen so probably best if it isn't me18:45
*** cznewt has quit IRC18:45
clarkbpabelanger: I +2'd and can approve if no one else reviews that soon18:45
pabelangerclarkb: sure, just waiting for jenkins to +1 the syntax18:46
clarkbmriedem: sdague es says cluster is red so now trying to figure out why18:47
*** mriedem has quit IRC18:47
clarkbwhich is also likely not helping18:47
geekinutahfungi, anteaya: for context, the Mike Wilson AndyU mentions is me18:47
anteayageekinutah: ah wonderful, thank you for context18:47
anteayageekinutah: so I think that from yesterday's chat if we could start with one node we can run tests and give you feedback18:48
anteayaand then you would have something more to go on18:48
geekinutahyeah, I am working on that18:48
anteayawhen you have a node18:48
anteayacool, thank you18:48
geekinutahbut I seriously doubt there will be any problems18:48
*** mriedem has joined #openstack-infra18:48
clarkbmriedem: sdague  curl -XGET '' if yo uwant to check that yourselves18:48
odyssey4meAJaeger clarkb if it's simply switching the templates around, I'm happy to submit the patch18:48
pleia2argh, devstack18:48
anteayaI afirm you doubt there will be problems18:48
clarkbmriedem: sdague maybe we can add that to the e-r banner?18:48
AJaegerodyssey4me: go for it ;)18:48
mriedemclarkb: i'll run it18:49
anteayageekinutah: for context we have never had a cloud that is problem free18:49
anteayageekinutah: but who knows yours might be the first18:49
clarkbodyssey4me: I think you need to update th eexisting template to use {name}-docs instead of the tags only one18:49
anteayapleia2: hey there, feeling better today?18:49
clarkbodyssey4me: and update the other template for clients to use tags only18:49
pleia2anteaya: hanging in there18:49
geekinutahanteaya: of course, what I mean is OpenStack problem free18:49
anteayapleia2: glad to hear it18:49
geekinutahmeaning it will work with some tinkering :-)18:49
anteayageekinutah: ah18:49
pleia2I'm going to need some help with at some point (this is for the i18n-checksite spec)18:50
anteayayour expectations are coming into convergence with my experience18:50
anteayain so much as watching others work counts as experience18:50
*** doug-fis_ is now known as doug-fish18:50
pleia2running through this puppet module again in testing, but we have some infra-side problems to solve that I'm not sure about18:50
pleia2like the fact that our nodes don't know their IP address, and devstack wants to know its ip address18:51
openstackgerritMerged openstack-infra/shade: Use warlock in the glance v2 tests
pleia2s/IP/public IP18:51
nibalizercan another core look at please?18:51
armaxAJaeger: ping18:51
pleia2hm, actually, the rackspace ones do18:51
*** hollaja has quit IRC18:52
anteayapleia2: I am no help to you unfortunately18:52
clarkbnibalizer: mtreinish approved but see comment18:52
*** cznewt has joined #openstack-infra18:52
*** harlowja_at_home has joined #openstack-infra18:52
anteayaI'm trying to figure out how to get the puppet-hound module to run18:52
nibalizerclarkb: ya18:53
anteayaso far puppet apply ../init.pp aren't the magic words18:53
clarkbpleia2: "infra-side problems" is code for "cloud design problems"? :)18:53
mriedemclarkb: so that says status is red but i'm not sure how to interpret why18:53
clarkbmriedem: I think it is because of the 3 unassigned shards18:53
pleia2anteaya: you probably want to create a local.pp that has info from our codesearch.pp in system-config18:53
clarkbmriedem: but my local networking si very poor today and making it hard to follow up on that18:53
pleia2anteaya: then follow instructions here:
anteayapleia2: oh, I can try that18:54
anteayaoh look, documentation18:54
anteayapleia2: thank you18:54
pleia2clarkb: heh, yes, "this puppet module works fine in testing" :)18:54
AJaegerwhat's up armax ?18:54
clarkbmriedem: it looks like elasticsearch07 might have gone away and is now recovering those shards18:54
clarkbthey belong to two indexes18:54
clarkbmriedem: you can get all this in really ugly json fro mthe api :/18:55
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Set tags-only docs publishing for client jobs, not server jobs
armaxAJaeger: I wonder if it’s safe to pull the trigger on the chain of patches pertaining the constraints suffix drop on neutron jobs18:55
armaxhow long are you gonna be around?18:56
odyssey4meclarkb AJaeger please see if is right?18:56
jesusaurclarkb: you may already know this, but i recently learned that you can do
AJaegerarmax: another hour - let me review them in a few minutes18:56
armaxAJaeger: ok18:56
clarkbjesusaur: I didn not know that18:57
clarkblooks like the last time es07 logged anything was february 19th18:57
clarkbI may try bouncing it and seeing if that unsticks things18:58
AJaegercould anothoer project config core help getting rid of constraints, please? and - I'll review the stack as well after looking at odyssey4me's change18:58
clarkb(this is a really interesting fail mode becausees07 is primary for those shards but it is derping so there is no replica for it either)18:58
*** ccrouch has joined #openstack-infra18:58
anteayaAJaeger: those patches are both waiting on your neutron patch to merge19:00
clarkbwow and of course when you restart an unhappy server all of the indexes it knew about that have since been deleted show back up again19:01
clarkbthis is unfun but I will delete those old thing sthat have been deleted once already19:02
anteayathat is interesting19:02
*** nwkarsten has joined #openstack-infra19:02
jesusaurclarkb: yeah, a manual run of the curator cron will take care of that though19:02
clarkbjesusaur: yup just did that19:03
openstackgerritMerged openstack-infra/puppet-subunit2sql: Change /etc/logstash dir ownership to root
clarkbits just an interesting way to rejoin a cluster19:03
clarkb"hey master you don't know anything about these indexes but I do therefore you are going to learn about them as fast as I can teach you19:03
clarkbrather than "hey master node what should I care about"19:03
*** annegent_ has joined #openstack-infra19:04
*** yaume has quit IRC19:04
AJaegeranteaya: my original neutron change, it has been changed significantly...19:04
odyssey4mepabelanger AJaeger so the workaround at this stage for the blank console file (due to 'curl' not being present on the d-i-b image) is to include 'curl' as a package in 'other-requirements.txt'?19:04
anteayaI'm sorry I never have gotten my head around constraints19:04
odyssey4mecan I submit a patch somewhere to ensure that curl is included in the built image?19:05
anteayaso I can look at your patches but I have a hard time understanding what is happening19:05
anteayasorry for being the slow person in the room19:05
pabelangerodyssey4me: yes19:05
clarkbmriedem: sdague cluster is green now, though it is rebalancing19:05
clarkblets see if this gets it happier19:06
jesusaurclarkb: yeah, but that same mode is what lets us do a full cluster restart without losing data19:06
clarkbjesusaur: in theory the "minimum data nodes" param should pretect against that19:07
*** annegentle has quit IRC19:07
*** ddieterly has joined #openstack-infra19:07
clarkbjesusaur: I have it set to n-1 bceause we have n+1 redundancy19:07
mriedemclarkb: cool19:08
clarkbI don't see anything in dmesg or the es log that would indicate what went wrong19:08
clarkbI am guessing some es process internal state went wonky19:09
*** isq has joined #openstack-infra19:09
*** isq has quit IRC19:09
jesusaurdisks aren't full or anything like that?19:09
clarkbjesusaur: nope I checked that, had 80% free on the 1TB volume19:09
jesusaurit looks like the mounted volume isnt in cacti19:09
*** isq has joined #openstack-infra19:09
AJaegerarmax: is ihrachsys still around? If not, I'll fix his change...19:09
*** ihrachys has joined #openstack-infra19:10
armaxI think he’s out19:10
clarkbjesusaur: huh it is mounted19:10
armaxAJaeger: even though he’s still lurking around19:10
armaxihrachys: ^19:10
clarkbpabelanger: AJaeger odyssey4me do we need to revert the openstack-ansible jobs use ubuntu-trusty change due to the lack of curl?19:12
clarkbthen merge that once we have updated image swith curl?19:12
pabelangerclarkb: I'll let odyssey4me make the call for ansible-jobs19:12
clarkbpabelanger: ok19:12
AJaegerclarkb: at which point is puppet applied for these images?19:13
AJaegerbuild time - or run time?19:13
*** noorul has joined #openstack-infra19:13
*** esikachev has joined #openstack-infra19:13
mtreinishclarkb: sure, I can push a change to change /etc/logstash to /etc/subunit2sql or something19:13
clarkbnibalizer: mordred pleia2 other infra-root needs to get in for curl on images19:13
*** ociuhandu has quit IRC19:13
clarkbAJaeger: pabelanger yup build, so my plan is to manually rebuild that image once ^ merges19:13
pabelangerokay, so my vote is land the patch and rebuild19:13
*** eharney has joined #openstack-infra19:13
*** pvaneck has joined #openstack-infra19:14
*** kencjohnston has quit IRC19:14
odyssey4mepabelanger no, we'll get curl added in to other-requirements.txt for now cc clarkb AJaeger19:14
AJaegerarmax, ihrachys : updated - the neutron change looks fine to me now.19:14
armaxAJaeger: ok, let’s get this deferred until first thing monday morning19:14
clarkbodyssey4me: ok, you should be able to pull that out by monday (I definitely want the fixes to be in place before the weekend)19:15
odyssey4meI do think that seeing as Jenkins requires curl to be there, it should be in the image - if you can point me to where the base package list is defined, I'll patch it up19:15
AJaegerarmax: sure19:15
clarkbodyssey4me: the patch is written, just need to get it merged and rebuild the image19:15
odyssey4meclarkb do you have a review link - I'd like to reference it19:15
*** yamamoto has joined #openstack-infra19:16
clarkbpabelanger: you should also remove the curl packages from thick_slave.pp in a follow up19:16
AJaegerarmax: 286777 has two +2s and a +A - once the neutron change is in, we need to toggle the +A to merge - should be quickly done...19:16
clarkbpabelanger: so that they bare-* builds don't break while we transition19:16
armaxAJaeger: ack19:16
*** noorul has quit IRC19:17
mordredShrews: is this similar to the races you saw recently?
*** hashar has joined #openstack-infra19:17
pabelangerclarkb: Oh, missed that19:17
pabelangerchecking now19:17
*** dtardivel has quit IRC19:18
*** rcarrillocruz has left #openstack-infra19:18
*** ccrouch1 has joined #openstack-infra19:18
Shrewsmordred: nope. that just looks like a timeout19:18
ihrachysAJaeger: thanks for respin. I will also kill venv-constraints target completely in that patch, it is probably already orphan and not used anywhere anyway.19:19
*** rcarrillocruz has joined #openstack-infra19:19
*** _nadya_ has joined #openstack-infra19:20
*** ccrouch has quit IRC19:20
AJaegerihrachys: kill it in the cleanup patch once the infra changes are in...19:20
*** yamamoto has quit IRC19:21
ihrachysAJaeger: ack, let's leave it that way19:21
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use non-constraint jobs for all neutron branches
AJaegerihrachys: I just updated this project-config change ^19:21
AJaegerihrachys: should be fine now - I hope ;)19:21
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
AJaegerihrachys: thanks for splitting these up and reworking!19:21
*** apoorvad has quit IRC19:22
ihrachysAJaeger: yeah, I still don't trust myself, thanks for checking all that. we'll need to be avail when it lands master ;)19:22
ihrachysin case something goes on fire19:22
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove curl from thick_slave
AJaegerihrachys: I'm off most off Monday but others should be around19:23
*** jed56 has quit IRC19:23
*** austin81 has left #openstack-infra19:23
ihrachysAJaeger: ok, I guess I will find someone from infra side to look at 'oh my, neutron gate is totally broken' report19:23
*** gyee has quit IRC19:23
pabelangerclarkb: Ah, you are right.  I have to squash both patches19:24
clarkbpabelanger: no just do two chagnes19:24
clarkbpabelanger: it will be fine because we don't build the images constantly19:25
pabelangerclarkb: 288634 will fail the gate19:25
pabelangersee zuul.o.o19:25
clarkboh gotcha19:25
*** apoorvad has joined #openstack-infra19:26
openstackgerritPaul Belanger proposed openstack-infra/system-config: Move curl to jenkins_common.pp
*** _nadya_ has quit IRC19:28
pabelangerokay, that should be better ^19:28
pabelangerwe'll likely need some sort of JJB-requirements.txt file for bindep moving forward19:29
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove experimental-bindep-jobs from repos
pabelangerbut that requires some discussion with fungi19:29
*** frauggy is now known as auggy19:30
clarkbI think we just keep baking those things into the image19:30
*** gyee has joined #openstack-infra19:31
SamYaplejust checking in, have we gotten mirror signing for packages done yet? if not is there a bp/ongoing work to do that?19:31
*** ociuhandu has joined #openstack-infra19:31
mordredno, I think it's on the backburner19:31
*** sdague has quit IRC19:32
openstackgerritMerged openstack/requirements: Add Freezer modules and projects for Mitaka release
*** kzaitsev_mb has joined #openstack-infra19:32
*** tqtran-afk has joined #openstack-infra19:34
*** piet has quit IRC19:35
*** sdake has quit IRC19:36
clarkbmordred: did the change to allow multiple reprepros merge in puppet or do we still need to rewrite it as a defined type? for ceph testing we need to be adding the ceph repo19:36
*** piet has joined #openstack-infra19:36
openstackgerritKaitlin Farr proposed openstack/requirements: Bump Castellan to 0.4.0
mtreinishnibalizer: are we manually running puppet on subunit-worker02.o.o? I only see the 1 run in puppetboard19:36
*** jcoufal has joined #openstack-infra19:36
mtreinishoh, I probably should have refreshed, now there are 219:37
*** sdake has joined #openstack-infra19:37
*** ihrachys has quit IRC19:38
*** jpr has quit IRC19:38
AJaegerdougwig: regarding - all of this is removed in a follow-on change, so I would not go for style. Or do you see a real bug there?19:39
dougwigAJaeger: you're right, i can live with it in that case.19:40
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Need a new minimum for oslo.cache in Mitaka
AJaegerdougwig: 288506 is the followup19:41
nibalizermtreinish: yea unfortunately when we fail puppet we don't tell puppetboard about it19:41
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Need a new minimum for oslo.cache in Mitaka
*** dizquierdo has quit IRC19:47
*** kzaitsev_mb has quit IRC19:47
*** _nadya_ has joined #openstack-infra19:47
*** kzaitsev_mb has joined #openstack-infra19:47
anteayajenkins06 doesn't have any jobs running on it anymore19:48
*** Sukhdev has joined #openstack-infra19:49
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Switch /etc/logstash to /etc/subunit2sql
*** annegent_ has quit IRC19:49
mtreinishclarkb: ^^^19:49
mtreinishnibalizer: really, ok :(19:49
*** sbalukoff has joined #openstack-infra19:50
*** alivigni|gone is now known as ari19:50
*** ari is now known as ari-19:51
nibalizermtreinish: ya so i guess ansible runs with -e basically, so if it fails puppet it doesn't do the submit task19:51
clarkbI am tethering on my phone now because networking is so bad locally, going to see if I can make it better19:51
nibalizerso we can mark the puppet task to pass even if it fails, but thats not really what we want19:51
nibalizerwhat we want is some kind of a 'finally' task19:51
nibalizerthat runs no matter what happened19:51
nibalizerso it look like we can do something clever where we report what happened into a variable, pass the puppet run, then do some stuff, then inspect the puppet run for failure and fail if we want to19:52
odyssey4mepabelanger clarkb thanks for keeping the review to as I'm using it as a reference point for follow-up19:52
*** fawadkhaliq has joined #openstack-infra19:52
*** tqtran-afk is now known as tqtran19:54
*** annegentle has joined #openstack-infra19:54
bkeroclarkb: is supposed to be working on v6? It has a v6 address, but doesn't seem to be answering19:56
bkero$ curl -v6
bkero*   Trying 2001:4800:780e:510:3bc3:d7f6:ff04:fb20...19:56
clarkbI think it should listen on v619:56
bkeroAh, it returned after about 28 seconds into a 30 second timeout19:57
clarkbtcp6       0      0 :::80                   :::*                    LISTEN      1820/apache219:57
bkeroMaybe Portland comcast >_>19:57
clarkbbkero: you might be suffering the comcast derp19:57
* bkero shakes fist19:57
clarkbI am tmobiling now19:57
*** ddieterly has quit IRC19:57
bkerocomcast seems to derp a lot more on v6 than v419:57
clarkbyes by far19:57
bkeroSome sites don't load at all, some are super slow19:57
*** asettle has joined #openstack-infra19:57
*** weshay has quit IRC19:57
clarkbthey seem to drop packets in seattle and in the denver area a lot19:58
*** flepied has quit IRC19:58
bkerooh yeah19:58
bkerothere's totally some big segments of packet loss, v6 and v419:58
bkeroI can't ping paste on v6 or v4 right now19:59
bkeroI was pinging it, then the packets just...stopped.19:59
*** flepied has joined #openstack-infra20:00
*** elmiko is now known as _elmiko20:00
AJaegerclarkb: how many jenkins are updated now?20:00
clarkbAJaeger: I started with 01-03 and 0720:00
*** ddieterly has joined #openstack-infra20:01
clarkbAJaeger: none are complete yet :/20:01
AJaegerfungi, pabelanger, clarkb : Seems that jobs are running fine now - if they run on ubuntu-trusty...20:01
clarkb07 and 03 are mostly done20:01
clarkb01 and 02 are slower20:01
AJaegerclarkb: thanks for the update...20:01
clarkband I saw some 502s go by so I will likely run twice on each master to make sure it is complete20:02
*** asettle has quit IRC20:02
clarkbwe have a lot of xstatic projects20:02
*** Sukhdev has quit IRC20:02
pabelangerAJaeger: Ya, the ubuntu-trusty stuff I switched this morning seems to be working fine.... minus curl20:02
clarkbI am listening to the radio on my tv, there is something wrong about that20:02
anteayaI like listening to the radio20:03
AJaegerpabelanger, clarkb : I checked a few job that recently run and those looked fine20:04
Clinti've listened to tv on a radio20:04
*** jamesmcarthur has joined #openstack-infra20:04
clarkbAJaeger: great20:04
pabelangerAJaeger: Ya, I think we are in good shape20:06
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci: WIP: Revert "Use swapfile environment in CI"
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Add examples directory
*** derekh has joined #openstack-infra20:06
openstackgerritSpencer Krum proposed openstack-infra/system-config: Run cacti's node generation from cron
*** ccrouch1 has quit IRC20:10
*** piet has quit IRC20:11
*** ccrouch has joined #openstack-infra20:11
nibalizermtreinish: at any rate20:12
nibalizeri think we've run puppet now20:12
*** weshay has joined #openstack-infra20:12
anteayaso I'm trying to get puppet-hound running on a vm I created20:13
anteayaand currently am seeing this error20:13
anteayaFailed to fetch ec2 uri 500 Internal Server Error20:13
anteayaas part of the output when I run puppet apply20:13
anteayathe catalog run reports that it finishes20:13
nibalizeroh boo20:13
mtreinishnibalizer: ok, cool. Yeah it looks that way20:13
anteayahowever there doesn't seem to be a process called hound20:14
nibalizerchown: invalid user: ‘logstash’20:14
nibalizerstart-stop-daemon: user 'logstash' not found20:14
mtreinishnibalizer: although I don't see it in cacti20:14
mtreinishnibalizer: oh, more logstash refs...20:14
nibalizeri can't see where in the init script its being called20:14
mtreinishnibalizer: I know where it is20:14
clarkbgrep for USER20:15
mtreinishI fixed it here:
nibalizerthe user oh there it is derp20:15
*** kzaitsev_mb has quit IRC20:15
*** yamamoto has joined #openstack-infra20:16
*** bodepd_ has quit IRC20:16
*** roxanaghe has quit IRC20:16
*** regXboi has quit IRC20:16
nibalizermtreinish: do we want to run it as root?20:16
nibalizeror some subunit user or w/e ?20:16
*** annegentle has quit IRC20:17
anteayanor is go installed, nor is /home/hound created20:17
anteayaso puppet apply doesn't seem to be doing much that I can see20:17
mtreinishI'm not sure, but I'd have to create a subunit user too20:17
krotscheckWhich is the shade channel?20:17
*** bodepd has joined #openstack-infra20:17
nibalizeranteaya: this is possible a facter bug20:18
nibalizerwhat puppet version do you have20:18
*** derekh has quit IRC20:18
*** ddieterly has quit IRC20:18
jeblairkrotscheck: this one20:18
krotscheckjeblair: Oh. Neat!20:18
anteayanibalizer: how do I find out?20:18
nibalizeranteaya: puppet --version20:18
anteayaubuntu@hound:~$ puppet --version20:18
nibalizerok so run 'puppet apply -e 'notify { "derp": } '20:19
krotscheckI just did a pip install --upgrade shade, and suddenly os_keypair is barfing on me.20:19
anteayanibalizer: I will20:19
*** jcoufal has quit IRC20:19
anteayanibalizer: I'm in a console?20:20
nibalizeroh god20:20
nibalizerso puppet apply -e 'notify { "derp": }'20:21
anteayabest advice I've had all day20:21
*** yamamoto has quit IRC20:21
nibalizerexactly like that20:21
* anteaya copy pastes20:21
mtreinishnibalizer: do you want me to respin that patch to add a subunit user and use that for the init script?20:22
nibalizermtreinish: idunno20:22
nibalizerfungi: clarkb do you care?20:22
clarkb07 just finished and I restarted it to redo the things that errored20:22
clarkbnibalizer: about?20:22
anteayaclarkb: just generally20:22
anteayado you care?20:22
anteayaI've always considered you a caring person20:23
*** rfolco has quit IRC20:23
mtreinishclarkb: about running the subunit worker as root20:23
clarkbmtreinish: nibalizer oh I would prefer we used a system user20:24
*** rfolco has joined #openstack-infra20:24
mtreinishclarkb: ok, then I'll respin the patch to add a subunit user and use that20:24
nibalizermtreinish: thanks20:24
clarkband I just restarted on 0320:24
nibalizersorry not trying to put a ton of hoops in your way20:24
nibalizeralso im curious how subunit01 works heh20:24
nibalizerprobably it as a logstash user for $REASONS20:24
mtreinishnibalizer: yeah, subunit-worker01 was death by a thousand patches getting it up and running20:25
*** piet has joined #openstack-infra20:25
mtreinishit was also > 1 yr ago when we started it20:25
nibalizerokay so maybe its earned itsefl a spot on the reload-block20:25
nibalizerif we can get 02 in a place where its happy20:26
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
mtreinishnibalizer: sure, although we'll have to be careful when we do to make sure we don't drop any queued streams it's processing20:27
anteayanibalizer: in case you missed it when your name wasn't in front of it:
anteayano hurry20:27
nibalizeranteaya: so that is great then20:28
nibalizernot a puppet/facter issue20:28
nibalizercan you paste the full output of the puppet run you are doing to install hound?20:28
nibalizerand what cloud are you using?20:28
*** asettle has joined #openstack-infra20:29
anteayayay that's great20:29
openstackgerritMatthew Treinish proposed openstack-infra/puppet-subunit2sql: Switch /etc/logstash to /etc/subunit2sql
mnaserhello #-infra -- we're investigating adding ipv6 by default for all instances (so all instances are dual stack for public network connectivity).  we used to do this for specific tenants that required it but we'd like to open it up for everyone.   i was wondering if there is any feedback as cloud consumers of "this is what i like with x ipv6 impl" "this is what i dont like" "this is what i would ideally like"20:29
mtreinishnibalizer: ^^^20:29
*** sc68cal has quit IRC20:29
mtreinishI think I did that right, my puppet syntax is a bit rusty20:29
AJaegernibalizer: could you review so that we get curl on ubuntu-trusty,please?20:29
clarkbmnaser: we like that it means no NAT with neutron :) but you don't NAT by default anyways on v420:30
anteayanibalizer: hpe's internal mce cloud:
mnaserthere seems to be many ways of how ips are allocated to instances, SLAAC, dhcpv6, etc.  i was just hoping to get some feedback about experiences / etc20:31
clarkbmnaser: currently the only cloud we use with ipv6 is rackspace and I am not sure how they assign those addrs20:32
*** salv-orlando has quit IRC20:33
mnaseri will guess that it is probably static with nova-agent20:33
mordreddreamhost also does ipv6, but we do not use it for anything20:33
anteayanibalizer: I'm totally guessing as to what to give puppet to make it hound20:33
*** salv-orlando has joined #openstack-infra20:33
*** jtomasek has quit IRC20:33
mordredmnaser: markmcclain could probably tell you what they did20:33
nibalizeranteaya: sure20:33
mnaserwould be good to hear20:33
greghaynesI have had a great experience doing slaac v6 with openstack fwiw20:34
greghaynesit has 'just worked'20:34
*** asettle has quit IRC20:34
greghayneshighly recommend :)20:34
* mtreinish checks which of the ipv6 allocation we still run on every tempest run20:34
anteayamtreinish: actually that was part of the feedback from ops meetup20:35
Shrewskrotscheck: plz to define "barf"20:35
markmcclainmnaser: sure happy to tell you what we did when dual stacking v620:35
krotscheckShrews: Sorry- was trying to debug20:35
krotscheckHave a stack trace, sec.20:35
anteayawe seem to get ipv6 running and then something new comes along a breaks it20:35
mtreinishit looks like slaac and dhcpv6:
mtreinishanteaya: sure, thats true for us in the gate too (despite having test coverage for some of it)20:36
mnaser -- this seems like a very very useful document20:36
mtreinishanteaya: which is why we demoted it to periodic (with the slow tag)20:36
anteayamtreinish: okay so we are in agreement on the current status, that's a start20:36
mtreinishbecause no one was looking at the failure20:36
anteayamtreinish: ah :(20:36
anteayaoh okay20:36
krotscheckRunning shade 1.4.0 and ansible
anteayaso ipv6 folks to qa20:36
mtreinishand those tests are really poorly constructed and very expensive to run20:37
nibalizeranteaya: that error looks totally benine to me20:37
mtreinishanteaya: it's all on the neutron side20:37
* krotscheck is building out a venv right now with all the things.20:37
*** yamamoto has joined #openstack-infra20:37
anteayanibalizer: yeah the catalog finished running20:37
Shrewskrotscheck: pull the latest version from ansible repo. Should fix it for you20:37
*** yamamoto has quit IRC20:37
*** bpokorny has quit IRC20:37
anteayanibalizer: but no hound process, no /home/hound dir, no installation of go20:37
anteayanibalizer: so not sure what apply actually did20:37
mtreinishanteaya: that thread was some of the discussion20:38
anteayamtreinish: what is?20:38
anteayamtreinish: thanks20:38
mordredkrotscheck: I generally prefer for os_keypair to not barf20:38
Shrewskrotscheck: if not, will have a closer look in a bit20:38
anteayamtreinish: not so very old20:38
*** yamamoto has joined #openstack-infra20:39
*** yamamoto has quit IRC20:39
Shrewskrotscheck: mordred   I think that's the ansible bug fix to default to str. Fixed with the latest modules20:39
krotscheckShrews: Ok, i'll see if I can figure out how to do that.20:39
*** sdake has quit IRC20:39
*** weshay has quit IRC20:39
openstackgerritMerged openstack-infra/system-config: Move curl to jenkins_common.pp
*** cfarquhar has joined #openstack-infra20:40
*** cfarquhar has quit IRC20:40
*** cfarquhar has joined #openstack-infra20:40
pabelangerclarkb: nibalizer thanks! ^20:41
*** annegentle has joined #openstack-infra20:41
clarkbpabelanger: I will kick of an image rebuild in a few (lunch now)20:41
anteayaclarkb: enjoy lunch20:42
Shrewskrotscheck: I can pull the PR for you later if you need it. afk atm20:42
mtreinishmordred, Shrews: btw, I meant to ask is this basic approach ok?
mtreinishI wanted to make sure it as ok before I start pushing it to completion20:43
mtreinishbecause it's far from working right now20:43
openstackgerritMerged openstack-infra/shade: Mock glance v1 image with object not dict
AJaegerclarkb: enjoy lunch20:43
*** bpokorny has joined #openstack-infra20:43
clarkbAJaeger: I have job updates running everywhere but 06 now20:43
clarkbsince 06 needs direct attention (which I will try to get to as well)20:44
*** esikachev has quit IRC20:44
*** nwkarsten has quit IRC20:44
anteayaI had hoped at friday afternoon the rush would be quietening down20:44
anteayabut the gate is at 3820:44
*** ericksonsantos has quit IRC20:45
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Follow name_or_id pattern on domain operations
mordredmtreinish: yes, looks good. I think you want self.list_images.invalidate instead of self._cache.invalidate20:45
*** nwkarsten has joined #openstack-infra20:45
AJaegerclarkb: thanks for taking care of all of this!20:45
mtreinishmordred: ok, cool. I'll do that when I respin it20:45
*** Sukhdev has joined #openstack-infra20:46
clarkbAJaeger: no problem, I wasn't able to help yesterday during feature freeze and :(20:47
anteayamtreinish armax do either of you happen to know if any overture has been made on the operators list to see if we can recruit some folks who care about ipv6 to at least champion the bugs?20:48
*** nwkarste_ has joined #openstack-infra20:48
*** thorst_ has joined #openstack-infra20:48
mtreinishanteaya: I haven't seen anything20:49
anteayamtreinish: thank you20:49
anteayaI'm open to composing an email draft to them but didn't want to step on toes20:49
*** nwkarsten has quit IRC20:49
*** _nadya_ has quit IRC20:49
krotscheckShrews, mordred: Well, that sortof worked, in that it's failing differently now "rror fetching keypair list (Inner Exception: Could not determine a suitable URL for the plugin)"20:50
* krotscheck probably doesn't have a link to his cloudconfig in his venv20:51
anteayanibalizer: I'm at a loss as to what to do next to get puppet to hound, about to eat something but if you have any thoughts, I'm listening20:51
*** asettle has joined #openstack-infra20:52
*** thorst has quit IRC20:52
mordredkrotscheck: 'could not determine suitable URL' is a keystone auth plugin discovery error20:52
*** weshay has joined #openstack-infra20:53
mordredkrotscheck: it usually means that keystoneauth can't figure out from your arguments what auth plugin to use - although I imagine it might also be that if you don't have a linked config20:53
krotscheckmordred: Yeah, I'm macgyvering my system's ansible install right now to see if that's it.20:54
*** geekinutah has quit IRC20:55
*** asettle has quit IRC20:56
anteayawhere are you applying the paper clip?20:56
*** cloudtrainme has joined #openstack-infra20:56
anteayathe paper clip does all things20:57
jrollanybody else seeing pillow install failures? 2016-03-04 20:48:12.367 |     ValueError: jpeg is required unless explicitly disabled using --disable-jpeg, aborting20:57
clarkbjroll: do you have other-requirements.txt?20:57
jrollclarkb: nope20:57
clarkbok so likely using the fallback list20:58
jrollclarkb: I don't know what that is :/20:59
*** Sukhdev has quit IRC20:59
krotscheckmordred: The config is definitely linked - if I change it it yells at me, and adding an explicit version just yells at me more because /tokens does not exist. Are we expecting v3.0?20:59
krotscheck(keystone v3)20:59
clarkbjroll: those jobs are installing their deps via a bindep list now20:59
pabelangerclarkb: jroll: I can get a patch for that20:59
clarkbjroll: or optionally a project specific other-requirements.txt20:59
clarkbpabelanger: ty that will also need a new image since it is the fallback list21:00
jrollpabelanger: clarkb: I assume this is new?21:00
jrollpabelanger: also, thank you21:00
pabelangerjroll: Yup, gets us off bare-trusty images, which are pinned to rax21:01
jrollpabelanger: yay!21:01
jrollNobodyCam: ^ fyi, might be a bit before that's done as it requires new images21:01
clarkb and my networking is completely hosed21:03
*** deray has joined #openstack-infra21:03
clarkbI need to get a phone charger then I can tether more21:04
mordredkrotscheck: can you paste your config without your password?21:04
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add libjpeg to bindep-fallback.txt
pabelangerclarkb: jroll: I think that will do the trick ^. Jenkins will confirm21:05
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: dump a release tracking dashboard
*** sdake has joined #openstack-infra21:05
clarkbpabelanger: if those are the package names from the old thick slave list then yup21:06
AJaegerpabelanger: apt is wrong for platform in your change, isn't it?21:06
pabelangerAJaeger: yes, thanks21:06
clarkbAJaeger: only if debian and other apt systems use a different name21:06
AJaegerpabelanger: shouldn't it be dpkg ?21:06
clarkbor is it ya that ok21:06
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add libjpeg to bindep-fallback.txt
pabelangerAJaeger: good eye21:06
*** rguillebert has quit IRC21:07
pabelangerI love how firefox is in bindep-fallback.txt21:07
*** piet has quit IRC21:07
pabelangerbrowser testing I guess?21:07
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: dump a release tracking dashboard
rcarrillocruzi guess21:08
AJaegerpabelanger: for horizon etc.21:08
AJaegerpabelanger: moving it to other-requirements would be a future cleanup...21:09
pabelangerAJaeger: indeed21:09
AJaegerbut first let's get everything working ;)21:10
*** apoorvad has quit IRC21:10
* AJaeger needs to call it a day now.21:10
anteayathanks AJaeger21:11
pabelangerAJaeger: enjoy21:11
anteayaenjoy the weekend21:11
AJaegerThanks everybody to get us that far. Hope there aren't anymore surprises ;)21:11
AJaegerenjoy the weekend, anteaya, pabelanger and everybody ;)21:11
clarkbwoot osic is ready to go21:14
* clarkb removes wip from first change21:14
*** ldnunes has quit IRC21:14
*** rguillebert has joined #openstack-infra21:15
clarkbinfra-root please review
clarkbwith that I can work on getting a mirror host deployed21:15
openstackgerritElizabeth K. Joseph proposed openstack-infra/system-config: Customize cgit max-repo-count and set it to 1500
pleia2clarkb: dns \o/21:17
*** dprince has quit IRC21:18
*** shardy has quit IRC21:18
anteayaclarkb: yay21:19
anteayamtreinish armax my first draft of an email to the operator's list recruiting help for ipv6:
clarkbjenkins 03 and 07 are all jjb'd. Every other jenkins is in progress (including 06), and a nodepool image build is happening but we will need a new one with new fallback bindep list21:20
anteayamtreinish armax I'm thinking Monday would be a better day to post, rather than Friday afternoon21:20
anteayaor Saturday for some21:20
anteayaclarkb: yay!21:20
pabelangerclarkb: nice21:20
clarkbdebating if I want to add the osic mirror build to the list or wait until a few things pop off :)21:20
anteayadimsum_: so my status on getting a test hound instance up is that I can't figure out how to get puppet to tell hound to run21:21
mnaseranteaya: actually we're discussing and we're thinking of using slaac by our router21:21
anteayadimsum_: sorry I don't have better news21:21
dimsum_anteaya : no worries. thanks. /me does not know puppet either21:21
anteayamnaser: is this in response to my etherpad?21:21
mnasersomewhat :-P21:21
anteayadimsum_: were you able to get your work done?21:21
anteayamnaser: okay great21:21
mnaserwe have a big customer that is pushing on using it21:22
mnaserand they're usually nice about understanding that we're bringing in new-er things21:22
mnaser(aka things might break)21:22
anteayamnaser: would this bug affect your usage of slaac?
openstackLaunchpad bug 1540983 in neutron "Gate failures for neutron in test_dualnet_multi_prefix_slaac" [Critical,Confirmed]21:22
anteayamnaser: yay for nice customers!21:22
mnaserthey use a ton of -infra projects like nodepool etc :)21:22
mnaserlet me see21:22
anteayacool, thanks21:22
anteayamnaser: sounds like a great customer :)21:23
* mnaser reads that test21:23
*** dmsimard has quit IRC21:23
*** roxanaghe has joined #openstack-infra21:24
mnaserthats strange, i dont think i can imagine this affecting us21:24
clarkbgah there is a bug in my osic dns change21:24
clarkbwomp womp21:24
mnaser2 ports generally does cause troubles, ipv4 or ipv621:25
anteayamnaser: can you add your perspective as a comment on the bug?21:25
mnasergood idea21:25
anteayamnaser: that would be helpful21:25
anteayathank you21:25
openstackgerritClark Boylan proposed openstack-infra/system-config: Verify SSL with OSIC
mnaserlet me read the initial commit21:25
anteayaclarkb: :(21:25
mnaserjust to see the idea behind that test21:25
anteayamnaser: yup, makes sense21:25
clarkbpleia2: do you mind rereviwing that really quick?21:25
armaxanteaya: let me reassess the severity of those issues, because I haven’t seen them  burning the gate recently21:26
clarkbI didn't actually remove the verify: false from one of the entries21:26
armaxanteaya: that doesn’t mean they should not be addressed though21:26
anteayaarmax: awesome thank you21:26
armaxanteaya: thanks for reviving this up, I know haleyb volunteered at one point21:27
armaxanteaya: but don’t count on it21:27
anteayaarmax: I also want the ops to know we listen to their needs21:27
anteayaarmax: yeah I know he is busy21:27
anteayaarmax: feel free to edit what I wrote so it reflects reality21:27
anteayaarmax: I was just making stuff up there21:27
armaxlet me double check the status of those bugs21:28
anteayaarmax: wonderful thank you21:28
clarkbzuul looks like it is falling behind processing the results queue21:29
clarkbjeblair: ^ if this persists you may be interested21:30
*** jpr has joined #openstack-infra21:30
odyssey4meis the build up of events and results for zuul a known thing? it's been more than 10 mins and I don't see a review in the queue21:30
dimsum_anteaya : no worries, it's not that bad21:30
odyssey4melol clarkb great minds21:30
anteayaclarkb jeblair zuul memory usage is high:
mnaserreading over the code, this comment here seems really relevent:
openstackLaunchpad bug 1540983 in neutron "Gate failures for neutron in test_dualnet_multi_prefix_slaac" [Critical,Confirmed]21:31
clarkbI keep waiting for the number to fall but it seems ot only grow21:31
mnaseri unfortunately don't have time to look into it even more but i believe that is largely unaffected21:31
anteayamnaser: thanks for looking21:31
mnaseralso, how often could we have logs regarding error'd instances from nodepool, we'd like to flatten it out to 0 error node launch attempts21:31
anteayaclarkb: only 2GB of memory available left for zuul21:32
*** rfolco has quit IRC21:32
anteayaclarkb: if jeblair doesn't show up soon, I would advocate for a restart of zuul21:32
armaxanteaya: unfortunately with the current logstash status we’re sort of shootign blind21:32
clarkbanteaya: I'd like to see all of these completed jobs result in merges :(21:33
anteayaarmax: I understand, it is an overture to work together on a common issue they have said is important21:33
armaxanteaya: ack21:33
anteayaarmax: I think a doorway to help is the important thing21:33
anteayaarmax: if you can make the words true that will accomplish that I'd be grateful21:33
anteayaarmax: going to send it monday, friday is poorly timed for an overture21:33
anteayaclarkb: I agree21:34
armaxanteaya: right now logstash is not returning any result in the past 30 days21:34
armaxanteaya: might be true, might be false, I simply don’t trust the tool any longer :(21:34
anteayaarmax: yes, clarkb has been working on getting logstash happy again21:34
clarkbarmax: we only ever record 10 days of info fwiw21:34
armaxclarkb: I know21:34
armaxbut one can only hope ;)21:35
anteayaclarkb: look at that lovely stack of patches ready to merge21:35
anteayac'mon zuul21:35
*** apoorvad has joined #openstack-infra21:36
anteayaarmax: the amount of storage we would have to have for 30 days is beyond our resources21:36
anteayaarmax: we have challenges keeping up with 10 days worth of logs21:36
anteayajeblair: nice to see you21:37
pleia2jeblair: it rained today :(21:37
jeblairpleia2: that's good, remember?21:37
clarkbjeblair: it looks like the zuul mainrun loop isn't looping21:37
pleia2jeblair: oh yes, but now I don't want to go outside!21:37
*** roxanaghe has quit IRC21:37
armaxI’d be happy with 7 days, if I knew that 0 results really mean 0 results21:38
anteayajeblair: zuul is almost out of memory21:38
openstackgerritEmilien Macchi proposed openstack-infra/project-config: gerrit: add puppet-neutron-core
armaxbut that’s ok, I realize you guys are working to make this great again, if there’s anything I can do, user testing if nothing else21:38
armaxplease give me a shout21:38
anteayaarmax: thank you21:38
armaxI don’t mean to pres21:38
anteayaI know21:39
anteayawe appreciate the support21:39
anteayaand that we have tools devs use21:39
anteayaand like on the good days21:39
*** yamamoto has joined #openstack-infra21:39
clarkb but not sure if that is related21:40
jeblairi think that's harmless21:41
anteayazuul eth0 out has peaked:
jeblairnow that's interesting...21:42
anteayait doesn't feel healthy21:43
clarkbI wonder if that is due to jenkins job updates21:43
clarkbthat inefficient gearman update21:43
*** rlandy has quit IRC21:43
clarkbI can halt some of them21:44
*** |-paul-| has joined #openstack-infra21:44
*** rlandy has joined #openstack-infra21:44
jeblairmost outbound traffic is to graphite21:45
jeblairabout 5mbps worth21:45
anteayasomething is querying graphite?21:45
jesusaurfungi: are you still around?21:45
anteayaa lot?21:45
anteayajesusaur: he is currently offline21:45
jesusauranteaya: thanks21:45
anteayajesusaur: he had to travel for a thing and ran out of battery21:45
clarkbanteaya: no zuul only sends data to graphite21:45
anteayajesusaur: I'm sure he'll read backscroll21:46
anteayajesusaur: welcome21:46
anteayaclarkb: okay, thanks21:46
jeblairclarkb: most of which are incrementers and timers for geard CAN_DO calls21:46
clarkbjeblair: ah that makes sense21:46
clarkbsince we are updating them right now21:46
fungiokay, i'm back home. looks like jenkins06 has been cleaned up and restarted (thanks!) but isn't running any jobs at all?21:47
anteayafungi: ah here you are21:47
*** Douhet has quit IRC21:47
clarkbfungi: zuul has stopped processing the results queue as quickly as we would like21:47
clarkbif at all21:47
*** salv-orlando has quit IRC21:47
*** kgiusti has left #openstack-infra21:47
jeblairthat will be from geard; which shouldn't stop the zuul main loop, except indirectly (by being slow to respond)21:47
*** yamamoto has quit IRC21:48
jesusaurahh, fungi a couple days ago you were talking with AJaeger about ZUUL_REF vs ZUUL_REFNAME, was there a decision to stop using ZUUL_REF in favor of ZUUL_REFNAME?21:48
*** salv-orlando has joined #openstack-infra21:48
jeblairclarkb: i see main loop activity at 21:44; last time i looked it was at 21:3221:49
jeblairclarkb: so i think it's moving21:49
clarkbjeblair: just slow21:49
fungijesusaur: i think prior consensus was that in jobs which will run in post and need to know the name of the branch to which the change merged, that both ZUUL_REF and ZUUL_REFNAME will contain the same values, but that using ZUUL_REFNAME is perhaps marginally more correct21:50
jeblairfungi: i would vote that way (ZUUL_REF is "the thing you should fetch")21:50
*** hashar has quit IRC21:50
fungiyep, and if you want a script to get the branch name in post but the tag name in tag-based queues, then ZUUL_REFNAME does that21:51
jeblairZUUL_REFNAME is "the name of the ref that triggered a ref-updated event"... right.21:51
*** salv-orlando has quit IRC21:51
clarkbI am tethering off of my phone so debugging is hard21:51
jesusaurah, well in that case we will need to teach zuul-cloner about ZUUL_REFNAME, it currently only has ZUUL_REF21:51
anteayaclarkb: :(21:52
*** salv-orlando has joined #openstack-infra21:52
clarkbit would not surprise me if somehow the jjb updates are adversely affecting zuul via gearman21:52
*** ddieterly has joined #openstack-infra21:52
openstackgerritSam Yaple proposed openstack-infra/shade: Add update_service()
*** ddieterly has quit IRC21:53
jeblairclarkb: i agree21:53
*** aysyd has quit IRC21:53
jeblairwhy are we updating everything?21:53
anteayamoving jobs from bare trusty to ubuntu trusty21:53
*** roxanaghe has joined #openstack-infra21:54
jeblairwhen did jjb start running?21:54
clarkbI started 01-03 and 07 several hours ago (~10:30am pst)21:55
clarkbwhen 03 nad 07 completed I started the remainder21:55
clarkbabout an hour ago21:55
jeblairthe traffic pattern changed around 204521:56
jeblairwhich is about an hour ago21:56
fungimaybe gearman registration flood slamming zuul's geard?21:57
clarkblooks like 01 is done now too21:58
jeblairfungi: yeah, that's the theory, though it's interesting that the 4 clarkb started earlier would not have caused this, but the second 4 (plus 2 still running) would...21:58
*** pcaruana has joined #openstack-infra21:58
cody-somervillejeblair, fungi, anteaya, mtreinish, etc: FYI, here is the review for the Gertty breadcrumb footer bar,
clarkb02,04-06 are the ones currently running21:58
jeblairmost gearman activity seems to be 2,4,5,621:58
anteayajenkins06 was having an off day prior to the jjb update, might it still be off?21:58
anteayacody-somerville: thank you21:59
clarkbanteaya: no, I brought it back and clenaed up its slaves21:59
anteayaclarkb: okay thanks21:59
fungiwell, 01 and 02 are seriously slow, but 4/5/6 are a good deal faster21:59
*** automagically has joined #openstack-infra21:59
clarkbso maybe thats what it is?21:59
clarkb04-06 can hammer the geard server harder?21:59
fungicould be they're able to flood better21:59
*** AndyU has joined #openstack-infra21:59
jeblairclarkb: you suggested suspending some of them... i'm in favor of doing that now to see the effect22:00
fungiruled out something spamming gerrit comments bogging down the event queue?22:00
*** _sarob has joined #openstack-infra22:00
anteayafungi: nope22:00
fungii'll take a peek in gerrit's logs22:01
cody-somervillejeblair: have to say it was pretty easy to implement thanks to good architecture on (assuming) your part. :)22:01
anteayafungi: thanks22:01
clarkbyou think just ^Z them?22:01
jeblairfungi: it's manifesting in zuul as a very slow main loop; doesn't look like a gerrit spam problem, but good to double check22:01
*** deray has quit IRC22:01
jeblaircody-somerville: thanks and thanks :)22:01
jeblairclarkb: yeah22:02
AndyUanteaya, pleia2, fungi, clarkb, etc....:  Hi all, I'm back with another question. I was give this specific question "Does the community mandate any specific OpenStack components running and what access level do they require to those services?"22:02
clarkbI ^Z'd two of the 422:02
clarkb04 and 05 are the suspended processes22:02
jeblairclarkb: i still see much tcp from them but will keep an eye out for that being reduced22:03
fungivery little comment activity, as befits late fridays22:03
anteayaAndyU: ah the only mandates regarding openstack componets come from refstack and <the name of the other thing>22:03
jeblairclarkb: i think it has reduced :)22:03
anteayafungi: yay22:03
jeblairmost gearman traffic is 02 and 06 now-22:03
clarkbI left 02 and 06 because they were further along so in theory will finish quicker22:03
AndyUI will wait patiently for your answer  ;)22:03
fungimost active commenter during the 21:5.* timeframe was fuel-ci and it left a scant 8 comments in that 10-minute span22:03
clarkbAndyU: we do!, we need nova, keystone, glance. We can in theory work with nova net but all our clouds are neutron now so that may be an implicit requirement at this point. And we like it if we can use cinder22:03
*** sarob has quit IRC22:04
anteayaAndyU: regarding a cloud for infra testing, we just need the ability to create vms, upload images, reach the internet and upload logs22:04
anteayaAndyU: and then delete them22:04
*** roxanaghe has quit IRC22:04
anteayaso whatever you need to run so we can do that22:04
*** rlandy has quit IRC22:04
jeblairclarkb: main loop activity (as indicated by report events appears to be increasing22:04
anteayaAndyU: and thanks for your patience :)22:04
*** _sarob has quit IRC22:04
*** thorst_ has quit IRC22:05
AndyUI'll take as much info as you can give me. No hurry. I'll pass along all your comments.22:05
anteayaAndyU: so what clarkb said22:05
*** roxanaghe has joined #openstack-infra22:05
*** dbrycen has joined #openstack-infra22:05
jeblairclarkb: zuul is now transferring more data to gerrit than graphite22:05
clarkbAndyU: for perms we don't need anything fancy. We need to boot and delete instances with public IPs and wide open security groups. We need to upload disk images. and thats about it22:05
openstackgerritMerged openstack/requirements: Bumps os-win version to 0.2.3
AndyUok, got it.22:06
clarkbjeblair: 06 is done, shoul dI want before foregrounding one of 04 and 05?22:06
fungizuul debug log shows it adding events pretty steadily, but also lots of job description setting in jenkins22:06
fungimost of the events look like ref-replicated22:06
jeblairclarkb: i think so, let's see if it can make forward progress22:06
fungiwhich it's going to ignore22:06
*** ddieterly has joined #openstack-infra22:07
clarkbthe top of the gate is no longer there22:07
AndyUSo, nova, keystone, glance neutron, cinder...22:07
*** dimtruck is now known as zz_dimtruck22:07
*** thorst has joined #openstack-infra22:07
clarkbAndyU: yes, are there particular services you are worried about?22:08
*** annegentle has quit IRC22:09
AndyUI don't think so but the question came from my boss22:09
openstackgerritColleen Murphy proposed openstack-infra/puppet-infracloud: Fix apache failure
dbrycendoes anybody know how to get around PBR_VERSION conflicting with an explicit pbr>=x.x in a's setup_requires?22:09
fungiAndyU: to the extent that MOS is hopefully "MOStly OpenStack" i would be surprised if it's unsuitable22:09
AndyUYeah, I think we're looking pretty good22:09
*** moravec has quit IRC22:10
*** bgmccollum has joined #openstack-infra22:10
*** roxanaghe has quit IRC22:10
clarkbseparately I am uploading a new ubuntu-trusty image everywhere that should have the curl fix but not the libjpeg-dev fix22:10
fungiat least the things i've heard about it imply it's no less openstacky than rdo (and possibly more openstacky than some of the cloud providers already donating resources to us)22:10
*** roxanaghe has joined #openstack-infra22:10
anteayaclarkb: ack22:10
clarkbdbrycen: is the problme that you have an older version preinstalled and it complains that it is not new enough?22:11
clarkbdbrycen: your best bet is probably to upgrade pbr?22:11
AndyUall the projects that you mentioned are in play and many more22:11
dbrycenclarkb: well i'm just trying to do a "python sdist" and i dont have pbr installed22:11
jeblairclarkb: i think we may want to let it run one at a time...22:11
*** e0ne has joined #openstack-infra22:11
clarkbjeblair: ok22:11
*** roxanagh_ has joined #openstack-infra22:11
jeblairclarkb: is it in a screen we can join?22:11
fungiclarkb: were we installing curl and libjpeg-dev on bare-trusty or were those things that rackspace was probably preinstalling on their images and used by unusual jobs which our numerous experimental vetting attempts didn't uncover?22:11
AndyUThanks all, I'll take back what you gave me. Have a great weekend.  ;-)22:12
clarkbjeblair: yes all in screens (owned by me not root which makes it a little tricky) on 02, 04 and 05)22:12
*** thorst has quit IRC22:12
clarkbdbrycen: if you don't have pbr installed it should install the latest version for you22:12
*** roxanaghe has quit IRC22:12
jeblairclarkb: which one is running now?22:12
clarkbjeblair: 0222:12
fungisudo su - clarkb ; script /dev/null ; screen -x22:12
clarkbdbrycen: might help if you can paste the error output somewhere22:13
*** weshay has quit IRC22:13
jeblairokay, i believe this is making forward progress and will eventually catch up22:13
fungidbrycen: where "somewhere" is probably
fungiand preferably not in channel at least22:13
jeblairi'm going to go afk for a bit, but will check back later and fg jjb runs as needed22:14
fungithanks jeblair!22:14
dbrycenclarkb: that's what i saw happening but it was making the version of pbr equal to what i set for PBR_VERSION and that version conflicts with what version of pbr is required by It sounds funky but i'm using PBR_VERSION env var to make a custom version for my openstack project.22:14
anteayaAndyU: have a good weekend22:14
*** baoli_ has quit IRC22:14
clarkbdbrycen: that shouldn't affect the version of PBR, that affects the version that PBR calculates for the project being installed22:15
clarkbI think pasted output may help22:15
dbrycenok let me do that22:16
clarkbjeblair: note that 02 may need to be restarted if it errored out along the way, the end of its output will tell you if it did22:16
clarkbI am watching it though and can likely get these done22:16
*** moravec has joined #openstack-infra22:16
*** gordc has quit IRC22:17
clarkbyup it did 502 so I have just restarted it22:17
*** asettle has joined #openstack-infra22:18
*** Sukhdev has joined #openstack-infra22:19
dbrycenclarkb: note that if i change PBR_VERSION to be a version which cooperates with setup_requires, then the sdist creation is successful22:20
*** Sukhdev has quit IRC22:20
clarkbdbrycen: I don't think that is related to PBR_VERSION I think instead that you already have pbr installed there and it isn't new enough? or maybe if you use a pypi mirror you don't have a new enough version of pbr there22:20
*** Sukhdev has joined #openstack-infra22:20
clarkbdbrycen: is there no pbr egg in that dir?22:21
AndyUanteaya: thanks!22:21
*** AndyU has quit IRC22:21
dbrycenclarkb: nope, no pbr egg in the repo22:21
clarkbdbrycen: what about pip freeze | grep pbr?22:21
dbrycenclarkb: pip isnt installed and it's not in dpkg list either22:22
clarkbhuh I wonder if that is realted, I think pbr requires pip22:23
*** asettle has quit IRC22:23
dbrycenhmm, i can try installing it to see if it helps22:23
clarkbwhat version of setuptools do you have?22:23
*** thorst has joined #openstack-infra22:23
fungizuul has basically emptied its results queue now and is starting to burn down the ~16k events that generate22:23
clarkbsince that is the thing attempting to install pbr in the first place22:23
*** annegentle has joined #openstack-infra22:23
dbrycenclarkb: setuptools is 3.322:24
clarkbfungi: care to review that? osic is ready for us22:24
clarkbdbrycen: updating setuptools is the other thing I would try, that version is fairly old22:24
* pabelanger loads to watch the light show of 'in use' nodes22:25
dbrycenclarkb: cool i will try that as well and report back22:25
odyssey4mealright, I'm out for the night - have a fantastic weekend everyone!22:25
pabelangerodyssey4me: later22:25
clarkbI like that we got our job count down to about 8k and a few weeks later its back to about 9k :)22:25
*** akshai has quit IRC22:25
pabelangerclarkb: where are you looking at that?22:26
fungiclarkb: hah, have you looked at the cert? i'm curious to see in 3 months whether they've punted on the auto-update process for that22:26
*** abregman|afk is now known as abregman22:26
*** cloudtrainme has quit IRC22:26
openstackgerritMerged openstack-infra/openstackid-resources: Fix on category groups expand
*** tpsilva has quit IRC22:27
fungiand now zuul's back to 0 events, 0 results22:27
pabelangerfungi: Yup, big spike in building nodes :)22:27
pabelanger102 ATM22:27
pabelanger144 :)22:27
fungiso the jjb job updates reregistering in gearman triggering job description updates from zuul back to jenkins could very well have explained the performance roadblock22:27
clarkbfungi: yes you can pull it up in your web browser22:28
*** annegentle has quit IRC22:28
clarkbfungi: it looks all kosher22:28
*** lascii is now known as alaski22:28
fungiclarkb: oh, i know. i mean, i did22:28
fungijust curious if you looked at the issuer/expiration22:28
funginot concerned, merely amused22:29
*** [1]Thelo has joined #openstack-infra22:29
pabelanger385 nodes building, clouds must love us ATM22:30
fungipabelanger: we are the definition of "thundering herd"22:30
*** jamesmcarthur has quit IRC22:30
clarkbfungi: oh you mean they went and did a free cert or something22:30
pabelangerfungi: indeed22:30
fungiclarkb: they've got it set up using letsencrypt22:31
fungiso 3-month expiration on the cert22:31
*** asettle has joined #openstack-infra22:31
fungiin theory they've got something set up to auto-update from le in a timely fashion22:31
fungii'm just curious to see if it ends up working22:31
*** Thelo has quit IRC22:31
*** [1]Thelo is now known as Thelo22:31
fungithe update process i mean22:32
*** asettle has quit IRC22:33
clarkbfungi: once that is in I can attempt to make a mirror using my shade-launch-node script changes iwth unreleased shade22:33
*** gildub has joined #openstack-infra22:33
dbrycenclarkb: OK, what solved the issue was actually installing pbr. i guess i just have to pre-install it now but it was nice before when it would auto-install for me (when setup_requires just required "pbr" and didnt need a specific version)22:33
*** thorst has quit IRC22:35
pabelangerSo, I just stumbled across: could be a cool grafana dashboard for infracloud22:36
clarkbdbrycen: I think if your setuptools was newer the preinstall machinery would work22:36
*** sorantis has quit IRC22:38
dbrycenclarkb: i did try installing pip and upgrading setuptools and it still asked me to have the right version of pbr installed22:38
dbrycenclarkb: still happy that i can successfully create the tarball though :)22:38
*** gildub has quit IRC22:39
*** amitgandhinz has quit IRC22:39
*** yonglihe has quit IRC22:42
*** piet has joined #openstack-infra22:42
*** rhallisey has quit IRC22:42
*** devananda has quit IRC22:45
*** mriedem has quit IRC22:46
*** erlon has quit IRC22:46
*** abregman is now known as abregman|afk22:46
fungiclarkb: you're missing an amusing private e-mail thread unfortunately... in response to our request for an ipv4 /19 and ipv6 /48 for infra-cloud, someone (presumably in charge of provisioning) in hpe responded asking whether we were asking for private or public assignments for both of those22:49
clarkbprivate ipv6? what is this22:49
clarkbwe could just run our own ULA if we wanted to ...22:49
fungialso, like... why would we be asking you to assign private addresses for us? wouldn't we just pick some if that's what we wanted?22:50
*** jpr has quit IRC22:50
*** zz_dimtruck is now known as dimtruck22:51
*** ari- has quit IRC22:51
clarkbopenstack-ansible has a job for everything22:51
fungiand for everything else, there's ansible-openstack22:52
clarkbI am guessing there is a different repo for all of these things22:52
clarkblike rsyslog22:52
*** abregman|afk has quit IRC22:52
fungiclarkb: on a positive note, i suppose that e-mail exchange does indicate someone in hpe is actually reviewing the request. that's faster than i could have hoped22:54
pabelangerSo, as a recap for the week. wiki.o.o is looking pretty good right now. New spammers have stopped, and we are only getting 1 spammer account every few days now22:54
pabelanger show valid contribs by the community22:54
*** e0ne has quit IRC22:55
*** geekinutah has joined #openstack-infra22:55
pabelangerSo, next week I think we can discuss opening registrations again, and see what happens22:55
pabelangerand land some inflight patches giving admins more spamming tools22:55
anteayapabelanger: right now new accounts are cut off?22:55
pabelangeranteaya: yes22:55
anteayapabelanger: did the prior spam pages get deleted?22:55
fungias of ~monday if memory serves22:55
anteayaor is that work to be done still22:56
pabelangerwe should be able to fix openid accounts, but need to enable some settings22:56
fungi(the disablement of account creation i mean)22:56
pleia2pabelanger: thanks for the update22:56
*** Sukhdev has quit IRC22:56
pabelangeranteaya: no, we still need to automate that.  I believe 287232 will help with that22:56
fungiyes, i'm surprised this has stymied them22:56
*** hdd has quit IRC22:56
anteayapabelanger: awesome, thank you22:57
anteayaand as pleia2 says, thanks for the update22:57
pleia2btw, on the Ubuntu wiki, they created a team in launchpad of users authorized to edit the wiki and have restricted edits to that team22:57
pleia2(did this yesterday)22:57
pabelangerI think if we set: $wgOpenIDUseEmailAsNickname for openid, new accounts will be created in mediawiki while having the createaccount flag disabled22:57
pabelangerbut we need to test that22:57
pleia2which makes me worry they aren't really tackling the "lp has tons of spam accounts" problem22:57
fungiwow, so ubuntu has basically closed down the public editing of their wiki without directly requesting permission?22:58
anteayapleia2: well does lp have any developers left?22:58
pabelangerpleia2: Ya, I would like to look into migrating to, but not sure what is needed22:58
*** tiswanso has quit IRC22:58
fungii hear they've re-grown a dev team, but that doesn't necessarily mean their openid policing is well-staffed22:58
*** devananda has joined #openstack-infra22:58
pleia2anteaya: they moved some people over there22:58
pleia2fungi: yeah22:58
fungiespecially since and are entirely separate systems22:59
anteayawell yay dev team22:59
pleia2yeah, IS handles operations22:59 is an authentication/identification system which happens to have a domain name in common with launchpad.net22:59
pleia2my very late lunch has arrived, so attending to that for a bit22:59
anteayapleia2: enjoy lunch22:59
fungior at least that's how i've heard it explained in the past23:00
*** geekinut1h has joined #openstack-infra23:00
*** geekinutah has quit IRC23:00
anteayaif we are doing updates, here's mine, I tried to create a hound test server and can't seem to get puppet to get hound to run: this is the command I try and the pp file:
anteayathe command completes but there is no hound proces, no /home/hound dir and no installation of go23:01
openstackgerritPaul Belanger proposed openstack-infra/puppet-mediawiki: Set wgOpenIDUseEmailAsNickname True by default
anteayaI'm at a loss as to what to do next23:02
pabelangerpleia2: fungi: So, untested ^ but should allow openid account to properly connect on first time auth.23:02
pabelangernew users will have usernames based on email address23:02
pabelangerif openid is missing it23:02
EmilienMfungi: hi, I don't know if you've time to look at but just to make sure it's in your radar23:02
*** Sukhdev has joined #openstack-infra23:02
*** geekinutah has joined #openstack-infra23:05
*** edmondsw has quit IRC23:05
*** nwkarste_ has quit IRC23:05
anteayaI think I've had as much fun as I can take for the day23:05
anteayaI think I'll have an evening23:05
anteayahave a good weekend everyone23:06
*** Sukhdev has quit IRC23:06
*** geekinut1h has quit IRC23:06
* mnaser is curious about the big uptick in test nodes all of a sudden23:07
clarkbmnaser: zuul was taking a break while we ddos'd it with jenkins updates and now the ddos is over so it is spawning a bunch of work to catch back up again23:07
mnaserah makes sense23:07
mnaseri thought it was something like23:08
mnaser~5pm EST on a friday, lets spawn everything and break everything! :p23:08
mordredclarkb: hey - I had stepped away a bit - did we do python->ubuntu-trusty?23:08
mnaseri still see a bunch of error nodes still so whenever anyone is free to pass on the logs for these failures, id appreciate it23:08
clarkbmordred: yes, (though jenkins updates are still in progress)23:09
mordredcool. I'll go see what I can find for mnaser23:09
clarkbmnaser: let me see what I can do23:09
*** bauwser is now known as bauzas23:09
mordredoh. or clarkb can23:09
mordredand I can work on releasing shade23:09
mnaseri mean i was meaning to bug infra team every few days so we can investigate a few of them, instead of every single time one happens :-p23:09
*** roxanagh_ has quit IRC23:09
fungimordred: i think the jenkins masters are having to get their job updates one at a time to keep from bogging down zuul's geard to the point where job reregistrations are all it's doing23:10
mordredmnaser: I believe we enjoy your proactive nature23:10
openstackgerritK Jonathan Harker proposed openstack-infra/zuul: Teach zuul-cloner about ZUUL_REFNAME
mnaserit'd be nice if these logs were accessible to us but I can imagine there's a few (good) reasons they're not23:10
fungimnaser: mostly it's that there are still some libraries we don't trust to not leak random credentials into debug logs23:11
fungiso we tend to go over them carefully before handing them out, just to be safe23:11
mnaserthat's completely understandable.  better safe than sorry23:11
mordredfungi: I do believe we're actually safe now23:11
fungithe situation's improving though23:11
fungii still have hopes we're not far from being able to stuff all our service logs into a central publishing location23:12
openstackgerritMerged openstack-infra/project-config: Add libjpeg to bindep-fallback.txt
clarkbmnaser: there is a few of them like that one23:12
mordredfungi: or, hrm. once we update to shade, I'll be comfortable with it23:12
clarkbmnaser: ssh timeouts, I can get you all the uuids and tracebacks for the ssh timouts if you want23:12
clarkbmnaser: there is also another class of subnode failures I need to look at23:12
mordredfungi: because we'll be using keystoneauth Sesison at that point for everything, and that goes to great pains to not log secrets23:12
pabelangerclarkb: fungi: mordred so far I am only seeing libjpeg failures for ubuntu-trusty23:12
mordredthat's great!23:13
mordredpabelanger: do I want to ask what we need libjpeg for?23:13
pabelangerbut we have a patch in flight for that23:13
mordredpabelanger: I'm guessing somehting sphinx related?23:13
mnaserclarkb: new images were uploaded after the zuul ddos possibly?23:13
pabelangermordred: Ya, pillow it seems23:13
mordredpabelanger: lovely23:13
pabelangerthere are flags to disable it23:13
pabelangerbut unsure who is using it23:13
mordredpabelanger: adding it seems safe enough23:14
*** geekinutah has quit IRC23:14
clarkbmnaser: yes we just did it23:14
mnaserim pretty sure we're ddosing our own glance-api23:15
clarkbI mean not relaly but cool23:15
mnaserbecause there's just so many nodes hitting it at once23:15
*** e0ne has joined #openstack-infra23:15
clarkbmnaser: is the other class of issues which appears to be a bug in nodepool's scheduling23:15
clarkbwe should count multinode envs as X hosts not 123:16
mnaserand glance-api is located at a single az, so it has to go between the local latency link between the two datacenters23:16
*** angdraug has quit IRC23:16
mnaserso basically with most nodes being deployed at az-1 and glance-api at az-2, the bottleneck is the link connecting both23:16
clarkbmnaser: ah, would it help if we deployed to az2 only?23:16
mnaserglance-api with caching is the solution in order for us to avoid going over the link multiple times23:16
mnaserprobably not, overall capacity is smaller there so you might see more failures.  but glance-api cache in az-1 will prevent this23:17
mnaserwe'll have to pull it once instead of N times23:17
pabelangerso far octavia seems unhappy with libjpeg in the gate23:18
clarkbpabelanger: the change to fix that isn't on any images yet23:19
clarkbcurl should be mostly out there though23:19
clarkbcouple more regions for curl23:19
pabelangerjust scrolling zuul.o.o/status looking at failures23:19
clarkbfungi: any idea how we missed libjpeg-dev?23:19
clarkbit will be neat to see projects switch over to other-requirements and explicitly declare their deps23:20
pabelangerlikely small sample size, we did the TC approved projects first, plus ansible and puppet23:20
pabelangernone seemed to need it23:20
clarkbonce the change to update the osic clouds.yaml hits the puppetmaster I will work on booting a mirror there23:21
fungiclarkb: my best guess is that rackspace preinstalled it onto their images so it wasn't in our puppet, but none of the hundred or so projects/branches/jobs we tested experimentally relied on that package being present23:21
*** piet has quit IRC23:21
fungiclarkb: the other possibility is that someone added it to our puppet after i created the bindep fallback list and reviewers missed asking the author to write an additional change to update the fallback23:22
fungiclarkb: looks like you added it for pillow in october
fungii probably should have done a fresh pass through our puppet to try to catch any which slipped through the cracks in recent months23:24
clarkbya doesn't surprise me it slipped through the cracks when packages were added more confused how octavia and ironic didn't catch it23:25
*** sdake has quit IRC23:25
clarkbbut I guess we didn't test those?23:25
openstackgerritMerged openstack-infra/system-config: Verify SSL with OSIC
fungigood question, i thought ironic was on my test list but perhaps not23:26
fungialso looking at git history, we're coming up on a year since i put together the first vestiges of that fallback list. wow (though i know i refreshed it sometime late last year, was obviously still before 230175 merged)23:27
*** roxanaghe has joined #openstack-infra23:27
fungiclarkb: that same change also introduces zlib1g-dev/zlib-devel which is not in the fallback list either23:28
fungiis it also needed?23:28
clarkb Idon't think we have seen jobs fail due to it missing23:28
fungithe change identifies both as build-time deps of pillow 3.0.023:29
*** xyang1 has quit IRC23:29
clarkbah could be that we just fail on the first on which is jpeg23:29
clarkband that once that is in zlib will be the fail23:29
pabelangerfungi: Ah, good catch23:30
clarkbfqdn for osic mirror will be mirror.regionone.osic-cloud1.openstack.org23:31
*** Sukhdev has joined #openstack-infra23:32
clarkber thats not right23:33
* clarkb has to figure out what value goes in for osic-cloud123:33
clarkbdoes that map to the cloud: foo value?23:34
*** ccrouch1 has joined #openstack-infra23:35
clarkbappears to23:35
*** thorst has joined #openstack-infra23:35
clarkbwhich means I need to update things23:35
*** Sukhdev has quit IRC23:37
*** ccrouch has quit IRC23:37
openstackgerritClark Boylan proposed openstack-infra/system-config: Use osic-cloud1
clarkbthats part of it23:38
*** fawadkhaliq has quit IRC23:38
*** fawadkhaliq has joined #openstack-infra23:39
*** annegentle has joined #openstack-infra23:39
openstackgerritClark Boylan proposed openstack-infra/project-config: Use OSIC with nodepool
clarkband thats the other part23:39
jeblairclarkb: yes, it's the oscc "cloud" name, so whatever we name it in clouds.yaml23:40
clarkbI guess I need to update the depends on23:41
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Be more specific in logging timeout exceptions
openstackgerritClark Boylan proposed openstack-infra/project-config: Use OSIC with nodepool
mordredthat should fix ianw's comments and the test failures23:42
clarkbthere we go23:42
*** Jeffrey4l has joined #openstack-infra23:43
*** sridhar_ram has quit IRC23:43
*** thorst has quit IRC23:44
*** kingia_ has joined #openstack-infra23:44
mordredclarkb: wow. pool: 'GATEWAY_NET'23:45
NobodyCamHey clarkb how are you doing today. just checking in to see if it worth firing a recheck off. I've been working with jroll on the libjpeg fall back issue. seems to have landed but jim sais something about a new image?23:46
fungimordred: it's the network where you find your gateway!23:46
NobodyCamhey hey mordred :) happy friday23:46
mordredmorning NobodyCam !23:46
NobodyCamand to fungi too23:46
jeblairclarkb: are the osic ssl things fixed?23:47
*** kingia has quit IRC23:47
fungiNobodyCam: i think there were image updates in progress to address that? i wasn't at a computer when the conversation started and sort of skimmed so far, though it's possible zlib1g-dev may also be needed since it was added in the same patch which brought in libjpeg-dev to puppet23:47
mordredinfra-root: eveybody ready for me to fire off a shade release?23:48
clarkbjeblair: yes they are23:48
fungijeblair: yeah, they issued a new cert (valid) and updated the keystone catalog23:48
clarkbI am booting the mirror now23:48
fungijeblair: though it's a letsencrypt cert, so only valid for 3 months... i'm curious to see if they set up the auto-updating for it correctly23:48
jeblairclarkb: what's the hostname you're using?23:49
clarkbjeblair: the one from above, mirror.regionone.osic-cloud1.openstack.org23:49
jeblairyeah that looks right23:49
fungiNobodyCam: i'm in the process of auditing recent updates to the packages list in puppet to see if there's anything else that might have slipped through the cracks besides those23:50
NobodyCamawesome thank you :) I'll hold off on the recheck for a while :)23:51
*** x00350071_ has quit IRC23:51
*** x00350071 has joined #openstack-infra23:51
*** salv-orl_ has joined #openstack-infra23:52
*** annegentle has quit IRC23:52
clarkbhrm might run into the same volume thing in osic where vdb is config drive but cinder and nova say it is the volume23:54
clarkbits ok we have a workaround, but it is annoying that config drive and cinder are basically incompatible23:55
*** salv-orlando has quit IRC23:55
nibalizermordred: im wonder why we would release on friday afternoon23:57
fungiclarkb: admittedly this is a nova-side issue, it just happens that the information cinder is relying on from nova is a lie (or perhaps at best a terrible guess)23:58
mordrednibalizer: because FUN!23:58
nibalizerfor who?23:58
mordrednibalizer: for you23:58
mordredmr. west coast23:58
* mordred waves23:58
* nibalizer waves back23:58
*** annegentle has joined #openstack-infra23:58
bkeroYou'd release because neither of you want to have a weekend :)23:59
mordrednibalizer: but also we're at slow gate anyway, so we can probably release, restart nodepool, watch for carnage, and rollback if needed in like 20 minutes23:59
mordrednibalizer: however, I could also wait23:59
clarkbfungi: yes but it is still cinder's api23:59
clarkbfungi: regardless of where hte info comes from if you have it built into your api well I think it hsould be correct23:59
nibalizerwe are not the only users mordred23:59
mordrednibalizer: oh - so, there's this other thing23:59

Generated by 2.14.0 by Marius Gedminas - find it at!