| * krotscheck_dcm goes to take a look | 00:00 | |
| jeblair | krotscheck_dcm: we like efficient | 00:00 |
|---|---|---|
| avarner | Anyway, I have a bug to report for Etherpad - ErrorId: 2ihxGNdS4W6IFmxvfBe5 | 00:00 |
| fungi | avarner: yes | 00:00 |
| avarner | NS_ERROR_SOCKET_CREATE_FAILED: in https://etherpad.openstack.org/socket.io/socket.io.js at line 5325' | 00:00 |
| avarner | fungi, thanks | 00:00 |
| avarner | Just started today - I actually accessed etherpad successfully this morning | 00:00 |
| fungi | avarner: what were you doing when you got that error, and is it repeatable? | 00:00 |
| clarkb | and what etherpad | 00:01 |
| avarner | Yes, happens every time I open an etherpad page | 00:01 |
| avarner | e.g. https://etherpad.openstack.org/p/glance-mitaka-virtual-mid-cycle | 00:01 |
| krotscheck_dcm | jeblair: Ok, looks like I need to build a client to work with live data. Gimme a bit. | 00:01 |
| fungi | avarner: it's certainly taking a while to load in my browser | 00:01 |
| anteaya | fungi: mine too, trying in ff and chrome | 00:01 |
| fungi | avarner: are you able to successfully load other pads on that site? | 00:02 |
| nibalizer | clarkb: if you're not itching to create patches to nodepool to add osic then maybee we should be landing the infracloud patches | 00:02 |
| avarner | I can't seem to load any pads | 00:02 |
| anteaya | avarner: what others are you trying? | 00:02 |
| clarkb | nibalizer: ya I would start with infracloud | 00:02 |
| *** Sukhdev has quit IRC | 00:02 | |
| avarner | anteaya, https://etherpad.openstack.org/p/aaaa | 00:02 |
| avarner | err wait | 00:02 |
| avarner | that actually hangs | 00:02 |
| clarkb | nibalizer: I am likly going to start by checking things, getting clients working, booting test VMs and running tests etc | 00:03 |
| jeblair | i would like to land one or both asap on account of everything being really slow :) | 00:03 |
| *** ybathia has quit IRC | 00:03 | |
| clarkb | its possible the daemon went away | 00:03 |
| *** FallenPegasus has quit IRC | 00:03 | |
| avarner | anteaya, ok, this one has the same error: https://etherpad.openstack.org/p/a | 00:03 |
| clarkb | and what we do have working is just apache serving static assets | 00:03 |
| nibalizer | clarkb: okay cool | 00:03 |
| *** ayoung_ has quit IRC | 00:03 | |
| *** austin81 has left #openstack-infra | 00:03 | |
| anteaya | load is spiking right now: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=113&rra_id=all | 00:03 |
| clarkb | I have logged into horizon successfully | 00:04 |
| *** SumitNaiksatam has quit IRC | 00:04 | |
| anteaya | I can't create a new etherpad | 00:04 |
| fungi | i see a node.js process on etherpad.o.o running since january 20 | 00:04 |
| clarkb | there goes that theory | 00:04 |
| *** rbrndt_ has quit IRC | 00:05 | |
| clarkb | openstackci look sgood from a quota perspective | 00:05 |
| *** EricGonczer_ has joined #openstack-infra | 00:05 | |
| anteaya | I have a 502 from etherpad from trying to open the glance mitaka pad in ff: http://paste.openstack.org/show/486033/ | 00:05 |
| fungi | and /var/log/eplite/error.log last updated on the 20th | 00:06 |
| anteaya | I can open this one in ff: https://etherpad.openstack.org/p/a | 00:06 |
| clarkb | openstackjenkins will need work | 00:06 |
| anteaya | I can create a new pad in chrome: https://etherpad.openstack.org/p/UP5gJ723RR | 00:06 |
| clarkb | well maybe, there are only 5 floating IPs but maybe we get public by default. Time to boot something | 00:06 |
| fungi | also /var/log/eplite/access.log is definitely logging requests for pads | 00:06 |
| anteaya | and the glance mitaka one opens in chrome | 00:07 |
| fungi | oh, that pad did eventually open for me | 00:07 |
| anteaya | yeah me too, in chrome, 502 in firefox | 00:07 |
| fungi | i'm testing on a very slow machine, btw. firefox on a 0.5gb ram netbook | 00:07 |
| *** ayoung has quit IRC | 00:07 | |
| fungi | so that might explain the slowness for me | 00:08 |
| *** dalgaaf has quit IRC | 00:08 | |
| fungi | i see 6 other participants on the glance-mitaka-virtual-mid-cycle pad besides me | 00:08 |
| anteaya | jeblair: mtreinish replied to https://review.openstack.org/#/c/276318/ | 00:08 |
| anteaya | fungi: one is me | 00:09 |
| anteaya | I forgot to sign in | 00:09 |
| anteaya | no it was very slow to open for me | 00:09 |
| fungi | checking cacti next | 00:09 |
| clarkb | osic is a very hpcloudy experience as far as networking goes | 00:09 |
| * clarkb creates a network | 00:09 | |
| fungi | there is another runaway abiword process | 00:10 |
| *** annegentle has quit IRC | 00:10 | |
| fungi | two actually | 00:10 |
| fungi | i've killed them both now | 00:10 |
| fungi | but they keep coming back | 00:10 |
| fungi | did we shoot down the crazy notion of symlinking /usr/bin/abiword to /bin/true on the server? | 00:11 |
| *** avarner has left #openstack-infra | 00:11 | |
| anteaya | clarkb: fun stuff | 00:11 |
| anteaya | I don't recall that discussion | 00:12 |
| anteaya | the symlinking one | 00:12 |
| *** sabeen1 has quit IRC | 00:12 | |
| fungi | cacti doesn't show anything odd for the etherpad server | 00:13 |
| fungi | though we are hovering around 2k open connections | 00:13 |
| jeblair | anteaya, mtreinish: thanks; replied! | 00:14 |
| *** moravec has joined #openstack-infra | 00:14 | |
| anteaya | jeblair: wonderful | 00:14 |
| fungi | that doesn't look out of line with the connection count this time on previous recent weeks | 00:14 |
| fungi | hrm, avarner disappeared on us | 00:15 |
| jeblair | nibalizer: i think you've failed to trick fungi into breaking the tie, so i'm going to. :) | 00:15 |
| nibalizer | jeblair: ok | 00:15 |
| * fungi is too easily distracted by shiny new broken, sorry :/ | 00:15 | |
| jeblair | nibalizer: still have suggestions on https://review.openstack.org/275491 tho | 00:16 |
| nibalizer | jeblair: kk | 00:16 |
| fungi | i need to curb the habit of looking into every last broken thing | 00:16 |
| *** nelsnelson has joined #openstack-infra | 00:16 | |
| fungi | speaking of which, the updates to the release announcement jobs don't seem to have propagated, so i'm going to manually rerun jjb on our jenkins masters again | 00:16 |
| mgagne | jeblair what's the path to secure github credentials used by infra? I see 2 default values in jeepyb | 00:17 |
| mgagne | /etc/github/github-projects.secure.config and /etc/github/github.secure.config | 00:17 |
| *** moravec has quit IRC | 00:18 | |
| *** nelsnels_ has quit IRC | 00:18 | |
| crinkle | for infra-cloud, there are still a number of patches that will be easier to get in before it goes into nodepool than after | 00:18 |
| dims | AJaeger : getting tired of permutations/combinations that i am trying with https://review.openstack.org/#/c/276313/ - any guidance please? At the moment "tox -e zuul" works locally and failed in CI | 00:18 |
| crinkle | the big thing is that infra-cloud is still kilo | 00:19 |
| dims | oops i meant anteaya ^^^ :) | 00:19 |
| *** sflanigan has joined #openstack-infra | 00:19 | |
| anteaya | dims: I have the patch open and I'm just about to eat dinner | 00:19 |
| crinkle | and the servers are going to be moved and have networking changes at some point in the future that i don't know that will cause much downtime | 00:19 |
| anteaya | dims: will get to it after food | 00:19 |
| anteaya | crinkle: can you prioritize the patches somehow? | 00:20 |
| anteaya | crinkle: I'm happy to review them in the order you want them merged, if I can figure that out | 00:20 |
| jeblair | jhesketh: when you are around, it looks like connections broke reconfiguration. i wrote a test case in https://review.openstack.org/276395 but have too many things on my plate to try to unwind it right now -- i did manage to do a little debugging and left notes in the change. i did verify that test case passes pre-connections. | 00:22 |
| nibalizer | crinkle: my thought is that I'm kindof out of testing I can do without connecting it to nodepool | 00:22 |
| *** thorst has joined #openstack-infra | 00:22 | |
| nibalizer | so having data from nodepool using infracloud is the next stup | 00:23 |
| nibalizer | plus our users could sure use the extra nodes | 00:23 |
| crinkle | anteaya: i'm more intending to bring it up as a "are we sure we want it to go into production yet?" question rather than a call for reviews of specific patches | 00:23 |
| *** thorst has quit IRC | 00:23 | |
| crinkle | anteaya: i would really like all of topic:infra-cloud that isn't in merge conflict or workflow-1 to be reviewed | 00:23 |
| *** thorst has joined #openstack-infra | 00:24 | |
| crinkle | nibalizer: that sounds good to me, i just want everyone involved to be aware that there are still large-ish changes to come and that there will certainly be downtime coming Soon, and factor that in to the decision | 00:25 |
| nibalizer | yup! | 00:26 |
| clarkb | thats interseting their image has cloud archive enabled by default | 00:27 |
| clarkb | I wonder if canonical has started doing that | 00:27 |
| *** Sukhdev has joined #openstack-infra | 00:27 | |
| dims | anteaya : thanks! | 00:28 |
| *** dingyichen has quit IRC | 00:30 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/project-config: Use omfracloud in nodepool https://review.openstack.org/275491 | 00:30 |
| *** thorst has quit IRC | 00:33 | |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Bump monotonic to >= 0.6 https://review.openstack.org/276514 | 00:34 |
| nibalizer | jeblair: i've updated the flavor and min number | 00:34 |
| nibalizer | and created the 'nodepool' flavor in IC | 00:34 |
| jeblair | nibalizer: +2 | 00:34 |
| anteaya | crinkle: fair enough, thank you | 00:35 |
| *** diana_clarke has quit IRC | 00:35 | |
| *** doug-fis_ has joined #openstack-infra | 00:36 | |
| *** dingyichen has joined #openstack-infra | 00:36 | |
| *** kzaitsev_mb has quit IRC | 00:36 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix typo in afs admin group https://review.openstack.org/276494 | 00:37 |
| anteaya | dims: who is we in your commit message? | 00:37 |
| anteaya | dims: we nova, we release team, we oslo | 00:38 |
| jeblair | royal we | 00:38 |
| anteaya | dims: you wear so many hats | 00:38 |
| anteaya | openstack we | 00:38 |
| anteaya | jeblair: btw is there any syntax symbol that is called abbot? | 00:38 |
| anteaya | in my dream last night you were lecturing in the other room (I was making candles) | 00:39 |
| anteaya | and you talked about something called abbot which I took to be some form of font symbol | 00:39 |
| *** dims_ has joined #openstack-infra | 00:39 | |
| *** doug-fish has quit IRC | 00:40 | |
| *** dims has quit IRC | 00:40 | |
| jeblair | anteaya: it does not ring a bell... | 00:40 |
| *** moravec has joined #openstack-infra | 00:40 | |
| anteaya | okay I'll listen closer tonight | 00:41 |
| anteaya | if you happen to give the same or a follow up lecture | 00:41 |
| jeblair | anteaya: and i'm sorry dream-me subjected you to a typography lecture; but at least you didn't have to be in the same room | 00:41 |
| *** sputnik13 has quit IRC | 00:41 | |
| anteaya | you had an attentive audience | 00:41 |
| anteaya | no it was good | 00:41 |
| anteaya | I had lots of room for my candle making | 00:41 |
| anteaya | I wasn't interrupted | 00:42 |
| jeblair | hopefully i was on point | 00:42 |
| anteaya | you seemed to be | 00:42 |
| jeblair | and wasn't out of sorts | 00:42 |
| anteaya | I didn't get the sense you were | 00:42 |
| anteaya | I think you were enjoying yourself | 00:42 |
| anteaya | but abbot stood out as a word I needed to remember | 00:43 |
| anteaya | don't know why yet | 00:43 |
| openstackgerrit | Tim Burke proposed openstack-dev/hacking: Add H203 to check that assertIs(Not)None is used https://review.openstack.org/276517 | 00:43 |
| * fungi only knows of abbots as clergy overseeing an abbey, or possibly lou costello's comedic partner | 00:43 | |
| jeblair | the madness continues in https://review.openstack.org/276517 | 00:43 |
| anteaya | fungi: that's what the internet confirms | 00:44 |
| anteaya | fungi: I still don't know why it was important for me to remember | 00:44 |
| *** angdraug has quit IRC | 00:45 | |
| *** krotscheck_dcm has quit IRC | 00:45 | |
| anteaya | jeblair: I'm so not interested in that fight | 00:46 |
| * anteaya tries to understand why jenkins is unhappy with dims patch | 00:46 | |
| *** weshay_xchat has joined #openstack-infra | 00:48 | |
| anteaya | dims_: so looking here: http://logs.openstack.org/13/276313/8/check/gate-project-config-layout/de9e8b1/console.html#_2016-02-04_23_47_07_867 | 00:48 |
| anteaya | in zuul.layout config there is the mention of two jobs that right now don't line up with definitions | 00:49 |
| * anteaya looks to see what is missing | 00:49 | |
| anteaya | dims_: in jenkins/jobs/projects.yaml you need to add periodic-python-jobs-with-oslo-master to nova | 00:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add job mutex support https://review.openstack.org/275974 | 00:51 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Try to make test_idle less racy https://review.openstack.org/276521 | 00:51 |
| clarkb | nova unitests http://paste.openstack.org/show/486034/ now to find something to compare that to | 00:52 |
| anteaya | dims_: I've put that in a comment on the patch | 00:52 |
| jeblair | clarkb: can you aprv https://review.openstack.org/276521 ? | 00:52 |
| clarkb | Ran: 14667 tests in 367.0000 sec. is rax so that is actually pretty decent | 00:53 |
| clarkb | jeblair: looking | 00:53 |
| *** moravec has quit IRC | 00:53 | |
| jeblair | no one has beat rax yet :) | 00:53 |
| clarkb | jeblair: osic is 40 seconds quicker | 00:54 |
| clarkb | with only 4vcpu | 00:54 |
| *** Daisy has joined #openstack-infra | 00:54 | |
| jeblair | ya | 00:54 |
| *** salv-orl_ has quit IRC | 00:54 | |
| *** moravec has joined #openstack-infra | 00:55 | |
| mordred | clarkb: what's osic? | 00:55 |
| clarkb | mordred: openstack innovation center | 00:56 |
| mgagne | openstack infra-cloud? | 00:56 |
| fungi | mordred: raxtel | 00:56 |
| mgagne | oh :P | 00:56 |
| clarkb | joint effort by intel and rax | 00:56 |
| mordred | clarkb: ah | 00:56 |
| mgagne | what are they running? any specs? | 00:56 |
| mordred | clarkb: are we going to nodepool it? | 00:56 |
| clarkb | I need to spend more time with the different flavors and tempest and figure out what our flavor/quota mapping should be | 00:56 |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Experimental periodic jobs for testing against oslo.* from master https://review.openstack.org/276313 | 00:56 |
| clarkb | mgagne: I don't know much just got credentials | 00:56 |
| clarkb | mordred: thats the plan | 00:56 |
| mordred | clarkb: neat! | 00:56 |
| mordred | clarkb: that makes me excited | 00:56 |
| clarkb | I only have 5 fips right now | 00:57 |
| clarkb | so that will be first order of business | 00:57 |
| mordred | clarkb: which networking does it do? the one we like or the one we tolerate and lke to mock? | 00:57 |
| mordred | ah | 00:57 |
| clarkb | mordred: its literally hpcloud | 00:57 |
| mordred | question answered | 00:57 |
| mordred | clarkb: sigh | 00:57 |
| clarkb | I just did everything for it that I did way back when on hpcloud | 00:57 |
| anteaya | how tiresome | 00:57 |
| mordred | too bad they made those choices | 00:57 |
| clarkb | mordred: in fairness to hpcloud and osic these are the semi defacto defaults as advertised by upstream | 00:58 |
| *** Daisy_ has joined #openstack-infra | 00:58 | |
| mgagne | ^ =) | 00:58 |
| anteaya | that would be why they made those choices | 00:58 |
| clarkb | I hva ea feeling that if we fix it upstream that others will follow | 00:58 |
| mordred | indeed | 00:58 |
| mgagne | and public IP cost? | 00:58 |
| mordred | mgagne: it's the same | 00:58 |
| mordred | mgagne: floating ips vs. provider network has zero difference in public ip cost | 00:59 |
| mordred | if your workload needs a public ip, it needs a public ip | 00:59 |
| *** Daisy__ has joined #openstack-infra | 00:59 | |
| mordred | whether you get it via nat or via direct routing is orthogonal | 00:59 |
| mgagne | mordred with provider, you always get a public IP no? and FIP, it's limited and not by default? | 00:59 |
| mordred | no | 00:59 |
| mordred | if you have a provider network and a tenant private network | 01:00 |
| mgagne | mordred what are the specifics? | 01:00 |
| mordred | you can choose to boot a VM on the private network | 01:00 |
| mordred | or you can choose to boot in on the public network | 01:00 |
| mgagne | mordred how many IPs do you have on the public network? | 01:00 |
| *** EricGonczer_ has quit IRC | 01:01 | |
| *** Daisy has quit IRC | 01:01 | |
| *** Daisy__ is now known as Daisy | 01:01 | |
| *** chlong has joined #openstack-infra | 01:01 | |
| mordred | the 'best' design, imo, is having a shared public provider network, the ability for users to create private networks, and the ability for users to create routers that attach to the shared public network and allow them to get a NATed IP from the shared public network | 01:01 |
| *** r1chardj0n3s has left #openstack-infra | 01:01 | |
| mordred | this covers all three use-case | 01:02 |
| mordred | cases | 01:02 |
| *** noorul has joined #openstack-infra | 01:02 | |
| noorul | Hi | 01:02 |
| mordred | the main problem with floating-ip only clouds is that they do not allow a user to express "I want this vm to be on the public network and I have no need of private networking" | 01:02 |
| mgagne | mordred my point is that FIP can help (provider) to artificially limit the number of public IPs you (the user) can use and reduce cost. | 01:02 |
| *** Daisy_ has quit IRC | 01:02 | |
| mordred | not really | 01:02 |
| noorul | Will it be easy to integrate zuul with some other review system like bitbucket? | 01:02 |
| anteaya | jeblair: mmedvede added a comment to the merged zuul fix the memory leak due to triggers patch: https://review.openstack.org/#/c/275483/1 | 01:03 |
| *** pahuang has quit IRC | 01:03 | |
| mordred | mgagne: I hear that argument, but on our clouds with floating ips we create a floating ip for every vm we boot | 01:03 |
| anteaya | noorul: setting up any of our tools with your environment always requires some work | 01:03 |
| mordred | mgagne: it's just a few extra API calls and then the IP is natted | 01:03 |
| mgagne | mordred but you only have 5 =) | 01:03 |
| mordred | mgagne: we have no usable clouds with only 5 floating ips | 01:04 |
| anteaya | noorul: I don't believe anyone would call working with our tools easy, however many feel they are worth it in that they accomplish the task | 01:04 |
| mordred | mgagne: that's problem #1 to solve with our account there | 01:04 |
| mgagne | "clarkb: I only have 5 fips right now" | 01:04 |
| mordred | yes | 01:04 |
| mordred | it's a bug in our account | 01:04 |
| noorul | I know that but any clue whether it is really worth attempting | 01:04 |
| anteaya | noorul: I have no idea | 01:04 |
| mordred | the cloud is unusable with that many FIPs for us | 01:04 |
| mordred | we need the number of public ips we need | 01:04 |
| mordred | the mechanism we use to get them does not change that need | 01:04 |
| anteaya | noorul: zuul was written specifically for our use case | 01:04 |
| mgagne | I don't argue about that need =) | 01:04 |
| mordred | mgagne: :) | 01:05 |
| mordred | mgagne: I'm just saying that as a user I much prefer the way you give us IPs | 01:05 |
| mordred | mgagne: to the forced-floating-ip way | 01:05 |
| mordred | forced-floating-ip isn't really solvin a problem | 01:05 |
| anteaya | noorul: if you want to play with a push button set up of our tools, puppet-openstackci is as push button as we have | 01:05 |
| mgagne | mordred I don't think we both disagree on that side | 01:06 |
| mordred | :) | 01:06 |
| noorul | But my review system is Bitbucket | 01:06 |
| anteaya | noorul: http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/ | 01:06 |
| *** Jokke__ has joined #openstack-infra | 01:06 | |
| *** Jokke__ has quit IRC | 01:06 | |
| mgagne | can't jenkins be configured to use a bastion? | 01:06 |
| anteaya | this was created for our third party ci operators | 01:06 |
| anteaya | noorul: that is the best I have to offer | 01:06 |
| *** eil397 has quit IRC | 01:08 | |
| mordred | mgagne: we could redesign our system to do that | 01:08 |
| mordred | mgagne: but since we find that design to be completely broken | 01:08 |
| mordred | mgagne: we are very unlikley to do so | 01:08 |
| mgagne | mordred I agree public IP are very convenient | 01:09 |
| *** ayoung has joined #openstack-infra | 01:09 | |
| clarkb | fwiw we can ipv6 | 01:09 |
| mordred | clarkb: \0/ | 01:09 |
| clarkb | so the no IPs argument is :( | 01:09 |
| mgagne | :D | 01:09 |
| clarkb | mordred not on osic | 01:09 |
| mordred | oh | 01:09 |
| mordred | wow | 01:09 |
| mordred | that's even lamer | 01:09 |
| clarkb | but in general I mean | 01:09 |
| mordred | yes | 01:09 |
| mordred | we like ipv6 | 01:09 |
| mgagne | can't at internap =( | 01:09 |
| clarkb | we are capable of ipv6 | 01:09 |
| mordred | mgagne: you should ipv6! | 01:09 |
| clarkb | my house has ipv6 | 01:10 |
| clarkb | need toaster ip | 01:10 |
| mgagne | mordred I think it's one of our oldest item in the backlog | 01:10 |
| pabelanger | noorul: anteaya: You'd have to write a new zuul.source interface for bitbucket, assuming they streamed events like gerrit | 01:10 |
| anteaya | clarkb: to do what? | 01:10 |
| *** bhunter71 has joined #openstack-infra | 01:10 | |
| anteaya | clarkb: what do you want a toaster ip to do? | 01:10 |
| mgagne | mordred right after "Update Gerrit project description" in jeepyb project :D | 01:10 |
| clarkb | anteaya run openstack | 01:11 |
| noorul | pabelanger ok | 01:11 |
| anteaya | ha ha ha ha | 01:11 |
| clarkb | object storage for my bread | 01:11 |
| anteaya | virtual bread | 01:11 |
| mordred | mgagne: hahaha | 01:11 |
| anteaya | it can tell you about toasting it | 01:11 |
| anteaya | and how much you enjoyed it as you ate it | 01:11 |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/jeepyb: Add support for GitHub Enterprise https://review.openstack.org/276528 | 01:12 |
| prometheanfire | mordred: oh hi :D | 01:12 |
| prometheanfire | mordred: have time to look at a dib review? | 01:12 |
| *** pfalleno1 has quit IRC | 01:13 | |
| anteaya | I miss the old toasters, with the metal door that held the bread against the element | 01:14 |
| *** sridhar_ram has quit IRC | 01:15 | |
| *** pahuang has joined #openstack-infra | 01:15 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Use SSL for rabbitmq traffic https://review.openstack.org/276375 | 01:16 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 01:17 |
| anteaya | I need to sign off, my cold is getting better though, yay! | 01:18 |
| anteaya | g'night | 01:18 |
| *** Jeffrey4l has joined #openstack-infra | 01:18 | |
| *** Sukhdev has quit IRC | 01:18 | |
| *** baoli has joined #openstack-infra | 01:19 | |
| *** bhunter71 has quit IRC | 01:19 | |
| *** gildub has joined #openstack-infra | 01:20 | |
| *** baoli has quit IRC | 01:23 | |
| *** moravec has quit IRC | 01:23 | |
| *** ianychoi has quit IRC | 01:23 | |
| *** r-daneel has quit IRC | 01:24 | |
| *** moravec has joined #openstack-infra | 01:24 | |
| *** pfallenop has joined #openstack-infra | 01:25 | |
| *** thorst has joined #openstack-infra | 01:30 | |
| *** pfallenop has quit IRC | 01:31 | |
| *** krotscheck1 has joined #openstack-infra | 01:31 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Fixed backreference in regular expression https://review.openstack.org/276534 | 01:32 |
| krotscheck1 | jeblair, fungi: ^^ Typos will be the death of me. | 01:32 |
| krotscheck1 | Also, building a thick slave on a home internet connection is slooooowwww. | 01:32 |
| fungi | krotscheck1: clearly you need a thicknet connection | 01:33 |
| fungi | (10base5 fo evah!) | 01:33 |
| krotscheck1 | fungi: I'll have the governmnet give me a bigger tube. | 01:33 |
| *** mtanino has quit IRC | 01:34 | |
| *** chlong has quit IRC | 01:34 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 01:35 |
| *** chlong has joined #openstack-infra | 01:35 | |
| fungi | dhellmann: http://logs.openstack.org/7b/7bced8a36a82415f9075f3730897fab834fbeb4b/release/python-keystoneclient-announce-release/a7fb2d0,3/console.html | 01:36 |
| fungi | a winnar is yuo | 01:36 |
| fungi | in theory ttx has a message in the moderation queue for -announce | 01:36 |
| Clint | in practice it's a hypercube? | 01:37 |
| *** thorst has quit IRC | 01:38 | |
| fungi | Clint: yes. so i'm _not_ the only one perceiving in four spatial dimensions? | 01:38 |
| *** bhunter71 has joined #openstack-infra | 01:38 | |
| *** piet has joined #openstack-infra | 01:38 | |
| * Clint whistles innocently. | 01:38 | |
| *** sdake has quit IRC | 01:38 | |
| * krotscheck1 didn't get that memo. It was probably printed on two-dimensional paper. | 01:39 | |
| krotscheck1 | fungi: How've complaints been about the AFS mirrors since we got the vos release thing figured out? | 01:40 |
| fungi | krotscheck1: nonexistent | 01:40 |
| fungi | also when anyone complains, i ignore them. makes it much easier to deal with | 01:41 |
| Clint | krotscheck1: he said he wants you to complain more | 01:41 |
| *** savihou has quit IRC | 01:41 | |
| *** unicell has left #openstack-infra | 01:41 | |
| krotscheck1 | Clint: I'm not certain those words mean what you think they mean. | 01:41 |
| Clint | that is certainly possible | 01:41 |
| krotscheck1 | fungi: Ignoring them is a great way to be incredibly productive. | 01:41 |
| *** diana_clarke has joined #openstack-infra | 01:42 | |
| *** shashank_hegde has quit IRC | 01:42 | |
| *** moravec has quit IRC | 01:42 | |
| *** raginbajin has quit IRC | 01:42 | |
| *** arxcruz has quit IRC | 01:42 | |
| *** coreyob has joined #openstack-infra | 01:43 | |
| *** oomichi_away has quit IRC | 01:44 | |
| *** moravec has joined #openstack-infra | 01:45 | |
| coreyob | hey all, have a quick question about gate jobs in jenkins. I'm seeing slaves dying mid-run frequently on magnum functional test jobs. is it plausible that our tests are stressful enough to cause the jenkins agent to die? | 01:46 |
| *** raginbajin has joined #openstack-infra | 01:46 | |
| clarkb | coreyob: more likely that something breaks networking | 01:46 |
| fungi | coreyob: in our ci or a ci system you're running? | 01:46 |
| coreyob | your ci. e.g. https://jenkins02.openstack.org/job/gate-functional-dsvm-magnum-api/ | 01:47 |
| fungi | coreyob: and are you sure the job wasn't aborted in a dependent pipeline and restarted on a new worker because a change ahead of it started to fail? | 01:47 |
| clarkb | it is relatively common for a bad change or similar to nuke networking and cause problems | 01:47 |
| coreyob | all the builds that don't have a triggered-by section | 01:47 |
| coreyob | fungi I'm pretty sure. most of the changes we have aren't dependent | 01:47 |
| coreyob | but not entirely sure | 01:48 |
| fungi | coreyob: you really, really, really can't trust the jenkins stats for these | 01:48 |
| clarkb | also that | 01:48 |
| coreyob | what do you mean the jenkins stats? | 01:48 |
| fungi | trust only what gets reported back into the code reviews | 01:48 |
| coreyob | oh definitely. i mean zuul is still showing them as queued, but the functional jobs are 1-2 hours and some of our changes hve been queued in zuul for 8 hours now because they keep getting aborted | 01:49 |
| fungi | coreyob: zuul will abort jobs and restart them when gating, so that they don't run to completion unnecessarily and waste our quota | 01:49 |
| clarkb | coreyob: have an example change? | 01:50 |
| *** Thelo has joined #openstack-infra | 01:50 | |
| fungi | coreyob: and often nodepool is deleting the worker before jenkins has had a chance to clean it up | 01:50 |
| coreyob | clarkb 273362,3 | 01:50 |
| *** noorul` has joined #openstack-infra | 01:50 | |
| coreyob | i think this was the most recent run:} https://jenkins02.openstack.org/job/gate-functional-dsvm-magnum-api/245/consoleText | 01:50 |
| clarkb | that one is in check so shouldn't get aborted like gate | 01:51 |
| clarkb | it is likely something is breaking networking | 01:51 |
| clarkb | when that happens zuul reenques it | 01:51 |
| *** krotscheck1 has quit IRC | 01:51 | |
| Thelo | Hi all, on my CI, zuul crashed and I cannot find any ERROR in the log. Has anyone faced something like this? | 01:51 |
| *** armax has joined #openstack-infra | 01:51 | |
| *** doug-fis_ has quit IRC | 01:51 | |
| coreyob | clarkb yeah i've noticed the re-queue but it is really holding up our gate. do you know of any way we can debug that to figure out what is happening? | 01:51 |
| *** pfallenop has joined #openstack-infra | 01:51 | |
| clarkb | coreyob: run the job locally and check what happens to your networking | 01:52 |
| *** doug-fish has joined #openstack-infra | 01:52 | |
| fungi | http://logs.openstack.org/62/273362/3/gate/gate-functional-dsvm-magnum-api/f522640/console.html#_2016-02-04_04_36_03_656 ended on an exit code of 137 | 01:52 |
| Thelo | besides, I cannot restart it. | 01:52 |
| clarkb | Thelo: disk space? | 01:53 |
| coreyob | fungi yeah those go to completion and report as failed as expected | 01:53 |
| clarkb | Thelo: check the lock file too | 01:53 |
| fungi | 137 is a kill -9 usually? | 01:53 |
| clarkb | fungi: under timeout ? | 01:53 |
| *** doug-fis_ has joined #openstack-infra | 01:53 | |
| Thelo | where is that file? | 01:53 |
| fungi | possibly, still looking | 01:53 |
| coreyob | yeah that's what you get when the gate runs more than 2 hours | 01:53 |
| jeblair | fungi: https://review.openstack.org/276534 could use your eyes | 01:53 |
| coreyob | clarkb so basically we need to spin up an 8gb perf vm and see if we can reproduce? | 01:54 |
| clarkb | coreyob: I mean you can probably just do it on your laptop and see if your networking goes fubar | 01:55 |
| coreyob | I know the jobs work in general because sometimes they pass | 01:55 |
| clarkb | its typically pretty blatant when it happens | 01:55 |
| fungi | indeed, backreferences use \ not $ | 01:55 |
| clarkb | the one you linked looks like a zuul abort though | 01:55 |
| coreyob | are there zuul logs that show when it aborts something> | 01:55 |
| clarkb | except that it was in check | 01:55 |
| coreyob | it works fine sometimes. if we wait long enough it will pass or fail. which is why i think it might be something more like oom | 01:56 |
| *** moravec has quit IRC | 01:56 | |
| clarkb | yes we can check that except it shouldn't abort in check like that | 01:56 |
| Thelo | clarkb: you mean the pid file? if so, its gone | 01:56 |
| clarkb | Thelo: yes | 01:56 |
| clarkb | coreyob: check the syslogs for oom | 01:56 |
| coreyob | clarkb when the slave craps out like that I get 0 logs :( | 01:56 |
| clarkb | oh right | 01:56 |
| *** arxcruz has joined #openstack-infra | 01:57 | |
| clarkb | the next time that test runs we can hold the VM | 01:57 |
| *** doug-fish has quit IRC | 01:57 | |
| clarkb | assuming someone catches it | 01:57 |
| coreyob | there's at least 3 changes in the queue now waiting for that job. if I catch one, can I ping you when it starts? | 01:58 |
| clarkb | sure | 01:58 |
| noorul` | Does anyone have experience adapting Zuul outside of OpenStack ? | 01:58 |
| coreyob | gracias sir | 01:58 |
| Thelo | clarkb: Space is ok, and the pid file is not here, anything else I could check? | 01:58 |
| clarkb | Thelo: off the top of my head I don't know | 01:59 |
| clarkb | Thelo: those are what I would check if the daemon has trouble running | 01:59 |
| *** sdake has joined #openstack-infra | 01:59 | |
| Thelo | ok I will take the init.d/zuul file and play the instruction one by one | 01:59 |
| *** Sukhdev has joined #openstack-infra | 02:02 | |
| *** rossella_ has quit IRC | 02:02 | |
| *** rossella_ has joined #openstack-infra | 02:03 | |
| *** pfallenop has quit IRC | 02:03 | |
| *** Sukhdev has quit IRC | 02:03 | |
| dmsimard | The zuul queue is almost empty, QUICK, submit your patches! :( | 02:04 |
| *** pfallenop has joined #openstack-infra | 02:04 | |
| dmsimard | Well, the check queue, that is. | 02:04 |
| *** tiswanso has joined #openstack-infra | 02:05 | |
| *** weshay_xchat has quit IRC | 02:06 | |
| openstackgerrit | Merged openstack-infra/yaml2ical: Add functionality to batch the meetings list https://review.openstack.org/275459 | 02:07 |
| *** noorul` has quit IRC | 02:07 | |
| *** Swami has quit IRC | 02:08 | |
| *** thorst has joined #openstack-infra | 02:09 | |
| *** thorst has quit IRC | 02:10 | |
| *** edmondsw has quit IRC | 02:10 | |
| *** thorst has joined #openstack-infra | 02:10 | |
| *** keedya has joined #openstack-infra | 02:13 | |
| openstackgerrit | Merged openstack-infra/zuul: Try to make test_idle less racy https://review.openstack.org/276521 | 02:16 |
| openstackgerrit | Merged openstack-infra/zuul: Add job mutex support https://review.openstack.org/275974 | 02:17 |
| openstackgerrit | Lana Brindley proposed openstack-infra/project-config: Modifying docimpact group https://review.openstack.org/276065 | 02:17 |
| *** davidlenwell has quit IRC | 02:18 | |
| *** davidlenwell has joined #openstack-infra | 02:19 | |
| *** thorst has quit IRC | 02:19 | |
| *** ayoung_ has joined #openstack-infra | 02:19 | |
| *** keedya has quit IRC | 02:20 | |
| *** pfallenop has quit IRC | 02:20 | |
| *** pfallenop has joined #openstack-infra | 02:21 | |
| *** keedya has joined #openstack-infra | 02:21 | |
| *** yamamoto_ has joined #openstack-infra | 02:21 | |
| openstackgerrit | Merged openstack-infra/project-config: Fixed backreference in regular expression https://review.openstack.org/276534 | 02:21 |
| *** tiswanso has quit IRC | 02:22 | |
| *** zz_dimtruck is now known as dimtruck | 02:22 | |
| *** tiswanso has joined #openstack-infra | 02:22 | |
| *** davideagnello has quit IRC | 02:23 | |
| *** davidlenwell has quit IRC | 02:23 | |
| *** baoli has joined #openstack-infra | 02:24 | |
| *** davideagnello has joined #openstack-infra | 02:24 | |
| *** daemontool has joined #openstack-infra | 02:25 | |
| *** apoorvad has quit IRC | 02:26 | |
| *** davidlenwell has joined #openstack-infra | 02:27 | |
| coreyob | clarkb 2 more just started, jobs 299 and 300 on https://jenkins05.openstack.org/job/gate-functional-dsvm-magnum-api/ are both check jobs | 02:27 |
| *** EricGonczer_ has joined #openstack-infra | 02:27 | |
| Thelo | clarkb: Just to let you know. playing the start script line by line didn't show anything, I updated zuul and its fine now | 02:27 |
| *** piet has quit IRC | 02:27 | |
| openstackgerrit | Merged openstack-infra/release-tools: remove the announce flag from the release wrapper scripts https://review.openstack.org/275986 | 02:27 |
| *** apoorvad has joined #openstack-infra | 02:29 | |
| clarkb | coreyob: I found the change for 300 what is the change for 299? | 02:30 |
| coreyob | 275003 clarkb | 02:31 |
| *** kzaitsev_mb has joined #openstack-infra | 02:31 | |
| clarkb | ok both are held, if at least one fails in that way we can hop on and take a look | 02:32 |
| coreyob | awesome, thanks! | 02:33 |
| mgagne | has anyone ever got "Command said: fatal: Cannot update paths and switch to branch 'master' at the same time." when running jeepyb manage-projects command? | 02:34 |
| *** Daisy_ has joined #openstack-infra | 02:35 | |
| *** zhurong has joined #openstack-infra | 02:35 | |
| clarkb | mgagne: it isn't ringing any bells | 02:36 |
| *** Daisy has quit IRC | 02:36 | |
| *** Daisy_ is now known as Daisy | 02:37 | |
| mgagne | thanks, will keep looking | 02:37 |
| *** shashank_hegde has joined #openstack-infra | 02:37 | |
| openstackgerrit | venkatamahesh proposed openstack/requirements: Bump up heatclient version to 0.9.0 https://review.openstack.org/272405 | 02:41 |
| *** achanda has joined #openstack-infra | 02:42 | |
| openstackgerrit | venkatamahesh proposed openstack/requirements: Bump python-senlinclient version to 0.3.0 https://review.openstack.org/263598 | 02:42 |
| *** nwkarsten has quit IRC | 02:43 | |
| *** sridhar_ram has joined #openstack-infra | 02:43 | |
| *** EricGonczer_ has quit IRC | 02:47 | |
| *** apoorvad has quit IRC | 02:47 | |
| *** sridhar_ram has quit IRC | 02:48 | |
| mgagne | clarkb found it, wrong private key for jeepyb, can't properly fetch origin. For some reasons, it can clone just fine but can't perform other actions | 02:48 |
| mgagne | or it could be limitations of "Deploy keys" in github. Adding the pubkey to my personal user works just fine but not project deploy key :-/ | 02:50 |
| rm_work | Hey, if global-requirements has "taskflow>=1.25.0" and we put "taskflow>=1.26.0" in our requirements.txt, will it be able to merge or will that be marked as invalid? | 02:51 |
| rm_work | I'm never quite sure what the global-requirements enforcement is | 02:51 |
| *** sflanigan is now known as sflanigan_afk | 02:52 | |
| *** ayoung_ has quit IRC | 02:57 | |
| *** ayoung has quit IRC | 02:57 | |
| *** ayoung has joined #openstack-infra | 02:58 | |
| rm_work | also, anyone have thoughts on "ndg-httpsclient" ( https://pypi.python.org/pypi/ndg-httpsclient ) and whether there might be any reason it wouldn't be accepted into global-requirements so we can use it in Octavia? | 02:58 |
| clarkb | rm_work: that would not be valid | 02:58 |
| clarkb | rm_work: has to match exactly | 02:58 |
| *** noorul has quit IRC | 02:59 | |
| rm_work | clarkb: hmm, ok... | 02:59 |
| *** oomichi has joined #openstack-infra | 02:59 | |
| coreyob | clarkb looks like 299 bombed out | 02:59 |
| rm_work | I guess we will need to make a patch to global-requirements to up the version? we *require* the new taskflow changes in 1.26 to function | 02:59 |
| clarkb | coreyob: it pings but ssh isn't responding | 03:00 |
| clarkb | rm_work: yes | 03:00 |
| *** baoli has quit IRC | 03:00 | |
| coreyob | clarkb so presumably without ssh we can't get on it. do we have monitoring installed? any way other than ssh to see inside, or is a reboot all that's left? | 03:01 |
| clarkb | coreyob: its possible we have console access | 03:02 |
| clarkb | I am not sure if ovh gives that and I don't know the password | 03:02 |
| *** noorul` has joined #openstack-infra | 03:02 | |
| *** doug-fis_ has quit IRC | 03:03 | |
| *** doug-fish has joined #openstack-infra | 03:04 | |
| openstackgerrit | Adam Harwell proposed openstack/requirements: Octavia requires fixes in taskflow 1.26.0 to function https://review.openstack.org/276549 | 03:04 |
| clarkb | I would be really surprised if sshd won an OOMkiller scoring right | 03:05 |
| clarkb | coreyob: ^ | 03:05 |
| coreyob | hrm, not sure. a little beyond my depth. sounds like maybe the best way to see is reboot and grab all the logs? | 03:06 |
| *** baoli has joined #openstack-infra | 03:06 | |
| *** doug-fish has quit IRC | 03:08 | |
| *** baoli has quit IRC | 03:09 | |
| *** sflanigan_afk is now known as sflanigan | 03:10 | |
| clarkb | coreyob: there are a bunch of hung processes on the console log | 03:13 |
| clarkb | INFO: task kswapd0:77 blocked for more than 120 seconds. for example | 03:13 |
| *** dimtruck is now known as zz_dimtruck | 03:13 | |
| coreyob | hrm | 03:14 |
| clarkb | and they all came in at the same time | 03:15 |
| *** noorul` has quit IRC | 03:15 | |
| *** eliqiao has quit IRC | 03:15 | |
| clarkb | java is on the list too which explains jenkins sad | 03:16 |
| clarkb | I don't see sshd | 03:16 |
| *** EricGonczer_ has joined #openstack-infra | 03:16 | |
| *** thorst has joined #openstack-infra | 03:17 | |
| coreyob | so that hang always happens on the test that spins up 2 nova vms with docker running a swarm cluster. could nova be doing something bad? | 03:17 |
| *** yuanying_ has joined #openstack-infra | 03:18 | |
| clarkb | or the kernel or qemu or the kernel or ovs or ya :) | 03:19 |
| clarkb | it definitely seems like a scheduling problem | 03:19 |
| coreyob | fun | 03:20 |
| clarkb | how big are these dockers? | 03:20 |
| clarkb | er vms running docker | 03:20 |
| clarkb | it is possible that some combination of qemu and swap is hosing the kernels ability ot get work done | 03:20 |
| clarkb | did they get droped to 1GB Ram instances? | 03:20 |
| *** zz_dimtruck is now known as dimtruck | 03:21 | |
| *** yuanying has quit IRC | 03:21 | |
| coreyob | not sure, lemme look at the tests | 03:21 |
| clarkb | ovh uses a swapfile too | 03:22 |
| clarkb | possible the passes happened where swap device was present | 03:22 |
| *** tphummel has quit IRC | 03:22 | |
| openstackgerrit | YAMAMOTO Takashi proposed openstack-infra/project-config: gate-networking-midonet-v2-dsvm-rally-nv: Don't forget to enable rally https://review.openstack.org/276550 | 03:23 |
| fungi | clarkb: if it's anything like the dvr jobs which were breaking nodes, the instance becomes responsive to ssh again in another 30-60 minutes when whatever's run away with the machine finally catches up/dies/gets oom-killed | 03:23 |
| *** gyee has quit IRC | 03:23 | |
| fungi | er, not dvr jobs. now i forget which driver it was | 03:23 |
| *** doug-fish has joined #openstack-infra | 03:24 | |
| coreyob | clarkb checked and they are 1gb instances | 03:24 |
| *** baoli has joined #openstack-infra | 03:24 | |
| *** thorst has quit IRC | 03:25 | |
| *** baoli has quit IRC | 03:25 | |
| coreyob | clarkb fungi, so other than not running 2 vms, is there anything else we can do to avoid the problem? | 03:26 |
| clarkb | reduce openstack's memory consumption? perform functional testing with fakes? I am sure there are options | 03:27 |
| clarkb | would likely be good to confirm the problem if the machine comes back in a bit | 03:28 |
| *** doug-fish has quit IRC | 03:28 | |
| *** baoli has joined #openstack-infra | 03:30 | |
| coreyob | sounds like a plan. so we leave it around for another 30-60 minutes and hope that it works itself back to usable and then if it is, we knwo that it is a memory/swap contention issue of some sort for sure | 03:30 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Cleanup bulder jobs and add disconnect support https://review.openstack.org/272384 | 03:30 |
| clarkb | well oomkiller should log if it did anything | 03:30 |
| coreyob | assuming it comes back, clarkb, is there an easy way to confirm all that? | 03:30 |
| *** Daisy_ has joined #openstack-infra | 03:31 | |
| greghaynes | jeblair: clarkb ^ I think that addresses all your comments | 03:31 |
| prometheanfire | greghaynes: hi :P | 03:31 |
| prometheanfire | https://review.openstack.org/#/c/273772/ kthnx :D | 03:31 |
| clarkb | I still can't ssh fwiw | 03:32 |
| coreyob | I'm guessing that a bigger instance isn't available right? | 03:32 |
| clarkb | so it hasn't self corrected yet | 03:32 |
| clarkb | coreyob: correct | 03:32 |
| fungi | clarkb: coreyob: if we can catch one running on rackspace, there's definitely an oob console we have access to on those | 03:32 |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/jeepyb: git checkout command needs SSH environment https://review.openstack.org/276552 | 03:32 |
| mgagne | clarkb found it ^ | 03:32 |
| coreyob | clarkb if it is actually stil runnig the tests, it is entirely possible that it will take a long time to recover. there are two tests that spin up the two vms and it died on the first one | 03:32 |
| *** Daisy has quit IRC | 03:32 | |
| *** Daisy_ is now known as Daisy | 03:33 | |
| *** dims_ has quit IRC | 03:33 | |
| clarkb | coreyob: I think the jenkins agent should kill them once the job is dead but it may not be able to do that | 03:33 |
| *** piet has joined #openstack-infra | 03:33 | |
| coreyob | fungi is it possible for me to tell which host is running the job from jenkins/zuul or is that something only you all can see? | 03:33 |
| greghaynes | prometheanfire: done | 03:33 |
| fungi | though most of the time in situations like this, the oob console ends up useless because either it's gone into "power save" mode and blanked itself (yay virtual vga!) and the machine is unresponsive to virtual keyboard input, or you have the last 25 lines of kernel panic spew and no buffer to scroll back | 03:33 |
| clarkb | coreyob: it tells you at the beginning of the console log | 03:34 |
| clarkb | coreyob: and on the main job status page | 03:34 |
| prometheanfire | greghaynes: thanks, that's the last one I think | 03:34 |
| greghaynes | prometheanfire: awesome :) | 03:34 |
| coreyob | oh i see it in the name of the node -ovh- | 03:34 |
| prometheanfire | greghaynes: I'm sure there will be something wrong, this week hasn't been so nice | 03:35 |
| greghaynes | Always | 03:35 |
| fungi | and also we have no login "password" for these because we disable password login and rely exclusively on ssh key auth, so wouldn't know what password to use anyway | 03:35 |
| clarkb | oh! | 03:35 |
| clarkb | we can remove the memory restriction on the kernel boot line | 03:36 |
| clarkb | no more hpcloud | 03:36 |
| prometheanfire | we are an optimistic bunch | 03:36 |
| prometheanfire | clarkb: how did the osic stuff go? | 03:36 |
| fungi | clarkb: YES! i totally forgot. happy day! | 03:36 |
| prometheanfire | less build fails | 03:37 |
| prometheanfire | that said, I do work for a competitor | 03:37 |
| *** dimtruck is now known as zz_dimtruck | 03:37 | |
| *** sdake has quit IRC | 03:37 | |
| coreyob | clarkb fungi here's on running on rax, hold it and fingers crossed for repro so we can look at oob console? https://jenkins05.openstack.org/job/gate-functional-dsvm-magnum-api/301/ | 03:37 |
| *** sdake has joined #openstack-infra | 03:38 | |
| prometheanfire | clarkb: which rax? | 03:38 |
| *** baoli has quit IRC | 03:41 | |
| *** baoli has joined #openstack-infra | 03:41 | |
| *** yuanying has joined #openstack-infra | 03:42 | |
| clarkb | coreyob: fungi I cant hold it right now feeding kids | 03:43 |
| clarkb | prometheanfire: which rax? | 03:43 |
| *** yamahata has quit IRC | 03:44 | |
| *** yuanying_ has quit IRC | 03:44 | |
| *** piet has quit IRC | 03:45 | |
| fungi | coreyob: clarkb: held node id 7732905 in rax-iad | 03:45 |
| prometheanfire | I know rax as short for rackspace, there are multiple DCs and stuff | 03:45 |
| prometheanfire | fungi: thanks :D | 03:45 |
| openstackgerrit | Adam Harwell proposed openstack/requirements: Add ndg-httpsclient to global requirements https://review.openstack.org/276554 | 03:45 |
| clarkb | prometheanfire: right but in relation to what? | 03:45 |
| prometheanfire | 21:37 < coreyob > clarkb fungi here's on running on rax, hold it and fingers crossed for repro so we can look at oob console? https://jenkins05.openstack.org/job/gate-functional-dsvm-magnum-api/301/ | 03:45 |
| prometheanfire | think that was a mistab... | 03:45 |
| fungi | prometheanfire: we use dfw, iad and ord regions of rackspace public cloud for nodepool | 03:45 |
| prometheanfire | right | 03:46 |
| fungi | prometheanfire: and also run the bulk of our long-lived servers in dfw | 03:46 |
| prometheanfire | everyone runs everything in dfw | 03:47 |
| prometheanfire | I think iad is generally better imo | 03:47 |
| *** gabor_antal has quit IRC | 03:49 | |
| *** EricGonczer_ has quit IRC | 03:49 | |
| *** baoli has quit IRC | 03:53 | |
| *** baoli has joined #openstack-infra | 03:54 | |
| *** baoli has quit IRC | 03:55 | |
| *** kzaitsev_mb has quit IRC | 03:55 | |
| mgagne | not easy to debug jeepyb manage-projects... Each time I think I found the cause, that's not it. feels like a schrodinger's bug | 03:57 |
| openstackgerrit | Masayuki Igawa proposed openstack-infra/system-config: Remove double slash for individual links https://review.openstack.org/276558 | 03:57 |
| mgagne | it's a schrödinbug | 03:57 |
| *** achanda has quit IRC | 03:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Fix wheel release job https://review.openstack.org/276559 | 04:01 |
| jeblair | clarkb, fungi: ^ we should probably not delay that too long in case another wheel release job sets our release slave offline | 04:02 |
| *** keedya has quit IRC | 04:02 | |
| *** keedya has joined #openstack-infra | 04:03 | |
| *** vgridnev has joined #openstack-infra | 04:04 | |
| *** sdake has quit IRC | 04:05 | |
| *** yuanying has quit IRC | 04:06 | |
| *** sdake has joined #openstack-infra | 04:07 | |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** zhurong has quit IRC | 04:08 | |
| *** Daisy has quit IRC | 04:09 | |
| *** keedya has quit IRC | 04:10 | |
| *** keedya has joined #openstack-infra | 04:11 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:11 | |
| *** keedya has quit IRC | 04:13 | |
| *** kushal has quit IRC | 04:13 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add support for OpenRC to dib-init-system https://review.openstack.org/273772 | 04:15 |
| *** sdake_ has joined #openstack-infra | 04:16 | |
| *** keedya has joined #openstack-infra | 04:16 | |
| *** kzaitsev_mb has quit IRC | 04:16 | |
| *** sdake has quit IRC | 04:17 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: Rework functional test runner https://review.openstack.org/276563 | 04:19 |
| prometheanfire | woot, time to test | 04:20 |
| *** thorst has joined #openstack-infra | 04:23 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Handle dib image deletion during periodic cleanup https://review.openstack.org/273703 | 04:26 |
| *** thorst has quit IRC | 04:30 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix wheel release job https://review.openstack.org/276559 | 04:30 |
| *** rlandy has quit IRC | 04:31 | |
| *** Sukhdev has joined #openstack-infra | 04:36 | |
| coreyob | clarkb so the rax vm didn't reproduce the problem. maybe rax vms are just too cool for school. I need to sleep, but I'll try to follow up more with this in the morning. | 04:42 |
| clarkb | rax vms have swap devices not files | 04:42 |
| clarkb | I bet thats the difference | 04:42 |
| coreyob | seems like a logical guess | 04:43 |
| coreyob | I'm guessing there isn't a way to only run that job on rax vms? | 04:43 |
| *** links has joined #openstack-infra | 04:43 | |
| clarkb | no we dont allow single provider gate tests | 04:44 |
| clarkb | because as just happened sometimes thry go away | 04:44 |
| coreyob | 'tis understandable | 04:44 |
| coreyob | all right, I'm going to start a discussion on the magnum ml about the gate issues we're having and include this one in the list. see how people want to proceed | 04:45 |
| coreyob | thanks for your help clarkb | 04:45 |
| *** moravec has joined #openstack-infra | 04:47 | |
| prometheanfire | testing now though | 04:48 |
| *** keedya has quit IRC | 04:49 | |
| *** Thelo has quit IRC | 04:49 | |
| *** keedya has joined #openstack-infra | 04:50 | |
| *** Thelo has joined #openstack-infra | 04:52 | |
| *** moravec has quit IRC | 04:54 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul: Handle a float in assertReportedStat https://review.openstack.org/248975 | 04:56 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul: Update to statsd 3.0 https://review.openstack.org/229232 | 04:56 |
| *** achanda has joined #openstack-infra | 04:58 | |
| *** Daisy has joined #openstack-infra | 05:00 | |
| openstackgerrit | Merged openstack/requirements: Add openstack/bifrost to projects.txt https://review.openstack.org/272334 | 05:02 |
| *** shashank_hegde has quit IRC | 05:02 | |
| *** sdake_ has quit IRC | 05:03 | |
| *** nwkarsten has joined #openstack-infra | 05:04 | |
| *** keedya has quit IRC | 05:04 | |
| *** dalgaaf has joined #openstack-infra | 05:05 | |
| *** dkehn has quit IRC | 05:05 | |
| *** dkehn has joined #openstack-infra | 05:05 | |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/requirements: Bump minimum tempest-lib version https://review.openstack.org/276571 | 05:07 |
| *** salv-orlando has joined #openstack-infra | 05:08 | |
| *** FallenPegasus has joined #openstack-infra | 05:09 | |
| *** amotoki has joined #openstack-infra | 05:11 | |
| *** tphummel has joined #openstack-infra | 05:12 | |
| *** Sukhdev has quit IRC | 05:15 | |
| *** sdake has joined #openstack-infra | 05:16 | |
| *** Sukhdev has joined #openstack-infra | 05:16 | |
| *** salv-orlando has quit IRC | 05:18 | |
| *** FallenPegasus has quit IRC | 05:24 | |
| prometheanfire | hmm, fails, neat | 05:25 |
| *** thorst has joined #openstack-infra | 05:27 | |
| *** shashank_hegde has joined #openstack-infra | 05:27 | |
| *** sdake_ has joined #openstack-infra | 05:31 | |
| *** mrmartin has joined #openstack-infra | 05:32 | |
| *** piet has joined #openstack-infra | 05:32 | |
| *** sdake has quit IRC | 05:32 | |
| prometheanfire | ianw: this is your fault :P | 05:32 |
| *** thorst has quit IRC | 05:34 | |
| *** mrmartin has quit IRC | 05:35 | |
| *** fedexo has joined #openstack-infra | 05:37 | |
| *** kushal has joined #openstack-infra | 05:38 | |
| prometheanfire | ianw: 9305ea4b6db95af9aeb46ce75d5aa03bd636b28c :P | 05:38 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Fix growroot for Gentoo's openrc https://review.openstack.org/276578 | 05:43 |
| *** gildub has quit IRC | 05:43 | |
| *** tiswanso has quit IRC | 05:43 | |
| prometheanfire | ianw: greghaynes https://review.openstack.org/276578 | 05:44 |
| *** sdake_ has quit IRC | 05:44 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/elastic-recheck: testtools 2,0.0 doesn't work in kilo https://review.openstack.org/276579 | 05:46 |
| *** arxcruz has quit IRC | 05:47 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default https://review.openstack.org/276581 | 05:49 |
| openstackgerrit | Tony Breeds proposed openstack/requirements: Cap testtools to <2.0.0 https://review.openstack.org/276275 | 05:50 |
| prometheanfire | ianw: should I add a test for building a gentoo image so ci hits it? | 05:50 |
| ianw | prometheanfire: well, that was partly my idea behind https://review.openstack.org/#/c/276563/ ... so we can better define our default tests but have others around | 05:51 |
| ianw | and we can think more about what we want to test by default | 05:52 |
| prometheanfire | building a gentoo image can take a while though | 05:52 |
| prometheanfire | each test is 15-30 min ish | 05:52 |
| ianw | yep, we can filter by path in project config, however, and maybe have separate targets. i.e. do a gentoo build when we touch gentoo elements | 05:52 |
| prometheanfire | that may be hard, since growroot now does gentoo stuff | 05:53 |
| prometheanfire | don't quite fully understand what you are doing there | 05:55 |
| *** woodster_ has quit IRC | 05:56 | |
| prometheanfire | ianw: this is why I wraped the package-uninstall in a check http://paste.openstack.org/show/486045/ | 05:56 |
| *** piet has quit IRC | 05:59 | |
| prometheanfire | ianw: the install probably needs it as well | 06:01 |
| ianw | prometheanfire: i've never seen that ... i'm sure the change is fine, just explain in the description why this situation occurs so we're all convinced it's not really a bug somewhere else | 06:01 |
| prometheanfire | I will | 06:02 |
| prometheanfire | I'll paste/refrence what I just showed you | 06:02 |
| prometheanfire | ianw: you think I should add the test to the rest of the uses? | 06:02 |
| prometheanfire | elements/package-installs/install.d/01-package-installs for instance | 06:02 |
| *** rossella_ has quit IRC | 06:02 | |
| prometheanfire | actually... | 06:03 |
| *** rossella_ has joined #openstack-infra | 06:03 | |
| *** pfallenop has quit IRC | 06:04 | |
| prometheanfire | my 99-cleanup runs before 99-package-uninstalls | 06:04 |
| prometheanfire | and I clean up /tmp... | 06:04 |
| prometheanfire | renaming to elements/gentoo/post-install.d/99-zzz-cleanup | 06:05 |
| ianw | promethanfire: i don't know. actually, i think maybe it should be fixed in package-installs-v2 ... i mean package-installs skips the empy list, why doesn't -v2? | 06:06 |
| prometheanfire | you are passing it a file | 06:06 |
| prometheanfire | the file should exist imo | 06:06 |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default https://review.openstack.org/276581 | 06:06 |
| prometheanfire | if you pass it a file and it doesn't exist I'd expect v2 to exit non-zero | 06:06 |
| prometheanfire | ianw: I updated that one instead since I was already touching it | 06:07 |
| *** yuanying_ has joined #openstack-infra | 06:08 | |
| *** oomichi has quit IRC | 06:08 | |
| *** yamamoto_ has quit IRC | 06:09 | |
| ianw | prometheanfire: i dunno, have to think about it. my eod here, so get back to it next week | 06:11 |
| prometheanfire | week? :( | 06:11 |
| *** yuanying has quit IRC | 06:11 | |
| openstackgerrit | Tony Breeds proposed openstack-infra/elastic-recheck: testtools 2.0.0 doesn't work in kilo https://review.openstack.org/276579 | 06:11 |
| *** Sukhdev has quit IRC | 06:12 | |
| *** yuanying_ has quit IRC | 06:13 | |
| *** yuanying has joined #openstack-infra | 06:14 | |
| prometheanfire | could just add it as an exclusion to rm -Rf /tmp/!(ccache|in_target*|profiledir*) | 06:14 |
| prometheanfire | many options | 06:14 |
| *** yuanying has quit IRC | 06:15 | |
| *** pfallenop has joined #openstack-infra | 06:17 | |
| openstackgerrit | Matthew Thode proposed openstack/diskimage-builder: Remove eclean-dist as it's not available by default https://review.openstack.org/276581 | 06:19 |
| *** achanda has quit IRC | 06:19 | |
| *** achanda has joined #openstack-infra | 06:20 | |
| bkero | (x-post from #puppet-openstack): Does puppet-openstack-cookiecutter manage modules | 06:21 |
| bkero | after they've been created, or is it a one-time operation at creation? | 06:21 |
| bkero | I'm trying to figure out if this patch belongs in cookiecutter/puppet-{project} or in each individual project (or both) | 06:22 |
| prometheanfire | mordred: is clean not expected to work if taken a image of after first run? (boot, glean runs, image taken, boot taken image, glean doesn't work (old net-config exists is my issue)) | 06:24 |
| *** esp has joined #openstack-infra | 06:26 | |
| *** NobodyCa1 has joined #openstack-infra | 06:26 | |
| *** pfallenop has quit IRC | 06:26 | |
| *** pfallenop has joined #openstack-infra | 06:28 | |
| *** yuanying has joined #openstack-infra | 06:28 | |
| prometheanfire | also, forgot to DIB_INSTALLTYPE_simple_init=repo so I'm using 1.4.3 which is old and crufty | 06:30 |
| prometheanfire | mordred: you should tag a new glean now :P | 06:30 |
| clarkb | bkero cookiecutter is typically just the one time | 06:31 |
| bkero | clarkb: Okay, thanks. I'll fix it in both. | 06:31 |
| *** thorst has joined #openstack-infra | 06:31 | |
| bkero | "How to write beaker tests" points to a 404 | 06:32 |
| bkero | and has for a long time :X | 06:32 |
| *** esp has quit IRC | 06:33 | |
| *** hdd has joined #openstack-infra | 06:33 | |
| *** bhunter71 has quit IRC | 06:34 | |
| *** NobodyCa1 has quit IRC | 06:34 | |
| *** sdake has joined #openstack-infra | 06:36 | |
| *** thorst has quit IRC | 06:39 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Make release afsadmin keytab owned by jenkins https://review.openstack.org/276587 | 06:40 |
| *** camunoz has quit IRC | 06:40 | |
| jeblair | clarkb, fungi, yolanda, krotscheck: ^ one more for the wheels :) | 06:43 |
| *** yamamoto_ has joined #openstack-infra | 06:44 | |
| *** pfallenop has quit IRC | 06:44 | |
| *** pfallenop has joined #openstack-infra | 06:44 | |
| *** hdd has quit IRC | 06:45 | |
| *** sdake has quit IRC | 06:55 | |
| *** tphummel has quit IRC | 06:55 | |
| *** sdake has joined #openstack-infra | 06:55 | |
| cody-somerville | Can nova still only load one hypervisor driver at a time? | 07:02 |
| *** links has quit IRC | 07:04 | |
| *** infra-red has joined #openstack-infra | 07:06 | |
| *** stevemar has joined #openstack-infra | 07:06 | |
| *** infra-re_ has joined #openstack-infra | 07:12 | |
| *** infra-red has quit IRC | 07:14 | |
| *** links has joined #openstack-infra | 07:14 | |
| *** sdake has quit IRC | 07:15 | |
| *** unicell has joined #openstack-infra | 07:22 | |
| *** shashank_hegde has quit IRC | 07:23 | |
| *** mase_x200 has joined #openstack-infra | 07:24 | |
| *** e0ne has joined #openstack-infra | 07:30 | |
| *** ianw has quit IRC | 07:30 | |
| *** ianw has joined #openstack-infra | 07:30 | |
| *** mase_x200 has quit IRC | 07:31 | |
| *** mase_x200 has joined #openstack-infra | 07:31 | |
| *** _nadya_ has joined #openstack-infra | 07:34 | |
| *** infra-red has joined #openstack-infra | 07:35 | |
| *** mrmartin has joined #openstack-infra | 07:35 | |
| *** thorst has joined #openstack-infra | 07:37 | |
| *** infra-re_ has quit IRC | 07:37 | |
| *** ociuhandu has joined #openstack-infra | 07:38 | |
| *** mase_x200 has quit IRC | 07:41 | |
| *** lezbar has joined #openstack-infra | 07:43 | |
| *** thorst has quit IRC | 07:44 | |
| *** jtomasek_ has joined #openstack-infra | 07:45 | |
| *** Daisy has quit IRC | 07:45 | |
| openstackgerrit | Matthew Thode proposed openstack-infra/glean: Remove the manual start of network interfaces https://review.openstack.org/276610 | 07:45 |
| *** Jeffrey4l has quit IRC | 07:48 | |
| *** scheuran has joined #openstack-infra | 07:51 | |
| *** eliqiao has joined #openstack-infra | 07:53 | |
| *** vgridnev has quit IRC | 07:54 | |
| *** esikachev has joined #openstack-infra | 07:54 | |
| *** mase_x200 has joined #openstack-infra | 07:58 | |
| *** Jeffrey4l has joined #openstack-infra | 07:58 | |
| *** henrynash has joined #openstack-infra | 07:59 | |
| yolanda | morning | 07:59 |
| *** tzn has joined #openstack-infra | 08:01 | |
| *** zeih has joined #openstack-infra | 08:02 | |
| AJaeger | morning, yolanda | 08:03 |
| *** MCoLo__ has quit IRC | 08:04 | |
| flip214 | does anybody know what to do about "raise ValueError, "No closing quotation"" in bashate? http://logs.openstack.org/38/276138/1/check/gate-drbd-devstack-bashate/b076b2f/console.html#_2016-02-04_10_56_15_233 | 08:04 |
| *** savihou has joined #openstack-infra | 08:06 | |
| *** sflanigan is now known as sflanigan_afk | 08:06 | |
| *** nwkarsten has quit IRC | 08:07 | |
| *** sflanigan_afk has quit IRC | 08:07 | |
| openstackgerrit | Merged openstack-infra/project-config: Add docs-on-rtfd hook for ironic-staging-drivers project https://review.openstack.org/276283 | 08:12 |
| *** jtomasek__ has joined #openstack-infra | 08:12 | |
| openstackgerrit | Akihiro Motoki proposed openstack-infra/project-config: networking-nec: Enable python3 and check-requirement jobs https://review.openstack.org/276613 | 08:13 |
| openstackgerrit | Merged openstack-infra/project-config: Modifying docimpact group https://review.openstack.org/276065 | 08:13 |
| *** doug-fish has joined #openstack-infra | 08:16 | |
| AJaeger | flip214: see here for list of cores on bashate, and looking at the included group I suggest you ask on #openstack-qa. https://review.openstack.org/#/admin/groups/333,members | 08:17 |
| *** kzaitsev_mb has joined #openstack-infra | 08:18 | |
| AJaeger | flip214: I know the problem... | 08:19 |
| AJaeger | flip214: I'll send a patch for the repo... | 08:19 |
| *** dgonzalez has quit IRC | 08:19 | |
| *** hashar has joined #openstack-infra | 08:20 | |
| *** zeih_ has joined #openstack-infra | 08:20 | |
| *** doug-fish has quit IRC | 08:21 | |
| *** zeih_ has quit IRC | 08:22 | |
| *** jlanoux has joined #openstack-infra | 08:22 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files https://review.openstack.org/243399 | 08:22 |
| AJaeger | flip214: still around? Get https://review.openstack.org/276617 merged... | 08:23 |
| *** zeih has quit IRC | 08:23 | |
| *** amaretskiy has joined #openstack-infra | 08:25 | |
| flip214 | AJaeger: thank you, looking at that. | 08:26 |
| AJaeger | flip214: newer tox broke this for you... | 08:26 |
| *** dgonzalez has joined #openstack-infra | 08:28 | |
| *** samuelBartel has joined #openstack-infra | 08:29 | |
| flip214 | AJaeger: thank you very much, that was really easy! ;) | 08:29 |
| *** doug-fish has joined #openstack-infra | 08:29 | |
| *** dtardivel has joined #openstack-infra | 08:30 | |
| *** ildikov has quit IRC | 08:30 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OpenStack-Ansible main repository from pep8 check to linters check https://review.openstack.org/276620 | 08:31 |
| *** matrohon has joined #openstack-infra | 08:31 | |
| *** alivigni has joined #openstack-infra | 08:31 | |
| flip214 | at least for someone who knows what's he doing.... | 08:32 |
| flip214 | any would somebody explain me why Jenkins ran twice for https://review.openstack.org/#/c/276242/? Is the recheck comment malformed | 08:33 |
| flip214 | ? | 08:33 |
| *** hashar has quit IRC | 08:34 | |
| *** doug-fish has quit IRC | 08:34 | |
| openstackgerrit | Merged openstack-infra/system-config: Increase the HTTP incoming connection queue https://review.openstack.org/275995 | 08:35 |
| AJaeger | flip214: you misled me with "bashate broken" but once I saw tox.ini I remembered - not the first patch like that I did | 08:36 |
| *** flepied has quit IRC | 08:36 | |
| openstackgerrit | Merged openstack-infra/project-config: removed unused job gate-openstack-chef-repo https://review.openstack.org/276106 | 08:36 |
| *** henrynash has quit IRC | 08:37 | |
| AJaeger | flip214: why twice? | 08:37 |
| AJaeger | flip214: ah, I see - no idea... | 08:38 |
| *** mikelk has joined #openstack-infra | 08:41 | |
| *** thorst has joined #openstack-infra | 08:42 | |
| *** placko has joined #openstack-infra | 08:43 | |
| *** ihrachys has joined #openstack-infra | 08:43 | |
| *** salv-orlando has joined #openstack-infra | 08:45 | |
| flip214 | AJaeger: thank you very much, anyway! I guess the "syntax" of the "recheck" command is wrong... | 08:45 |
| flip214 | might be interesting to reduce the load on jenkins, though | 08:45 |
| AJaeger | yolanda: what do you think about https://review.openstack.org/273436 ? | 08:47 |
| openstackgerrit | Merged openstack-infra/project-config: add project poppy-ui a Horizon Plugin to work with Poppy's CDN API https://review.openstack.org/267806 | 08:47 |
| yolanda | AJaeger, checking... that will need an specific project rename task right? | 08:47 |
| AJaeger | flip214: no, that's just the jobs.... | 08:47 |
| AJaeger | I meant yolanda ^ | 08:48 |
| AJaeger | flip214: better ask others... | 08:48 |
| yolanda | ah yes, i think it makes sense to get rid of stackforge concept | 08:48 |
| *** thorst has quit IRC | 08:48 | |
| yolanda | i found a small typo | 08:49 |
| *** salv-orlando has quit IRC | 08:49 | |
| *** savihou has quit IRC | 08:49 | |
| AJaeger | thanks, will fix... | 08:50 |
| *** savihou has joined #openstack-infra | 08:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename stackforge-specs-jobs to build-only-specs-jobs https://review.openstack.org/273436 | 08:50 |
| AJaeger | yolanda: thanks, fixed ^ | 08:50 |
| *** alivigni has quit IRC | 08:52 | |
| *** chlong has quit IRC | 08:52 | |
| *** savihou has quit IRC | 08:52 | |
| *** savihou has joined #openstack-infra | 08:53 | |
| *** ihrachys has quit IRC | 08:53 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add experimental CentOS check to OpenStack-Ansible roles https://review.openstack.org/276626 | 08:58 |
| *** vgridnev has joined #openstack-infra | 08:58 | |
| *** salv-orlando has joined #openstack-infra | 09:00 | |
| *** armax has quit IRC | 09:01 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OpenStack-Ansible main repository from pep8 check to linters check https://review.openstack.org/276620 | 09:01 |
| *** flepied has joined #openstack-infra | 09:03 | |
| *** fedexo has quit IRC | 09:03 | |
| *** HeOS has quit IRC | 09:03 | |
| *** HeOS has joined #openstack-infra | 09:04 | |
| *** Hal has joined #openstack-infra | 09:05 | |
| *** Hal is now known as Guest66109 | 09:05 | |
| *** oomichi has joined #openstack-infra | 09:06 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Add information about enabling translations https://review.openstack.org/273759 | 09:06 |
| openstackgerrit | Timur Sufiev proposed openstack/requirements: Remove version exclusions for xvfbwrapper and selenium https://review.openstack.org/276288 | 09:07 |
| openstackgerrit | Merged openstack-infra/project-config: Add 'create reference' to fuel-plugin-detach-* ACL https://review.openstack.org/275780 | 09:08 |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 09:10 |
| *** tzn has quit IRC | 09:10 | |
| *** kzaitsev_mb has quit IRC | 09:13 | |
| odyssey4me | AJaeger hmm, so my only option is to put it into the check pipeline as a non-voting job? | 09:13 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Rename stackforge-specs-jobs to build-only-specs-jobs https://review.openstack.org/273436 | 09:14 |
| *** yolanda has quit IRC | 09:14 | |
| *** zeih has joined #openstack-infra | 09:14 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-infra | 09:17 | |
| *** ociuhandu has quit IRC | 09:18 | |
| *** zeih has quit IRC | 09:19 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add non-voting CentOS check to OpenStack-Ansible roles https://review.openstack.org/276626 | 09:19 |
| *** markvoelker has quit IRC | 09:20 | |
| *** lucas-dinner is now known as lucasagomes | 09:22 | |
| *** jordanP has joined #openstack-infra | 09:23 | |
| *** ifarkas has joined #openstack-infra | 09:23 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Add information about enabling translations https://review.openstack.org/273759 | 09:24 |
| AJaeger | odyssey4me: you can put it into experimental - but then don't use the template | 09:25 |
| *** infra-re_ has joined #openstack-infra | 09:25 | |
| AJaeger | odyssey4me: or create a new template... | 09:25 |
| AJaeger | odyssey4me: the template adds only to check | 09:25 |
| *** mase_x200 has quit IRC | 09:25 | |
| odyssey4me | AJaeger all those roles aren't going through much churn (not many commits), so adding it to the check queue is actually fine | 09:26 |
| *** henrynash has joined #openstack-infra | 09:26 | |
| AJaeger | odyssey4me: that makes it easier ;) | 09:26 |
| odyssey4me | yep :) | 09:26 |
| *** ildikov has joined #openstack-infra | 09:28 | |
| *** infra-red has quit IRC | 09:28 | |
| odyssey4me | AJaeger with regards to openstack-ansible-plugins, the docs job name should also be changed, right? | 09:28 |
| odyssey4me | it should also be 'gate-{name}-docs'? | 09:29 |
| AJaeger | yes | 09:29 |
| *** achanda has quit IRC | 09:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OpenStack-Ansible main repository from pep8 check to linters check https://review.openstack.org/276620 | 09:31 |
| *** ihrachys has joined #openstack-infra | 09:33 | |
| odyssey4me | thanks for the advice AJaeger - as always you're a pleasure to work with :) | 09:35 |
| *** ihrachys has quit IRC | 09:35 | |
| *** stevemar has quit IRC | 09:35 | |
| *** stevemar has joined #openstack-infra | 09:36 | |
| AJaeger | you're welcome, odyssey4me | 09:36 |
| AJaeger | time for a break now ;) | 09:36 |
| *** esikachev has quit IRC | 09:37 | |
| *** stevemar has quit IRC | 09:40 | |
| *** ihrachys has joined #openstack-infra | 09:44 | |
| *** abregman has joined #openstack-infra | 09:45 | |
| *** thorst has joined #openstack-infra | 09:47 | |
| *** zeih has joined #openstack-infra | 09:47 | |
| *** zeih_ has joined #openstack-infra | 09:48 | |
| *** esikachev has joined #openstack-infra | 09:48 | |
| *** ihrachys has quit IRC | 09:51 | |
| *** zeih has quit IRC | 09:51 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Add information about enabling translations https://review.openstack.org/273759 | 09:52 |
| *** ociuhandu has joined #openstack-infra | 09:52 | |
| *** boris-42 has joined #openstack-infra | 09:52 | |
| *** ihrachys has joined #openstack-infra | 09:53 | |
| *** thorst has quit IRC | 09:54 | |
| *** tzn has joined #openstack-infra | 09:56 | |
| openstackgerrit | Marton Kiss proposed openstack-infra/system-config: Enable ask-staging.openstack.org puppet run https://review.openstack.org/276656 | 09:57 |
| *** zeih_ has quit IRC | 09:57 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Add non-voting CentOS check to OpenStack-Ansible roles https://review.openstack.org/276626 | 09:58 |
| *** ikalnitsky has left #openstack-infra | 09:58 | |
| *** pahuang has quit IRC | 10:00 | |
| *** tzn has quit IRC | 10:00 | |
| *** ihrachys has quit IRC | 10:02 | |
| *** ihrachys has joined #openstack-infra | 10:05 | |
| *** ihrachys has quit IRC | 10:08 | |
| *** ihrachys has joined #openstack-infra | 10:08 | |
| *** ociuhandu has quit IRC | 10:08 | |
| *** rguillebert has joined #openstack-infra | 10:09 | |
| *** yolanda has joined #openstack-infra | 10:13 | |
| *** yamamoto_ has quit IRC | 10:15 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Switch OpenStack-Ansible main repository from pep8 check to linters check https://review.openstack.org/276620 | 10:15 |
| *** yamamoto_ has joined #openstack-infra | 10:17 | |
| *** dalgaaf has quit IRC | 10:18 | |
| *** markvoelker has joined #openstack-infra | 10:21 | |
| *** rfolco has joined #openstack-infra | 10:22 | |
| *** ihrachys has quit IRC | 10:24 | |
| *** bnemec has joined #openstack-infra | 10:25 | |
| *** ihrachys has joined #openstack-infra | 10:25 | |
| *** markvoelker has quit IRC | 10:26 | |
| *** jaosorior has joined #openstack-infra | 10:27 | |
| *** jaosorior has quit IRC | 10:28 | |
| *** ldnunes has joined #openstack-infra | 10:29 | |
| *** jaosorior has joined #openstack-infra | 10:30 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:30 | |
| *** ihrachys has quit IRC | 10:30 | |
| *** ihrachys has joined #openstack-infra | 10:31 | |
| *** electrofelix has joined #openstack-infra | 10:33 | |
| *** ihrachys has quit IRC | 10:36 | |
| *** stevemar has joined #openstack-infra | 10:37 | |
| *** salv-orl_ has joined #openstack-infra | 10:39 | |
| *** stevemar has quit IRC | 10:40 | |
| *** salv-orlando has quit IRC | 10:42 | |
| *** electrofelix has quit IRC | 10:42 | |
| *** electrofelix has joined #openstack-infra | 10:43 | |
| *** aysyd has joined #openstack-infra | 10:43 | |
| *** reed_ has joined #openstack-infra | 10:45 | |
| *** henrynash has quit IRC | 10:45 | |
| *** tcammann has joined #openstack-infra | 10:48 | |
| *** zeih has joined #openstack-infra | 10:49 | |
| *** thorst has joined #openstack-infra | 10:52 | |
| *** tzn has joined #openstack-infra | 10:54 | |
| *** e0ne has quit IRC | 10:57 | |
| *** sfinucan has joined #openstack-infra | 10:59 | |
| *** thorst has quit IRC | 10:59 | |
| *** tcammann has quit IRC | 10:59 | |
| *** tzn has quit IRC | 10:59 | |
| *** abregman is now known as abregman|afk | 11:01 | |
| *** jaosorior has quit IRC | 11:04 | |
| *** zeih has quit IRC | 11:05 | |
| *** ashishk has joined #openstack-infra | 11:06 | |
| *** ashishk has left #openstack-infra | 11:07 | |
| *** reed_ has quit IRC | 11:09 | |
| *** cbartz has joined #openstack-infra | 11:09 | |
| *** mrodden has quit IRC | 11:10 | |
| *** aysyd has quit IRC | 11:10 | |
| *** ruhe has quit IRC | 11:11 | |
| *** pkoniszewski_ has joined #openstack-infra | 11:11 | |
| *** aysyd has joined #openstack-infra | 11:11 | |
| *** pkoniszewski has quit IRC | 11:12 | |
| *** timrc_ has quit IRC | 11:12 | |
| *** skraynev has quit IRC | 11:12 | |
| *** dmsimard has quit IRC | 11:12 | |
| *** katyafervent_awa has quit IRC | 11:12 | |
| *** pkarikh has quit IRC | 11:12 | |
| *** jokke_ has quit IRC | 11:12 | |
| *** toabctl_ has quit IRC | 11:12 | |
| *** vsaienko has quit IRC | 11:12 | |
| *** pedroalvarez_ has joined #openstack-infra | 11:12 | |
| *** pedroalvarez has quit IRC | 11:12 | |
| *** Daisy_ has joined #openstack-infra | 11:12 | |
| *** zhurong has joined #openstack-infra | 11:13 | |
| *** jokke_ has joined #openstack-infra | 11:13 | |
| *** ruhe has joined #openstack-infra | 11:16 | |
| *** zhurong has quit IRC | 11:17 | |
| *** mrodden has joined #openstack-infra | 11:17 | |
| *** toabctl has joined #openstack-infra | 11:17 | |
| *** timrc_ has joined #openstack-infra | 11:17 | |
| *** jaosorior has joined #openstack-infra | 11:17 | |
| *** zeih has joined #openstack-infra | 11:17 | |
| *** yamamoto_ has quit IRC | 11:18 | |
| *** zhurong has joined #openstack-infra | 11:19 | |
| *** pkarikh has joined #openstack-infra | 11:20 | |
| *** katyafervent_awa has joined #openstack-infra | 11:20 | |
| *** zhurong has quit IRC | 11:22 | |
| *** FallenPegasus has joined #openstack-infra | 11:23 | |
| *** vsaienko has joined #openstack-infra | 11:24 | |
| *** skraynev has joined #openstack-infra | 11:24 | |
| *** dmsimard has joined #openstack-infra | 11:26 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Check branch-override in ofagent job-template https://review.openstack.org/274930 | 11:27 |
| *** Daisy_ has quit IRC | 11:30 | |
| *** bnemec has quit IRC | 11:31 | |
| *** Daisy_ has joined #openstack-infra | 11:31 | |
| dgonzalez | Hi AJaeger: do you know where i can find the code for the api documentation? | 11:31 |
| *** nikil89_ has joined #openstack-infra | 11:34 | |
| *** Daisy_ has quit IRC | 11:35 | |
| *** Daisy_ has joined #openstack-infra | 11:39 | |
| *** BobBall_ has joined #openstack-infra | 11:42 | |
| *** BobBall_ is now known as Guest75949 | 11:42 | |
| *** BobBall has quit IRC | 11:45 | |
| *** dims has joined #openstack-infra | 11:46 | |
| *** doug-fish has joined #openstack-infra | 11:49 | |
| *** ociuhandu has joined #openstack-infra | 11:51 | |
| *** esikachev has quit IRC | 11:53 | |
| *** zeih has quit IRC | 11:53 | |
| dims | AJaeger : thanks! | 11:54 |
| *** jaosorior has quit IRC | 11:54 | |
| dims | sdague : can you please bless this experiment? (Run py27/py34 from nova periodically with oslo.* master) https://review.openstack.org/#/c/276313/ | 11:55 |
| *** e0ne has joined #openstack-infra | 11:55 | |
| *** tzn has joined #openstack-infra | 11:56 | |
| *** thorst has joined #openstack-infra | 11:57 | |
| *** esikachev has joined #openstack-infra | 11:58 | |
| *** rvba` is now known as rvba | 12:01 | |
| *** salv-orl_ has quit IRC | 12:01 | |
| *** tzn has quit IRC | 12:01 | |
| *** doug-fish has quit IRC | 12:02 | |
| *** thorst has quit IRC | 12:04 | |
| *** mrmartin has quit IRC | 12:06 | |
| *** mrmartin has joined #openstack-infra | 12:07 | |
| *** sfinucan has quit IRC | 12:08 | |
| *** mase_x200 has joined #openstack-infra | 12:08 | |
| sdague | dims: +A | 12:13 |
| dims | thanks Sean | 12:13 |
| sdague | that's a lot of master changes to test all at once, will be interesting to see if regressions can be sifted easily enough in ther e | 12:13 |
| *** HeOS has quit IRC | 12:14 | |
| *** HeOS has joined #openstack-infra | 12:14 | |
| dims | sdague : i do this currently once or twice a week on my travis setup. trying to move that to CI | 12:16 |
| sdague | ok, cool | 12:17 |
| *** yamamoto has joined #openstack-infra | 12:19 | |
| *** Guest75949 is now known as BobBall | 12:19 | |
| openstackgerrit | Merged openstack-infra/project-config: Experimental periodic jobs for testing against oslo.* from master https://review.openstack.org/276313 | 12:21 |
| *** markvoelker has joined #openstack-infra | 12:22 | |
| *** jaosorior has joined #openstack-infra | 12:22 | |
| *** jaosorior has quit IRC | 12:23 | |
| *** jaosorior has joined #openstack-infra | 12:23 | |
| *** yamamoto has quit IRC | 12:25 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: bump oslo.config in upper constraints https://review.openstack.org/274748 | 12:26 |
| *** cbartz has left #openstack-infra | 12:26 | |
| *** markvoelker has quit IRC | 12:27 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: bump oslo.config in upper constraints https://review.openstack.org/274748 | 12:27 |
| *** FallenPegasus is now known as MarkAtwood | 12:27 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: bump openstackclient in u-c to 2.1.0 https://review.openstack.org/276373 | 12:28 |
| *** salv-orlando has joined #openstack-infra | 12:29 | |
| *** vgridnev has quit IRC | 12:32 | |
| bookwar_ | is there a place where one can check the 'recheck' stats? like, how many 'recheck' comments happening in openstack/neutron in last week | 12:33 |
| *** gordc has joined #openstack-infra | 12:33 | |
| *** weshay_xchat has joined #openstack-infra | 12:34 | |
| openstackgerrit | Merged openstack-infra/release-tools: remove notable changes option https://review.openstack.org/276468 | 12:34 |
| *** stevemar has joined #openstack-infra | 12:35 | |
| *** lucasagomes is now known as lucas-hungry | 12:35 | |
| *** hashar has joined #openstack-infra | 12:35 | |
| *** tzn has joined #openstack-infra | 12:36 | |
| *** vgridnev has joined #openstack-infra | 12:36 | |
| *** mase_x200 has quit IRC | 12:37 | |
| *** mase_x200 has joined #openstack-infra | 12:38 | |
| *** mase_x200 has quit IRC | 12:38 | |
| *** daemontool has quit IRC | 12:38 | |
| *** stevemar has quit IRC | 12:38 | |
| *** mase_x200 has joined #openstack-infra | 12:39 | |
| *** dingyichen has quit IRC | 12:39 | |
| *** mase_x200 has quit IRC | 12:39 | |
| *** thorst has joined #openstack-infra | 12:41 | |
| *** krotscheck1 has joined #openstack-infra | 12:41 | |
| *** krotscheck1 has quit IRC | 12:41 | |
| mordred | prometheanfire: ah! that's a use case I don't think we considered (duh) | 12:41 |
| *** links has quit IRC | 12:43 | |
| *** savihou has quit IRC | 12:44 | |
| *** zeih has joined #openstack-infra | 12:44 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Add check requirements job to octavia https://review.openstack.org/276711 | 12:46 |
| *** zeih_ has joined #openstack-infra | 12:46 | |
| *** notnownikki has joined #openstack-infra | 12:47 | |
| *** mrmartin has quit IRC | 12:48 | |
| *** zeih has quit IRC | 12:49 | |
| *** mtanino has joined #openstack-infra | 12:50 | |
| *** mrmartin has joined #openstack-infra | 12:51 | |
| *** placko has quit IRC | 12:51 | |
| *** jaosorior has quit IRC | 12:53 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-infra/project-config: Add check requirements job to octavia https://review.openstack.org/276711 | 12:54 |
| *** jaosorior has joined #openstack-infra | 12:56 | |
| *** erlon has joined #openstack-infra | 12:56 | |
| *** rlandy has joined #openstack-infra | 12:58 | |
| *** trown|outttypeww is now known as trown | 12:59 | |
| *** jaypipes has quit IRC | 13:00 | |
| *** krotscheck has joined #openstack-infra | 13:01 | |
| *** jaypipes has joined #openstack-infra | 13:04 | |
| *** esikachev has quit IRC | 13:04 | |
| *** EricGonczer_ has joined #openstack-infra | 13:06 | |
| *** esikachev has joined #openstack-infra | 13:09 | |
| *** Jeffrey4l has quit IRC | 13:09 | |
| *** Jeffrey4l has joined #openstack-infra | 13:09 | |
| *** dizquierdo has joined #openstack-infra | 13:10 | |
| *** daemontool has joined #openstack-infra | 13:11 | |
| AJaeger | dgonzalez: what exactly do you mean? | 13:11 |
| *** esp has joined #openstack-infra | 13:12 | |
| * krotscheck has survived an hour without coffee. | 13:12 | |
| * krotscheck should do something about this. | 13:12 | |
| *** mrmartin has quit IRC | 13:12 | |
| *** tcammann has joined #openstack-infra | 13:12 | |
| dgonzalez | AJeager: i already found it. I meant the api-site repository | 13:12 |
| *** salv-orlando has quit IRC | 13:13 | |
| dgonzalez | but thanks anyway :) | 13:13 |
| AJaeger | good ;) | 13:13 |
| *** jaosorior has quit IRC | 13:14 | |
| *** NobodyCa1 has joined #openstack-infra | 13:15 | |
| *** dizquierdo has quit IRC | 13:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Do not publish openstackdocstheme documents https://review.openstack.org/276722 | 13:16 |
| *** tcammann has quit IRC | 13:19 | |
| krotscheck | So, how comfortable is everyone feeling about shutting down the old pypi mirrors? | 13:19 |
| *** deva_ has joined #openstack-infra | 13:19 | |
| *** tcammann has joined #openstack-infra | 13:20 | |
| *** tcammann_ has joined #openstack-infra | 13:20 | |
| *** NobodyCa1 has quit IRC | 13:21 | |
| AJaeger | krotscheck: let's fix https://jenkins.openstack.org/job/wheel-release-trustyx64/ first IMHO | 13:21 |
| *** NobodyCa1 has joined #openstack-infra | 13:21 | |
| AJaeger | or let's merge a few changes that show it's fixed ;) | 13:21 |
| AJaeger | s/merge/publish/ | 13:22 |
| *** markvoelker has joined #openstack-infra | 13:22 | |
| *** deva_ has quit IRC | 13:23 | |
| *** samueldmq1 has joined #openstack-infra | 13:23 | |
| *** boris-42 has quit IRC | 13:23 | |
| *** deva_ has joined #openstack-infra | 13:24 | |
| krotscheck | AJaeger: I feel like the afs mirrors are on par with the former pypi mirrors | 13:24 |
| *** markvoelker_ has joined #openstack-infra | 13:24 | |
| *** markvoelker has quit IRC | 13:24 | |
| krotscheck | The wheel things are icing on the cake :) | 13:24 |
| *** tcammann has quit IRC | 13:25 | |
| *** yaume has joined #openstack-infra | 13:25 | |
| *** tcammann_ has quit IRC | 13:26 | |
| AJaeger | krotscheck: I see... | 13:26 |
| *** kgiusti has joined #openstack-infra | 13:27 | |
| *** esp has quit IRC | 13:27 | |
| *** NobodyCa1 has quit IRC | 13:27 | |
| *** deva_ has quit IRC | 13:28 | |
| *** samueldmq1 has quit IRC | 13:28 | |
| *** xyang1 has joined #openstack-infra | 13:30 | |
| krotscheck | AJaeger: I'm just polling opinions. Not saying we should shutdown the old servers and/or get rid of the manifests yet | 13:30 |
| *** pradk has joined #openstack-infra | 13:32 | |
| AJaeger | to me it looks fine but it's also quite soon that we have them up. So, my comfortability level is rising every day ;) | 13:33 |
| *** edmondsw has joined #openstack-infra | 13:34 | |
| *** baoli has joined #openstack-infra | 13:34 | |
| *** baoli_ has joined #openstack-infra | 13:37 | |
| *** baoli has quit IRC | 13:39 | |
| *** sdake has joined #openstack-infra | 13:40 | |
| *** woodster_ has joined #openstack-infra | 13:42 | |
| mordred | krotscheck: I have heard that we have wheel mirrors now! | 13:46 |
| mordred | bookwar_: http://status.openstack.org/elastic-recheck/ | 13:47 |
| krotscheck | mordred: We do! | 13:47 |
| mordred | krotscheck: that's sooper exciting! | 13:47 |
| krotscheck | http://mirror.dfw.rax.openstack.org/wheel/ubuntu-14.04-x86_64 | 13:47 |
| krotscheck | mordred: https://review.openstack.org/#/q/(topic:afs-mirror+OR+topic:wheel_mirror)+status:open+branch:master | 13:48 |
| mordred | http://mirror.dfw.rax.openstack.org/wheel/ubuntu-14.04-x86_64/lxml/ | 13:48 |
| mordred | is exciting | 13:48 |
| krotscheck | We'd need a couple of reviews :) | 13:48 |
| * mordred goes to look | 13:48 | |
| mordred | also, I guess I should finish the apt mirror work now | 13:48 |
| mordred | you're making me look lazy :) | 13:49 |
| *** doug-fish has joined #openstack-infra | 13:49 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 13:49 |
| *** annegentle has joined #openstack-infra | 13:49 | |
| krotscheck | mordred: Blame jeblair. He's been a very helpful consistent +2 | 13:49 |
| *** ihrachys has joined #openstack-infra | 13:50 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 13:50 |
| *** keedya has joined #openstack-infra | 13:51 | |
| mordred | krotscheck: I have approved the ones that have passed tests | 13:53 |
| *** ccrouch has quit IRC | 13:53 | |
| *** sdake_ has joined #openstack-infra | 13:53 | |
| *** moravec has joined #openstack-infra | 13:53 | |
| *** ihrachys has quit IRC | 13:54 | |
| *** bastelfreak has quit IRC | 13:54 | |
| *** julim has joined #openstack-infra | 13:54 | |
| *** MarkAtwood has quit IRC | 13:54 | |
| *** bastelfreak has joined #openstack-infra | 13:55 | |
| *** sdake has quit IRC | 13:55 | |
| *** jaosorior has joined #openstack-infra | 13:56 | |
| *** mattt has joined #openstack-infra | 13:56 | |
| mattt | hi all, not overly familiar with python packaging but i notice when i install keystone from source i get some additional wsgi scripts in my venv's bin dir ... but when i install from wheel i don't ... anyknow know what i'm doing wrong? | 13:56 |
| *** tiswanso has joined #openstack-infra | 13:57 | |
| *** tiswanso has quit IRC | 13:58 | |
| *** eharney has joined #openstack-infra | 13:59 | |
| *** dkranz has joined #openstack-infra | 13:59 | |
| *** tiswanso has joined #openstack-infra | 13:59 | |
| krotscheck | mordred: Awesome, thanks :) | 14:00 |
| *** lucas-hungry is now known as lucasagomes | 14:00 | |
| *** hashar has quit IRC | 14:01 | |
| *** rossella_ has quit IRC | 14:02 | |
| *** rossella_ has joined #openstack-infra | 14:03 | |
| *** arxcruz has joined #openstack-infra | 14:03 | |
| *** jaosorior has quit IRC | 14:03 | |
| openstackgerrit | Merged openstack-infra/system-config: Make release afsadmin keytab owned by jenkins https://review.openstack.org/276587 | 14:06 |
| openstackgerrit | Merged openstack-infra/system-config: Update bandersnatch documentation https://review.openstack.org/273519 | 14:07 |
| openstackgerrit | Merged openstack-infra/system-config: Remove trusted-hosts from pip.conf https://review.openstack.org/273636 | 14:07 |
| *** abregman|afk is now known as abregman | 14:08 | |
| cbader | I have a question on starting nodepool I am getting an error "nodepoold: error: unrecognized arguments: --no-builder anyone know this error or seen before or where to look? | 14:11 |
| *** salv-orlando has joined #openstack-infra | 14:13 | |
| *** ccrouch has joined #openstack-infra | 14:14 | |
| *** jcoufal has joined #openstack-infra | 14:14 | |
| krotscheck | Yesssss | 14:14 |
| krotscheck | mordred: Do you know how to rerun https://jenkins.openstack.org/job/wheel-release-trustyx64/ ? | 14:15 |
| *** infra-re_ is now known as infrared | 14:15 | |
| *** infrared is now known as Guest72378 | 14:16 | |
| *** mkoderer__ has quit IRC | 14:16 | |
| *** Guest72378 is now known as infra-red | 14:16 | |
| *** mkoderer__ has joined #openstack-infra | 14:17 | |
| onovy | patch 275181 // someone for final +A please? | 14:18 |
| mordred | krotscheck: it's a periodic or a post job? | 14:19 |
| krotscheck | mordred: Both | 14:19 |
| krotscheck | I think this one's periodic | 14:19 |
| mordred | periodic in this case - one sec | 14:19 |
| *** piet has joined #openstack-infra | 14:19 | |
| *** sabeen1 has joined #openstack-infra | 14:19 | |
| *** amitgandhinz has joined #openstack-infra | 14:22 | |
| *** sabeen3 has joined #openstack-infra | 14:23 | |
| *** yamamoto has joined #openstack-infra | 14:24 | |
| *** jsavak has joined #openstack-infra | 14:24 | |
| mordred | krotscheck: I can retrigger - but puppet has not yet run on that host with that change | 14:24 |
| krotscheck | mordred: Ah, right. We can wait | 14:25 |
| *** sabeen1 has quit IRC | 14:25 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/272587 | 14:29 |
| *** pkoniszewski_ is now known as pkoniszewski | 14:29 | |
| *** zeih has joined #openstack-infra | 14:29 | |
| *** doug-fish has quit IRC | 14:29 | |
| *** doug-fish has joined #openstack-infra | 14:30 | |
| *** salv-orlando has quit IRC | 14:30 | |
| krotscheck | mordred: That APT patch, what does the regular APT folder structure look like? | 14:31 |
| mordred | ./data/pool/main/m/mono/libmono-accessibility4.0-cil_3.2.8+dfsg-4ubuntu1_all.deb | 14:31 |
| mordred | for instance | 14:31 |
| krotscheck | So, naturally AFS Supportive. Nice | 14:32 |
| krotscheck | Lemme go run this on a thing | 14:32 |
| *** zeih_ has quit IRC | 14:32 | |
| *** aysyd has quit IRC | 14:33 | |
| mordred | krotscheck: woot! | 14:33 |
| *** aysyd has joined #openstack-infra | 14:33 | |
| mordred | krotscheck: I tested the reprepro config stuff locally, but have not actually tested the puppet yet | 14:33 |
| krotscheck | mordred: Runinng puppet, let's see what happens | 14:34 |
| mordred | neat! | 14:34 |
| *** alaski is now known as lascii | 14:35 | |
| *** annegentle has quit IRC | 14:36 | |
| *** amotoki has quit IRC | 14:37 | |
| *** nikil89_ has quit IRC | 14:39 | |
| *** xyang1 has quit IRC | 14:41 | |
| *** Daisy_ has quit IRC | 14:41 | |
| *** jtomasek__ has quit IRC | 14:43 | |
| *** jsavak has quit IRC | 14:43 | |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Cap osprofiler to avoid 1.0.0 https://review.openstack.org/276761 | 14:44 |
| *** Daisy has joined #openstack-infra | 14:44 | |
| *** jsavak has joined #openstack-infra | 14:46 | |
| *** abregman is now known as abregman|afk | 14:46 | |
| *** regXboi has joined #openstack-infra | 14:47 | |
| *** [1]Thelo has joined #openstack-infra | 14:47 | |
| zigo | mordred: Hi there! Would you mind reviewing this one? https://review.openstack.org/#/c/264726/ | 14:47 |
| zigo | I've added you, but I don't think you've paid attention... | 14:47 |
| *** sdake_ has quit IRC | 14:49 | |
| *** Thelo has quit IRC | 14:49 | |
| *** [1]Thelo is now known as Thelo | 14:49 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add Create Story Button in Worklists https://review.openstack.org/269656 | 14:51 |
| krotscheck | mordred: It seems to be working. | 14:53 |
| krotscheck | mordred: As in, the puppet didn't fail, and my VM is complaining about not having a keytab in the reprepro log | 14:53 |
| *** NobodyCa1 has joined #openstack-infra | 14:53 | |
| *** esp has joined #openstack-infra | 14:54 | |
| krotscheck | mordred: I | 14:55 |
| *** jordanP has quit IRC | 14:56 | |
| *** mriedem has joined #openstack-infra | 14:57 | |
| mordred | krotscheck: that's excellent! | 14:57 |
| mordred | krotscheck: I'm about to push up a couple of follow on patches | 14:57 |
| krotscheck | Coolio | 14:58 |
| *** mrmartin has joined #openstack-infra | 14:58 | |
| krotscheck | mordred, AJaeger If one of you gets a chance, puppet has run on the release slave. https://jenkins.openstack.org/job/wheel-release-trustyx64 (periodic) is ready to be triggered again | 14:58 |
| openstackgerrit | Sean McGinnis proposed openstack/requirements: Cap osprofiler to avoid 1.0.0 https://review.openstack.org/276761 | 14:59 |
| *** tcammann_ has joined #openstack-infra | 14:59 | |
| *** dizquierdo has joined #openstack-infra | 14:59 | |
| *** samueldmq1 has joined #openstack-infra | 15:00 | |
| *** yamamoto has quit IRC | 15:01 | |
| *** hdd has joined #openstack-infra | 15:01 | |
| *** NobodyCa1 has quit IRC | 15:02 | |
| *** bhunter71 has joined #openstack-infra | 15:02 | |
| mordred | krotscheck: https://jenkins.openstack.org/job/wheel-release-trustyx64/8/console | 15:03 |
| *** mrmartin has quit IRC | 15:03 | |
| *** doug-fis_ has joined #openstack-infra | 15:03 | |
| *** pradk has quit IRC | 15:03 | |
| *** tcammann_ has quit IRC | 15:03 | |
| *** baoli_ has quit IRC | 15:03 | |
| *** pradk has joined #openstack-infra | 15:03 | |
| krotscheck | Hurm. | 15:04 |
| *** doug-fish has quit IRC | 15:04 | |
| *** baoli has joined #openstack-infra | 15:04 | |
| *** mrmartin has joined #openstack-infra | 15:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Drive config from list of distro releases https://review.openstack.org/276770 | 15:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Shrink list of reprepro updates https://review.openstack.org/276771 | 15:04 |
| AJaeger | doug-fis_: are you around (if you're doug fish)? | 15:04 |
| mordred | krotscheck: ^^ it's possible we want to collapse those all into one patch | 15:04 |
| krotscheck | mordred: kk, will look | 15:04 |
| mordred | krotscheck, jeblair: but I figured I'd leave them expanded for now | 15:05 |
| krotscheck | mordred: VLDB no such entry seems to me like the volume name is wrong | 15:05 |
| *** samueldmq1 has quit IRC | 15:05 | |
| *** doug-fish has joined #openstack-infra | 15:05 | |
| *** esp has quit IRC | 15:05 | |
| mordred | krotscheck: yes. I agree with that | 15:05 |
| *** doug-fish has quit IRC | 15:05 | |
| *** doug-fish has joined #openstack-infra | 15:05 | |
| fungi | krotscheck: mordred: did it run before https://review.openstack.org/276559 and https://review.openstack.org/276587 merged and propagated? | 15:06 |
| mordred | fungi: nope - just re-ran it | 15:07 |
| *** doug-fi__ has joined #openstack-infra | 15:07 | |
| fungi | ahh, okay, so yet still more new fun broken | 15:07 |
| krotscheck | fungi: Yeah, the typos are gone | 15:07 |
| krotscheck | I'm trying to figure out how to list the volume names. | 15:07 |
| *** doug-fis_ has quit IRC | 15:07 | |
| *** annegentle has joined #openstack-infra | 15:08 | |
| mordred | mirror.wheel.trustyx64 | 15:08 |
| mordred | is what I see | 15:08 |
| krotscheck | Ah | 15:08 |
| krotscheck | mirror.wheel.trustyx64 | 15:08 |
| krotscheck | Yes that | 15:08 |
| * krotscheck goes and fixes that. | 15:08 | |
| *** arxcruz has quit IRC | 15:09 | |
| *** doug-fish has quit IRC | 15:10 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:10 | |
| openstackgerrit | Michael Krotscheck proposed openstack-infra/project-config: Corrected AFS volume name for wheel release https://review.openstack.org/276774 | 15:11 |
| *** mrmartin has quit IRC | 15:11 | |
| *** doug-fi__ has quit IRC | 15:11 | |
| *** zz_dimtruck is now known as dimtruck | 15:12 | |
| krotscheck | fungi, mordred ^^ | 15:12 |
| mordred | +A | 15:12 |
| *** jaosorior has joined #openstack-infra | 15:13 | |
| jeblair | mordred, krotscheck: o/ | 15:13 |
| mordred | look it's a jeblair! | 15:13 |
| mordred | jeblair: so - non-urgent - but I have 3 patches up for apt mirrors - would be happy to squash to one patch but figured I'd leave it expanded while you look at it | 15:14 |
| jeblair | that's totally going to be the last wheel release patch! :) | 15:14 |
| mordred | jeblair: bah. I'm sure we can find a reason for another one :) | 15:14 |
| *** amotoki has joined #openstack-infra | 15:14 | |
| mordred | jeblair: btw - my local trusty mirror that I used that config on has grown by 4G in the last week | 15:15 |
| mordred | jeblair: (that's with me not deleting unreferenced files at all) | 15:15 |
| krotscheck | jeblair: We still have to turn them on :) | 15:15 |
| jeblair | mordred: the vos release time for those may be a bit longer than elsewhere :) | 15:16 |
| mordred | jeblair: :) | 15:16 |
| jeblair | krotscheck: today! | 15:16 |
| krotscheck | jeblair: Yay! | 15:16 |
| jeblair | speaking of vos release -- the initial vos release of the pypi volume has apparently finished, i'm going to check on that before looking at mordred's patches | 15:17 |
| *** nikhil_k is now known as nikhil | 15:17 | |
| mordred | jeblair: wot! | 15:17 |
| jeblair | mordred: btw, the think i've learned is that if you expect a vos release to take a long time, run it in screen on a fileserver with localauth | 15:17 |
| jeblair | localauth tickets never expire | 15:18 |
| mordred | jeblair: that's an excellent thing to have learned | 15:18 |
| *** annegentle has quit IRC | 15:18 | |
| *** annegentle has joined #openstack-infra | 15:18 | |
| jeblair | it did in fact complete successfully, running second vos release | 15:18 |
| mordred | jeblair: so - the reason I brought up the 4G increase in size is because one of my ideas with making the apt repos bulletproof (no more "file has gone away" issues) is to never delete things from the pool | 15:19 |
| jeblair | ya | 15:19 |
| mordred | that way you get rid of the race condition beteween apt-get update and the install/upgrade | 15:19 |
| mordred | however - I htink we could go halfsies | 15:19 |
| *** rbrndt_ has joined #openstack-infra | 15:19 | |
| mordred | and do reprepro dumpunreferenced - which gives us a list of things not in the index files anymore - and then delete any that have become unreferenced more than a day ago | 15:20 |
| * mordred waves hands | 15:20 | |
| *** ildikov has quit IRC | 15:20 | |
| *** jpeeler has quit IRC | 15:20 | |
| mordred | bcause all we really care about is that an apt-get update state is good for about an hour | 15:20 |
| jeblair | mordred: maybe it would be good for me to understand exactly the different failure cases | 15:20 |
| jeblair | has somebody written those up somewhere? | 15:21 |
| mordred | jeblair: I don't think so - there's basically one main one | 15:21 |
| mordred | jeblair: the main failure case is that devstack runs apt-get update, then we vos release a new consistent snapshot, then devstack does "apt-get install foo" - but foo got a new version in the vos release from from apt-get update fetched | 15:21 |
| mordred | s/from from/from what/ | 15:22 |
| mordred | jeblair: that's because in normal operation the old version of foo would be considered 'unreferenced' by the archive and would be garbage collected from the pool | 15:22 |
| *** doug-fish has joined #openstack-infra | 15:22 | |
| jeblair | mordred: i thought the atomic updates alone would solve something -- some kind of race between the index and data... | 15:23 |
| mordred | jeblair: atomic updates without pool pruning will solve that race | 15:23 |
| mordred | jeblair: the two together will get us a 100% complete lack of that failure | 15:23 |
| jeblair | jenkins04 is having problems and hoarding a bunch of ready nodes | 15:24 |
| mordred | silly jenkins04 | 15:24 |
| *** xyang1 has joined #openstack-infra | 15:24 | |
| *** hdd has quit IRC | 15:25 | |
| jeblair | nothing is running there so i will restart it | 15:25 |
| *** keedya has quit IRC | 15:25 | |
| *** esker has quit IRC | 15:26 | |
| jeblair | (we really need to stop the nodepool commands from connecting to geard if they don't need to) | 15:26 |
| jeblair | (it makes delete all slave operations slow) | 15:26 |
| mordred | jeblair: ++ | 15:26 |
| *** esker has joined #openstack-infra | 15:26 | |
| krotscheck | Incidentally, I have a bouncer again :) | 15:26 |
| *** esikachev has quit IRC | 15:26 | |
| mordred | jeblair: but it also means that the apt pool will grow at a faster rate than we might like, given that the apt world assume a garbage collected pool. I think we can work around the unbounded growth by being slightly more clever about how we prune the pool: basically, mark files for pruning, then only delete when a file's deletion mark has aged past X hours | 15:26 |
| jeblair | (easy patch if anyone wants it ^) | 15:26 |
| mordred | krotscheck: woot | 15:26 |
| jeblair | krotscheck: yay! | 15:26 |
| jeblair | mordred: yeah, sounds like an excellent idea | 15:27 |
| jeblair | ok, jenkins04 cleaned up and restarted | 15:28 |
| mordred | jeblair: k. I have a simple script which can implement that - I'll update the patch series :) | 15:29 |
| jeblair | graphite is under unusual load | 15:29 |
| *** yamamoto has joined #openstack-infra | 15:29 | |
| mordred | jeblair: tl;dr - cron becomes "if -f unreferenced-files ; then cat unreferenced-files | reprepro deleteifunreferenced; fi ; reprepro update ; reprepro dumpunreferenced > unreferenced-files" | 15:30 |
| *** jpr has joined #openstack-infra | 15:30 | |
| *** jaosorior has quit IRC | 15:30 | |
| mordred | jeblair: so that each time we do the cron, we start by trying to delete the files that became unreferenced last time we ran the mirror cron | 15:30 |
| jeblair | mordred: and run the cron no less often that we want our delete delay interval? | 15:31 |
| mordred | yah | 15:31 |
| jeblair | er no more often | 15:31 |
| *** yamamoto has quit IRC | 15:31 | |
| *** jaosorior has joined #openstack-infra | 15:31 | |
| mordred | I tihnk we can get more cleverer with that if we want to run the mirror more frequently | 15:31 |
| tsufiev | hey there! a quick and simple question: do I need to rebase a commit failed on gate to make it consume a fix for the tests just merged or reverify is enough? | 15:31 |
| mordred | but that might take a local sqlite db or sometihng | 15:31 |
| jeblair | i'm just going to restart apache on graphite | 15:32 |
| mordred | so I think mirrors updated every couple of hours is likely fine for the gate | 15:32 |
| jeblair | ya | 15:32 |
| *** yamamoto has joined #openstack-infra | 15:32 | |
| jeblair | okay, to mordred's patches now | 15:33 |
| *** eharney has quit IRC | 15:33 | |
| *** arxcruz has joined #openstack-infra | 15:34 | |
| jeblair | mordred: one thing on 587 | 15:36 |
| *** yamamoto has quit IRC | 15:37 | |
| *** infra-red has quit IRC | 15:37 | |
| jeblair | mordred: others lgtm | 15:38 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Add python-jobs and python3-jobs to freezer* https://review.openstack.org/269138 | 15:38 |
| fungi | jeblair: mordred: the failure mode atomic updates save us from is if we generate a mirror we can make sure that the index updates don't get consumed before the files they reference are in place | 15:38 |
| jeblair | fungi: aha, so that can happen tooo | 15:38 |
| jeblair | basically, indexes can update before files appear, and files can disappear while runs are in progress. | 15:39 |
| jeblair | so atomic and don't-delete solve those respectively | 15:39 |
| fungi | another way of solving that is to update/replace the indexes last, but yeah depending on how we mirror we may also want to run consistency checks before releasing | 15:39 |
| *** bnemec has joined #openstack-infra | 15:39 | |
| fungi | of solving the first failure mode i mean | 15:39 |
| jeblair | fungi: i think mordred does do consistency checks in his update script | 15:39 |
| fungi | perfect | 15:40 |
| *** amotoki has quit IRC | 15:40 | |
| jeblair | at least, there's a 'reprepro check' command; assume it does something like that :) | 15:40 |
| fungi | i am going to go assume some more coffee (or is that subsume?). brb | 15:40 |
| mordred | fungi: yes, we should do all th ethings | 15:42 |
| *** doug-fish has quit IRC | 15:42 | |
| mordred | jeblair: I have updated the patch - do you prefer these as a series? or you want me to squash them? | 15:42 |
| *** piet has quit IRC | 15:42 | |
| *** tonytan4ever has joined #openstack-infra | 15:42 | |
| *** yamamoto has joined #openstack-infra | 15:42 | |
| jeblair | mordred: no pref, though they may land faster as one :) | 15:42 |
| mordred | k. I'm going to squash | 15:43 |
| *** piet has joined #openstack-infra | 15:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/276796 | 15:43 |
| mordred | jeblair: thereyago | 15:44 |
| mordred | fungi, krotscheck: ^^ | 15:44 |
| *** yamamoto has quit IRC | 15:44 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/272587 | 15:45 |
| mordred | oops. messed up the change-id last time | 15:45 |
| dims | jeblair : when i add a new script in project-config (run-tox-with-oslo-master.sh) that should end up in /usr/local/jenkins/slave_scripts/ is that script only available when new images are built? | 15:45 |
| mordred | dims: yes | 15:46 |
| jeblair | mordred: how long does an update take? | 15:46 |
| dims | mordred : thanks | 15:46 |
| mordred | jeblair: an update I just ran took 1.7 seconds | 15:46 |
| jeblair | that is less than 30m | 15:47 |
| mordred | jeblair: but then I had previously run an update 10 mintes or so ago | 15:47 |
| mordred | jeblair: the one I ran that was a week old also did not take 30m | 15:47 |
| *** david-lyle has joined #openstack-infra | 15:47 | |
| mordred | but I did not run time on it :( | 15:47 |
| jeblair | okay, sounds like we're probably good. | 15:47 |
| dims | sdague : quick one-liner (adding check-requirements to octavia) please if you get a chance - https://review.openstack.org/#/c/276711/ | 15:48 |
| AJaeger | dims, your change merged at 12:21, so today's images should have it. | 15:48 |
| AJaeger | dims: they are rebuild starting at 14:00 UTC | 15:48 |
| dims | AJaeger : not yet it seems - http://logs.openstack.org/03/254703/35/experimental/periodic-nova-py27-with-oslo-master/bd3b488/console.html#_2016-02-05_15_39_54_975 | 15:49 |
| jeblair | mordred: you changed the bandersnatch interval to every 2 minutes | 15:49 |
| jeblair | er | 15:49 |
| jeblair | mordred: you changed it to every 2 hours | 15:49 |
| mordred | jeblair: doh | 15:50 |
| jeblair | mordred: comments left -- i think maybe you swapped that part of the cron defn | 15:50 |
| mordred | yup | 15:50 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/272587 | 15:50 |
| mordred | jeblair: I believe that should be better :) | 15:50 |
| mordred | jeblair: actually - perhaps we should break this in to two patches - one without the cron and one with | 15:51 |
| mordred | jeblair: so that we can land all th ethings, do a manual run and release in screen | 15:51 |
| mordred | jeblair: then land the cron change to update it | 15:51 |
| fungi | that's a great incremental plan | 15:52 |
| krotscheck | mordred: Ok, lemme run puppet on that | 15:52 |
| jeblair | mordred: ok -- i also have a trick of "flocking bash" to easily avoid the cron | 15:52 |
| jeblair | either way | 15:52 |
| mordred | jeblair: ah - we can just do that too | 15:52 |
| krotscheck | Passes puppet | 15:53 |
| mordred | so, I guess, really we just need to create the volumes, principal and keytab and put the keytab in hiera then this should be ready to go ish :) | 15:53 |
| *** doug-fish has joined #openstack-infra | 15:53 | |
| jeblair | mordred: yup; you want to do that or shall i? | 15:54 |
| jeblair | mordred: (oh, also, add the principal to pts, create a group in pts, add the principal to the group, and give the group 'write' perms on the volume) | 15:54 |
| *** amotoki has joined #openstack-infra | 15:54 | |
| *** gnuoy_ has joined #openstack-infra | 15:54 | |
| mordred | jeblair: if it's easy for you to, that would likely be quicker | 15:55 |
| jeblair | mordred: can do | 15:55 |
| mordred | jeblair: I'll work on a patch to configure-mirror to consume them | 15:55 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: remove octavia from requirements sync list https://review.openstack.org/276801 | 15:55 |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Restore Octavia to the requirements sync list https://review.openstack.org/276802 | 15:55 |
| AJaeger | jeblair, fungi, dims: Last build for bare-trusty seems to have been 28th of January, see http://nodepool.openstack.org/rax-dfw.bare-trusty.log | 15:55 |
| jeblair | AJaeger: yeah, nodepool is broken | 15:56 |
| AJaeger | dims: that explains your problem ^ | 15:56 |
| *** zeih has quit IRC | 15:56 | |
| greghaynes | jeblair: clarkb Are we aware of http://logs.openstack.org/84/272384/8/check/gate-dsvm-nodepool/97df99d/logs/screen-nodepool.txt.gz ? | 15:57 |
| greghaynes | seems like something has broken nodepool gate | 15:57 |
| greghaynes | er, not gate since it is non voting, nodepool dsvm | 15:57 |
| jeblair | greghaynes: no -- i wonder if mordred, Shrews, or notmorgan can easily identify that ^ | 15:58 |
| notmorgan | jeblair: oh hai | 15:58 |
| jeblair | mordred, krotscheck: 2nd vos release (2 days of updates) took 39m. running 3rd now. | 15:59 |
| *** gnuoy_ has quit IRC | 15:59 | |
| mordred | jeblair: cool | 15:59 |
| Shrews | jeblair: nope. something from novaclient itself | 15:59 |
| *** bnemec has quit IRC | 15:59 | |
| krotscheck | jeblair, mordred: That's improving :) | 15:59 |
| *** annegent_ has joined #openstack-infra | 15:59 | |
| jeblair | mordred, krotscheck: 3rd vos release (no updates) took 38s. | 16:00 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 16:00 |
| notmorgan | jeblair: uhmm ooh | 16:00 |
| mordred | jeblair: woot! | 16:00 |
| notmorgan | neat error | 16:00 |
| fungi | dhellmann: looking back at the last release announcement job log, it looks like maybe it had a somewhat silent failure (see where get-repo-owner errors): http://logs.openstack.org/7b/7bced8a36a82415f9075f3730897fab834fbeb4b/release/python-keystoneclient-announce-release/a7fb2d0,3/console.html#_2016-02-05_01_30_29_165 maybe it's trying to use the job name instead of the repo name? i definitely passed | 16:00 |
| mordred | Shrews: are we seeing that in shade too I wonder? | 16:00 |
| fungi | --project openstack/python-keystoneclient when i triggered it too, just double-checked | 16:00 |
| dims | AJaeger : ah thanks | 16:00 |
| *** bnemec has joined #openstack-infra | 16:00 | |
| *** tongli has quit IRC | 16:00 | |
| notmorgan | mordred: fwiw, we did land novaclient -> ksa [but i don't think a release has happened], occ patch is still pending | 16:00 |
| mordred | notmorgan: awesome | 16:00 |
| notmorgan | but that doesn't look like it would be directly related | 16:01 |
| mordred | no - this is something about nw_info entry - which is not a thing we do anything with | 16:01 |
| *** tongli has joined #openstack-infra | 16:02 | |
| *** TheJulia_ has joined #openstack-infra | 16:02 | |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Revert "Temporarily stop releasing after bandersnatch" https://review.openstack.org/275300 | 16:02 |
| jeblair | mordred, fungi, krotscheck: ^ that's ready to merge | 16:02 |
| *** jsavak has quit IRC | 16:02 | |
| *** jcoufal has quit IRC | 16:02 | |
| *** aarefiev_ has joined #openstack-infra | 16:03 | |
| *** NobodyCa1 has joined #openstack-infra | 16:03 | |
| *** jsavak has joined #openstack-infra | 16:03 | |
| *** HeOS__ has joined #openstack-infra | 16:03 | |
| *** _nadya__ has joined #openstack-infra | 16:03 | |
| krotscheck | jeblair: +1 | 16:03 |
| *** annegent_ has quit IRC | 16:03 | |
| *** annegentle has quit IRC | 16:04 | |
| *** TheJulia has quit IRC | 16:04 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 16:04 |
| jeblair | mordred: oh, i just thought of one more suggestion for https://review.openstack.org/272587 | 16:04 |
| *** annegentle has joined #openstack-infra | 16:04 | |
| jeblair | mordred: left comment | 16:04 |
| openstackgerrit | Merged openstack-infra/project-config: Add check requirements job to octavia https://review.openstack.org/276711 | 16:04 |
| *** ccrouch has quit IRC | 16:04 | |
| *** ccrouch has joined #openstack-infra | 16:04 | |
| daemontool | AJaeger, re: https://review.openstack.org/#/c/269138/ I'll follow the same instructions from you for the others Depends-On commits | 16:05 |
| greghaynes | http://logs.openstack.org/84/272384/8/check/gate-dsvm-nodepool/97df99d/logs/screen-n-api.txt.gz#_2016-02-05_05_24_58_669 is the nw_info cache error on the server side | 16:05 |
| greghaynes | so it seems like a devstack breakage | 16:05 |
| *** cburgess_ has joined #openstack-infra | 16:05 | |
| *** ccrouch has quit IRC | 16:05 | |
| *** TheJulia_ is now known as TheJulia | 16:05 | |
| *** ccrouch has joined #openstack-infra | 16:05 | |
| *** imcsk8 is now known as imcsk8|vsTheGvt | 16:05 | |
| prometheanfire | mordred: mind taking a look at https://review.openstack.org/#/c/276610/ :D | 16:06 |
| *** ifarkas_ has joined #openstack-infra | 16:06 | |
| *** gnuoy_ has joined #openstack-infra | 16:06 | |
| AJaeger | daemontool: yes, please. Just wanted to review once ;) | 16:07 |
| *** gnuoy_ has quit IRC | 16:07 | |
| rm_work | I *felt* like something was wonky with Octavia not checking against global-req | 16:08 |
| rm_work | sorry guys | 16:08 |
| *** jaosorior has quit IRC | 16:09 | |
| prometheanfire | or any other glean core really | 16:09 |
| prometheanfire | with octavia replacing lbaas-v1 and being in the state it's in, is lbaasv1 being depricated to fast? | 16:10 |
| *** HeOS has quit IRC | 16:11 | |
| *** timrc has joined #openstack-infra | 16:12 | |
| rm_work | prometheanfire: octavia doesn't replace lbaasv1 | 16:12 |
| *** fbo_ has joined #openstack-infra | 16:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/272587 | 16:12 |
| rm_work | lbaasv2 does | 16:12 |
| rm_work | which is still inside neutron-lbaas | 16:12 |
| mordred | prometheanfire: basically nothing should use lbaasv1 | 16:12 |
| mordred | prometheanfire: also, I will look at your patch (still catching up from being out) | 16:13 |
| sigmavirus24 | dims: dhellmann I updated the twine PR | 16:13 |
| *** austin81 has joined #openstack-infra | 16:13 | |
| prometheanfire | mordred: thanks | 16:13 |
| mordred | prometheanfire: lgtm! | 16:13 |
| *** bnemec has quit IRC | 16:13 | |
| prometheanfire | rm_work: yes, I know lbaasv2, but atm octavia is the only implimentation I know of | 16:14 |
| prometheanfire | mordred: thanks again :D | 16:14 |
| *** timrc_ has quit IRC | 16:14 | |
| *** ldnunes has quit IRC | 16:14 | |
| *** ifarkas has quit IRC | 16:14 | |
| *** _nadya_ has quit IRC | 16:14 | |
| *** HeOS_ has quit IRC | 16:14 | |
| *** cburgess has quit IRC | 16:14 | |
| *** aarefiev has quit IRC | 16:14 | |
| *** NobodyCam has quit IRC | 16:14 | |
| *** fbo has quit IRC | 16:14 | |
| *** jlanoux_ has joined #openstack-infra | 16:14 | |
| dims | sigmavirus24 : +1 to https://github.com/sigmavirus24/twine/commit/15fdc9f0153fe459cb9468b65bfd60559b5d1e44 | 16:14 |
| rm_work | prometheanfire: the haproxynamespace driver still exists i believe | 16:15 |
| *** Jeffrey4l has quit IRC | 16:15 | |
| rm_work | but deprecating THAT should be your question | 16:15 |
| *** jpeeler has joined #openstack-infra | 16:15 | |
| *** jpeeler has quit IRC | 16:15 | |
| *** jpeeler has joined #openstack-infra | 16:15 | |
| sigmavirus24 | dims: do you mean https://github.com/sigmavirus24/twine/commit/f29c18d1127e6b0d8fafa37e0ab1b007e3e1d7a8 ? | 16:15 |
| prometheanfire | rm_work: it is | 16:15 |
| *** jlanoux has quit IRC | 16:15 | |
| rm_work | yeah, the distinction is that "lbaasv1->lbaasv2" should be fine, "haproxynamespace->octavia" *should* be fine but is really what you might be concerned with | 16:16 |
| *** rkukura has quit IRC | 16:16 | |
| *** jsavak has quit IRC | 16:16 | |
| *** rkukura has joined #openstack-infra | 16:17 | |
| prometheanfire | as octavia is the only implimentation of lbaasv2 I know of, I'm concerned about the early deprication of lbaasv1 | 16:17 |
| prometheanfire | that work for you? | 16:17 |
| fungi | clarkb: we got a support ticket from rackspace that there was a problem on wednesday with the host serving elasticsearch07. any idea if it had any impact? | 16:17 |
| mordred | prometheanfire: I thnk the problem is that v1 to v2 is known to be incompatible so we're trying to communicate as quickly as possible that new things should not deploy v1 under any circumstances | 16:18 |
| *** jsavak has joined #openstack-infra | 16:18 | |
| rm_work | i'd say "I'm concerned about the early deprecation of haproxynamespace for lbaasv2" | 16:18 |
| rm_work | well, i wouldn't say that | 16:18 |
| rm_work | but you can :P | 16:18 |
| prometheanfire | mordred: I get that | 16:18 |
| rm_work | i would NOT be worried about deprecating lbaasv1 | 16:18 |
| *** esikachev has joined #openstack-infra | 16:19 | |
| prometheanfire | one of the concerns I have is that the VMs running octavia have access to the management network and also are not updatable (in place) | 16:19 |
| rm_work | as long as lbaasv2 has the haproxynamespace driver, it is *just better* | 16:19 |
| *** ldnunes has joined #openstack-infra | 16:19 | |
| rm_work | (not that haproxynamespace is good...) | 16:19 |
| *** vgridnev has quit IRC | 16:19 | |
| prometheanfire | ya, don't really like that either | 16:19 |
| rm_work | but it's equivilent to what lbaasv1 has | 16:19 |
| rm_work | it's ... the same driver | 16:20 |
| prometheanfire | so, guess my question is how quickly is haproxynamespace for lbaasv2 going away? | 16:20 |
| rm_work | the plan was "soon™" | 16:20 |
| rm_work | but now you're asking the right question :P | 16:21 |
| rm_work | I can verify | 16:21 |
| *** annegentle has quit IRC | 16:21 | |
| *** EricGonc_ has joined #openstack-infra | 16:21 | |
| *** armax has joined #openstack-infra | 16:21 | |
| fungi | infra-root: i also see a ticket warning of disruptive block storage maintenance in iad on february 16. that could cause our mirror server to drop its afs cache filesystem or mark it read-only | 16:21 |
| prometheanfire | security concerns about octavia prevent it's use | 16:21 |
| rm_work | yeah, we're working on that | 16:22 |
| rm_work | but it is a whole new thing, so i can totally see that | 16:22 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-jenkins: Use default ruby packages for Xenial https://review.openstack.org/276811 | 16:22 |
| *** jlanoux_ has quit IRC | 16:22 | |
| mmedvede | yolanda: any advise on ^? Want to make puppet-jenkins usable for Xenial slaves | 16:22 |
| *** david-lyle has quit IRC | 16:23 | |
| rm_work | prometheanfire: sounds like maybe the answer is "we want it out of tree, but we aren't deprecating HaproxyNamespace for lbaasv2" | 16:23 |
| rm_work | so you may be fine | 16:23 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: Creator's Guide: Add some pointer https://review.openstack.org/276813 | 16:23 |
| *** dimtruck is now known as zz_dimtruck | 16:23 | |
| prometheanfire | rm_work: ok | 16:23 |
| prometheanfire | rm_work: if you have an issue/review I can pay attention to for that it would be nice | 16:24 |
| *** annegentle has joined #openstack-infra | 16:24 | |
| *** EricGonczer_ has quit IRC | 16:24 | |
| fungi | infra-root: and then again on the 18th, network maintenance which could also disrupt block storage access in iad | 16:24 |
| rm_work | prometheanfire: for octavia being secure, or for haproxynamespace? :P | 16:24 |
| prometheanfire | octavia | 16:25 |
| rm_work | are you in #openstack-lbaas ? | 16:25 |
| prometheanfire | I do want to use it, but just seems half baked | 16:25 |
| prometheanfire | didn't know lbaas had a specific channel | 16:25 |
| prometheanfire | will be soon | 16:25 |
| rm_work | lots of us working very actively on it :P | 16:25 |
| openstackgerrit | Doug Fish proposed openstack-infra/project-config: Add JavaScript gate jobs to LBaaS dashboard https://review.openstack.org/272771 | 16:25 |
| jeblair | fungi: it definitely could do that :) | 16:26 |
| jeblair | fungi: our cache is really small so far, we could move it to local storage (temporarily?) | 16:26 |
| fungi | that might be less disruptive than the capacity loss from downing iad in nodepool for 3-4 days | 16:27 |
| jeblair | fungi: otoh, if something happens, it's easy to blow away the cache and restart | 16:28 |
| fungi | true. worst case a bunch of jobs fail because they can't install packages | 16:28 |
| fungi | and we'll likely catch that fairly quickly | 16:28 |
| *** david-lyle has joined #openstack-infra | 16:29 | |
| jeblair | fungi: (also, i don't actually know openafs behavior when its cache goes read-only :) | 16:29 |
| fungi | perhaps this is an opportunity to find out | 16:30 |
| fungi | maybe it falls back to all cache misses? | 16:30 |
| fungi | that would be really exciting | 16:30 |
| jeblair | maybe | 16:30 |
| *** esikachev has quit IRC | 16:30 | |
| jeblair | what does the path look like for an apt mirror? | 16:31 |
| jeblair | i'm trying to figure out the volume layout... | 16:32 |
| fungi | it can vary widely | 16:32 |
| jeblair | do we want something like /afs/openstakck.org/mirror/apt/ubuntu/... or something? | 16:32 |
| *** esikachev has joined #openstack-infra | 16:32 | |
| mordred | jeblair: nah - we want a single one | 16:32 |
| mordred | jeblair: because of how the pool works | 16:32 |
| *** zz_dimtruck is now known as dimtruck | 16:33 | |
| fungi | for example this is an apt repository with one package all in one directory http://fungi.yuggoth.org/debian/cool-retro-term/ | 16:33 |
| mordred | jeblair: http://paste.openstack.org/show/486113 | 16:33 |
| mordred | jeblair: that's what the data dir is going to look like | 16:33 |
| fungi | the layout is pretty flexible | 16:33 |
| jeblair | mordred: so that's what /afs/.openstack.org/mirror/apt | 16:33 |
| jeblair | will hold? | 16:33 |
| mordred | jeblair: yes | 16:33 |
| fungi | better question is probably what reprepro will expect/generate | 16:33 |
| jeblair | mordred: and that can contain future debian or other distros? | 16:34 |
| mordred | jeblair: yup | 16:34 |
| jeblair | mordred: becauese they just show up in dists/ | 16:34 |
| prometheanfire | mordred: if you have time to look at dib patches... https://review.openstack.org/276578 https://review.openstack.org/276581 | 16:34 |
| mordred | yes - and they reference files in the pool | 16:34 |
| jeblair | dists/trusty ; dists/jesse ; dists/sid ? | 16:34 |
| mordred | yup! | 16:34 |
| jeblair | nifty | 16:34 |
| *** harlowja_at_home has joined #openstack-infra | 16:34 | |
| *** tongli has quit IRC | 16:35 | |
| mordred | and if trusty and jesse happen to share a package because ubuntu did a binary sync and then didn't change anything | 16:35 |
| mordred | we'll only have one copy of it | 16:35 |
| *** dimtruck is now known as zz_dimtruck | 16:35 | |
| fungi | right, ever since debian switched its official repository layout to be pool-based (can't remember if that was for sarge?) the dists trees are just reference metadata and all the debs themselves are now in a common pool | 16:35 |
| jeblair | so smart | 16:36 |
| fungi | there is a slight risk that debian and ubuntu have packages with exactly the same filename but different contents | 16:36 |
| openstackgerrit | Merged openstack-infra/system-config: add new vhost for releases.openstack.org https://review.openstack.org/266510 | 16:36 |
| *** piet has quit IRC | 16:36 | |
| fungi | so i'm not convinced it's 100% safe to share pools across different derivative distros, but i have a feeling ubuntu attempt to avoid that situation since they also direct import lots of packages from debian/testing anyway | 16:37 |
| *** piet has joined #openstack-infra | 16:37 | |
| *** e0ne has quit IRC | 16:37 | |
| mordred | fungi: yah - I believe if that happened it would be considered a massive bug | 16:38 |
| fungi | but yeah, the implementation was mostly because there were lots of duplicate files between unstable/stable/oldstable (don't remember if testing existed yet) and the repository size had reached the point where deduplication was going to save lots of space | 16:38 |
| mordred | fungi: and against policy | 16:38 |
| jeblair | mordred: quota for mirror.apt? | 16:38 |
| mordred | jeblair: my local trusty mirror with this setup is currently 90G | 16:39 |
| jeblair | mordred: wow. pypi is 2.5 times larger. | 16:39 |
| fungi | mordred: is that just amd64 plus arch-indep, or all supported architectures? | 16:40 |
| fungi | source packages too? | 16:40 |
| mordred | jeblair: it helps that trusty has one version for every package | 16:40 |
| *** jsavak has quit IRC | 16:40 | |
| mordred | fungi: I did not mirror source packages - just amd64 - but I believe mirroring amd64 with reprepro gets arch indep as well | 16:40 |
| mordred | (basically I ony mirrored what we consume in the gate) | 16:41 |
| mordred | yes | 16:41 |
| mordred | ./pool/main/r/requests/python-requests_2.2.1-1_all.deb | 16:41 |
| mordred | that includes arch indep | 16:41 |
| *** jsavak has joined #openstack-infra | 16:41 | |
| jeblair | mordred: so maybe 300G for starters? | 16:41 |
| fungi | yeah, i expect it would also pick up arch-indep. just making sure we're not mirroring all architectures (because while it's not terrible for ubuntu's limited set, when we start mirroring debian that will get ginormous) | 16:41 |
| mordred | jeblair: sounds great! | 16:41 |
| mordred | fungi: yah - I think mirroring for architectures we don't run is ... well ... let's not do it :) | 16:42 |
| *** shardy has joined #openstack-infra | 16:43 | |
| fungi | however, if we end up with an aarch64/arm64 cloud at some point, as has been tentatively offered to us recently, it shouldn't be hard to pick a cloud-specific architecture (or list of architectures) to mirror | 16:43 |
| mordred | fungi: it would be a very simple patch to add ... | 16:43 |
| fungi | yep | 16:44 |
| mordred | fungi: https://review.openstack.org/#/c/272587/5/modules/openstack_project/templates/reprepro/distributions.erb line 5 (and associated lines) | 16:44 |
| fungi | very cool. i've not used reprepro before, but assumed it was something along those lines to tell it what we'd want | 16:44 |
| *** yamamoto has joined #openstack-infra | 16:45 | |
| mordred | yup. super simple and whatnot | 16:45 |
| *** samuelBartel has quit IRC | 16:46 | |
| *** tphummel has joined #openstack-infra | 16:46 | |
| openstackgerrit | Imre Farkas proposed openstack-infra/project-config: Add openstack/python-wsmanclient project https://review.openstack.org/269122 | 16:46 |
| fungi | mordred: now that you're around a little more, strange situation i've noticed this week with ansible inventory | 16:47 |
| jeblair | mordred: oh, i think your patch may be missing the site.pp bit to include the key from hiera? | 16:48 |
| openstackgerrit | Merged openstack-infra/project-config: Corrected AFS volume name for wheel release https://review.openstack.org/276774 | 16:48 |
| mordred | fungi: neat! | 16:48 |
| mordred | jeblair: oh - yeah | 16:48 |
| zaro | morning | 16:48 |
| fungi | mordred: from time to time, it appears that our ansible-inventory.cache ends up with all of dfw identified by instance uuids instead of hostnames, but then on the next cache update it's back to hostnames again. wreaking havoc with puppetdb reports | 16:48 |
| *** scheuran has quit IRC | 16:49 | |
| mordred | hrm | 16:49 |
| mordred | fungi: that seems exceptionally odd | 16:49 |
| fungi | i agree | 16:49 |
| mordred | fungi: I will investigate | 16:49 |
| fungi | i've seen it twice this week so far | 16:49 |
| fungi | moving the wonky cache out of the way and letting it regenerate fixes it | 16:49 |
| *** esikachev has quit IRC | 16:49 | |
| jeblair | mordred: i have created the principal, added it to pts, created the group, added the principal to the group, added the principal's keytab to hiera, created the volume, made ro replicas, released them, mounted the volume, and given the group write perms at the root of the volume | 16:50 |
| *** calebb1 has joined #openstack-infra | 16:50 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/272587 | 16:50 |
| mordred | jeblair: neat! | 16:50 |
| jeblair | mordred: i think that's everything; so i think your patch is ready to land when you ... update it :) | 16:50 |
| *** calebb has quit IRC | 16:50 | |
| mordred | jeblair: I have updated the puppet patch | 16:50 |
| krotscheck | Typo merged. It'll likely be on the release slave shortly after the hour. | 16:50 |
| jeblair | krotscheck: roger | 16:51 |
| *** yamamoto has quit IRC | 16:51 | |
| *** bpokorny has joined #openstack-infra | 16:51 | |
| *** calebb1 is now known as calebb | 16:51 | |
| mordred | jeblair: what is your trick for preventing the cron from happening? | 16:51 |
| mordred | jeblair: is it just to flock in the screen session? | 16:51 |
| jeblair | mordred: inside of screen, i run : flock -n /var/run/bandersnatch/mirror.lock bash | 16:51 |
| mordred | neat | 16:51 |
| jeblair | mordred: adjust logfile path as necessary | 16:51 |
| mordred | jeblair: I'm going to start a screen (or join yours if you prefer) and do that | 16:52 |
| jeblair | mordred: then i usually run "cat" to leave myself a message telling myself that's what that session is doing, and then go do work in a different window | 16:52 |
| *** hdd has joined #openstack-infra | 16:52 | |
| *** NobodyCa1 is now known as NobodyCam | 16:52 | |
| *** esikachev has joined #openstack-infra | 16:52 | |
| mordred | ah | 16:52 |
| openstackgerrit | Alistair Coles proposed openstack-infra/project-config: Add job gate-swift-tox-func-in-process-fast-post https://review.openstack.org/276823 | 16:52 |
| jeblair | mordred: i don't have anything going on there, so you can start your own | 16:52 |
| *** e0ne has joined #openstack-infra | 16:52 | |
| jeblair | mordred: (i do the cat thing so i don't accidentally close it) | 16:53 |
| *** annegentle has quit IRC | 16:53 | |
| *** mikelk has quit IRC | 16:53 | |
| *** annegentle has joined #openstack-infra | 16:53 | |
| *** gyee has joined #openstack-infra | 16:55 | |
| jeblair | anyone else want to review https://review.openstack.org/272587 ; if not, i'll go ahead and approve | 16:55 |
| fungi | you can also name screen windows | 16:55 |
| fungi | i usually do that so i can keep track of what they're doing | 16:56 |
| mordred | okie | 16:56 |
| mordred | I've got the lock | 16:56 |
| jeblair | fungi: that's a good idea, but it probably wouldn't be enough to overcome my instinctive "oh look, a prompt, i'll start typing in it" :) | 16:56 |
| mordred | so the patch should be safe to land | 16:56 |
| jeblair | tell you what, i went ahead and approved https://review.openstack.org/272587 -- its had lots of eyes and testing. but it will still take a long time to wind through the gate, so there's still plenty of time for folks to look at it if they want. | 16:57 |
| *** esikachev has quit IRC | 16:58 | |
| dims | AJaeger : fungi : any tips on figuring out the nodepool/images issue? | 16:59 |
| greghaynes | Is the nodepool/images issue that we haven't had images uploaded in a while? | 17:00 |
| greghaynes | if so, https://review.openstack.org/#/c/275997/ would be nice to merge | 17:00 |
| jeblair | greghaynes: yes | 17:00 |
| dims | greghaynes : yes | 17:00 |
| *** sputnik13 has joined #openstack-infra | 17:00 | |
| *** amaretskiy has quit IRC | 17:01 | |
| *** sputnik13 has quit IRC | 17:02 | |
| fungi | dims: not really, i've been a bit out of touch. was gone last week and this week has been full of insane | 17:03 |
| *** stevemar has joined #openstack-infra | 17:03 | |
| fungi | so i'm not certain what exactly is broken with image updates | 17:04 |
| greghaynes | fungi: everything ;) | 17:04 |
| dims | fungi : :) no worries | 17:04 |
| prometheanfire | greghaynes: mind looking at those optimistic patches from last night? https://review.openstack.org/276578 https://review.openstack.org/276581 https://review.openstack.org/276610 | 17:06 |
| prometheanfire | don't think you have glean core | 17:06 |
| prometheanfire | but the last link is that | 17:06 |
| prometheanfire | needs one more +2 | 17:06 |
| cbader | clarkb, I am seeing an issue with your patch https://review.openstack.org/#/c/271541/5/templates/nodepool.default.erb where I am getting --no-builder injected into /etc/default/nodepool I can't seem to figure out why this is happening any clue? | 17:07 |
| *** matrohon has quit IRC | 17:09 | |
| mordred | jeblair: I've got to go return the truck - when I get back, I'll kick off the initial mirror build if the patch has made its wy through the gate by then | 17:09 |
| *** ildikov has joined #openstack-infra | 17:10 | |
| *** hdd has quit IRC | 17:10 | |
| jeblair | mordred: sounds good | 17:11 |
| *** jsavak has quit IRC | 17:11 | |
| fungi | cbader: is it possible you're deploying an older version of nodepool than we are? | 17:11 |
| *** jsavak has joined #openstack-infra | 17:13 | |
| cbader | fungi, I have the latest which has this patch, I have two other CI systems that have older version which don't have the patch and they are working Clarkb introduced this value --no-builder in the patch, just wondering what for. | 17:13 |
| fungi | cbader: that option seems to have been added to nodepool in https://review.openstack.org/232251 which merged on january 14 | 17:13 |
| *** _nadya__ has quit IRC | 17:14 | |
| cbader | fungi, my nodepoold fails with argument --no-builder unknown | 17:14 |
| *** tongli has joined #openstack-infra | 17:14 | |
| *** david-lyle has quit IRC | 17:14 | |
| fungi | cbader: that suggests it's an older codebase than the state of nodepool's master branch from ~3 weeks ago | 17:14 |
| *** achanda has joined #openstack-infra | 17:15 | |
| dhellmann | fungi : yeah, I'm going to dig into that announcement job. I think it's using directory name to try to find the repo name. I also didn't see the email go through to the list. I'll see if I can add some debugging to the script to show what it's actually doing. | 17:15 |
| *** qb_ has quit IRC | 17:15 | |
| dhellmann | fungi: I see that releases.openstack.org shows a directory listing. Do I just need to trigger the docs build for the releases repo to update that, or are there more steps you need to handle first? | 17:15 |
| dhellmann | (I just don't want to do it too soon) | 17:16 |
| fungi | dhellmann: yeah, is the job updated to publish there instead of docs.o.o now? | 17:16 |
| dhellmann | fungi : ah, this needs to land first I think: https://review.openstack.org/#/c/266515/ | 17:16 |
| *** harlowja_at_home has quit IRC | 17:17 | |
| *** stevemar has quit IRC | 17:17 | |
| fungi | dhellmann: approved and winding its way through the tubes now | 17:17 |
| *** stevemar has joined #openstack-infra | 17:18 | |
| *** tonytan4ever has quit IRC | 17:18 | |
| dhellmann | fungi : thanks! I'll queue up some doc changes to trigger a build when that's done | 17:18 |
| *** ianychoi has joined #openstack-infra | 17:18 | |
| fungi | awesome. i need to bug out here in a bit to help my wife take some photos of a house. she just started officially working as a real estate agent at the beginning of the week and she already has someone who wants her to sell one but still needs to learn her way around a camera that isn't an iphone | 17:20 |
| fungi | tripod? remote bulb? what are those? | 17:20 |
| *** david-lyle has joined #openstack-infra | 17:21 | |
| *** jlanoux has joined #openstack-infra | 17:22 | |
| *** stevemar has quit IRC | 17:22 | |
| *** matrohon has joined #openstack-infra | 17:24 | |
| *** david-lyle has quit IRC | 17:25 | |
| *** hdd has joined #openstack-infra | 17:25 | |
| mfisch | Can someone tell me how I can subscribe to the [tc] tag and in openstack-dev? | 17:25 |
| mfisch | the ML options dont have it that I see | 17:25 |
| *** Daisy has quit IRC | 17:26 | |
| *** lucasagomes is now known as lucas-dinner | 17:27 | |
| dhellmann | fungi : thanks for your help as always; tell kris congratulations, too | 17:27 |
| *** jsavak has quit IRC | 17:28 | |
| *** jsavak has joined #openstack-infra | 17:29 | |
| fungi | mfisch: they're spelled out by name... it's probably "technical committee" there? | 17:29 |
| fungi | or it may not be an official tag | 17:29 |
| fungi | looking | 17:29 |
| mfisch | its not there as TC or spelled out | 17:29 |
| dhellmann | fungi : does someone actually audit the list of tags in mailman compared to what folks use on the list? | 17:30 |
| *** mriedem is now known as mriedem_afk | 17:31 | |
| fungi | mfisch: yeah, i don't see it there either so it's presumably not a real subject tag | 17:31 |
| *** vgridnev has joined #openstack-infra | 17:31 | |
| mfisch | fungi: ok, this one was also sent with [all] so I did sub to that | 17:31 |
| krotscheck | jeblair: I suspect that that change has landed on the release slave, but puppetboard has been unresponsive for the last half hour | 17:31 |
| fungi | dhellmann: not to my knowledge, no. if someone wants an ml tag added to mailman they just ask the list admin to do it | 17:31 |
| *** annegentle has quit IRC | 17:31 | |
| dhellmann | fungi : ok | 17:31 |
| jeblair | krotscheck: checking | 17:31 |
| jeblair | krotscheck: yes | 17:32 |
| *** tpsilva has joined #openstack-infra | 17:32 | |
| krotscheck | Done! | 17:32 |
| krotscheck | WOO | 17:32 |
| krotscheck | It works! | 17:32 |
| jeblair | yay! | 17:32 |
| *** annegentle has joined #openstack-infra | 17:32 | |
| jeblair | krotscheck: so i think we're at: switch on the wheel mirror ? (!!!?!?!?!) | 17:32 |
| *** jsavak has quit IRC | 17:32 | |
| krotscheck | ....I do believe we are! | 17:33 |
| *** apoorvad has joined #openstack-infra | 17:33 | |
| krotscheck | jeblair, fungi, mordred, sdague ^^ | 17:33 |
| krotscheck | https://review.openstack.org/#/c/267117/ | 17:33 |
| *** rkukura_ has joined #openstack-infra | 17:33 | |
| jeblair | krotscheck: oh that's an image update change :( | 17:33 |
| jeblair | krotscheck: i went ahead and approved it, but we'll need image updates for it to take effect, and nodepool is still broken | 17:34 |
| *** haleyb has quit IRC | 17:34 | |
| *** rkukura has quit IRC | 17:34 | |
| *** rkukura_ is now known as rkukura | 17:34 | |
| krotscheck | jeblair: Who's working on unbreaking nodepool? | 17:34 |
| jeblair | krotscheck: perhaps in the interim, we should do a do-not-merge devstack-gate change that bashes those values into place to make sure everything's working | 17:35 |
| jeblair | krotscheck: greghaynes and clarkb mostly, with an occasional assist from me | 17:35 |
| openstackgerrit | Merged openstack-infra/project-config: publish from openstack/releases to releases.openstack.org https://review.openstack.org/266515 | 17:35 |
| *** flepied has quit IRC | 17:35 | |
| *** tphummel has quit IRC | 17:35 | |
| krotscheck | Gotcha | 17:36 |
| krotscheck | jeblair: How about we resurrect your old patch? | 17:36 |
| *** jsavak has joined #openstack-infra | 17:37 | |
| jeblair | krotscheck: yeah.. do you know what the AFS_SLUG will be? | 17:37 |
| fungi | krotscheck: this calls for a #success once it's getting hit by jobs and passing | 17:37 |
| jeblair | krotscheck: (because that script probably won't be there) | 17:37 |
| *** unicell has quit IRC | 17:37 | |
| jeblair | fungi: probably the pypi-in-afs mirrors too | 17:37 |
| krotscheck | jeblair: if it's running on trusty, it should always be ubuntu-14.04-x86_64 | 17:37 |
| *** annegentle has quit IRC | 17:38 | |
| jeblair | krotscheck: patch in progress | 17:39 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:39 | |
| fungi | jeblair: yep, that's already in production right? | 17:39 |
| fungi | anyway, i need to head out for a couple hours. back in a while | 17:39 |
| jeblair | fungi: yes | 17:39 |
| clarkb | jeblair: I think it happens on boot in the ready script, not an image update thing | 17:40 |
| jeblair | clarkb: ready scripts are in the image | 17:40 |
| krotscheck | jeblair: Beat me to it! | 17:40 |
| clarkb | oh right it doesn't copy them over like a snapshot build | 17:40 |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate: DNM - test out afs mirrors https://review.openstack.org/272215 | 17:42 |
| jeblair | krotscheck, clarkb: ^ | 17:42 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: fix repository name detection https://review.openstack.org/276847 | 17:43 |
| dhellmann | fungi : that should fix the naming issue ^^ | 17:43 |
| AJaeger | jeblair: do you plan to followup on the build-timeout changes and remove DEVSTACK_GATE_TIMEOUT? | 17:43 |
| krotscheck | jeblair: Ok, I'll keep an eye on it. | 17:43 |
| krotscheck | (Not in the check queue yet) | 17:43 |
| jeblair | AJaeger: i would love it if someone else did that | 17:43 |
| rbradfor | seeking some help with a project-config/zuul/layout.yaml gate error | 17:43 |
| jeblair | i have more open self-owned patches than i have ever had before | 17:44 |
| clarkb | jeblair: that looks like a good test, but likely will fail on centos/fedora (thats expected so ok) | 17:44 |
| AJaeger | jeblair: then let me do a first patch quickly and you review, ok? | 17:44 |
| *** amotoki has quit IRC | 17:45 | |
| *** eharney has joined #openstack-infra | 17:45 | |
| jeblair | clarkb: do you mean fail-fail, or just not-use-the-mirrors-fail ? | 17:45 |
| clarkb | jeblair: I expect that the wheels that will be installed will not function causing fail fail | 17:45 |
| clarkb | lxml for example which nova depends on | 17:45 |
| jeblair | clarkb: ah, because i'm forcing the slug | 17:45 |
| jeblair | yeah | 17:45 |
| jeblair | agreed | 17:45 |
| *** lezbar has quit IRC | 17:46 | |
| jeblair | clarkb: the script that actually generates that is also probably not locally installed... i could copy in that logic just for completeness? | 17:46 |
| *** shashank_hegde has joined #openstack-infra | 17:46 | |
| jeblair | that way we can make sure fedora doesn't do anything weird when we put a wheel url that doesn't exist | 17:46 |
| clarkb | ++ | 17:46 |
| krotscheck | jeblair: I'm actually working on the fedora wheel right now. | 17:48 |
| *** haleyb has joined #openstack-infra | 17:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/devstack-gate: DNM - test out afs mirrors https://review.openstack.org/272215 | 17:48 |
| *** burgerk has quit IRC | 17:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 1] https://review.openstack.org/276850 | 17:48 |
| jeblair | clarkb, krotscheck: ^ updated to include afs-slug generation | 17:48 |
| AJaeger | jeblair: here's a first patch - is that all that we should do? ^ | 17:49 |
| *** zz_dimtruck is now known as dimtruck | 17:49 | |
| jeblair | clarkb, krotscheck: now we should see it use wheels on trusty, and succeed without using wheels on !trusty | 17:49 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 1] https://review.openstack.org/276850 | 17:49 |
| krotscheck | jeblair: GOtcha | 17:51 |
| *** e0ne has quit IRC | 17:51 | |
| krotscheck | Hrm, my fedora box doesn't have guest additions :/ | 17:51 |
| *** salv-orlando has joined #openstack-infra | 17:55 | |
| *** sdake has joined #openstack-infra | 17:56 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 1] https://review.openstack.org/276850 | 17:56 |
| AJaeger | jeblair: I missed some unsaved files in my editor and extended for letter c ^ | 17:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add wheel mirror to configure_mirror.sh https://review.openstack.org/267117 | 17:57 |
| *** esikachev has joined #openstack-infra | 17:57 | |
| *** sdake_ has joined #openstack-infra | 17:59 | |
| *** yaume has quit IRC | 17:59 | |
| *** tphummel has joined #openstack-infra | 17:59 | |
| *** sdake has quit IRC | 18:01 | |
| *** rossella_ has quit IRC | 18:02 | |
| *** esikachev has quit IRC | 18:03 | |
| *** jlanoux has quit IRC | 18:03 | |
| *** rossella_ has joined #openstack-infra | 18:03 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 18:05 |
| *** trown is now known as trown|lunch | 18:05 | |
| *** achanda has quit IRC | 18:06 | |
| clarkb | greghaynes: are you looking at the fialed jobs on 272384? | 18:06 |
| *** zeih has joined #openstack-infra | 18:06 | |
| *** vgridnev has quit IRC | 18:07 | |
| greghaynes | clarkb: I did some digging (see backscroll in -nova), https://bugs.launchpad.net/nova/+bug/1249065 is relevant but for nova-net | 18:07 |
| openstack | Launchpad bug 1249065 in OpenStack Compute (nova) "Nova throws 400 when attempting to add floating ip (instance.info_cache.network_info is empty)" [Medium,Fix released] - Assigned to Matt Riedemann (mriedem) | 18:07 |
| greghaynes | clarkb: I have to go do other things though, so if you want to pick up that'd be awesome | 18:07 |
| *** dtardivel has quit IRC | 18:08 | |
| clarkb | greghaynes: kk, I will start with a recheck to see if it is consistent | 18:08 |
| greghaynes | clarkb: the patch depending on it also failed | 18:08 |
| greghaynes | clarkb: so I think it is consistentish | 18:09 |
| *** jpr has quit IRC | 18:09 | |
| clarkb | huh would be interesting if this test caught a real regression in upstream | 18:09 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 2] https://review.openstack.org/276859 | 18:09 |
| AJaeger | It's nice to remove lines... | 18:09 |
| greghaynes | clarkb: I think it did, although it could be something like "devstack networking is wonky due to a config change" | 18:09 |
| *** unicell has joined #openstack-infra | 18:10 | |
| *** Guest66109 has quit IRC | 18:10 | |
| *** flepied has joined #openstack-infra | 18:10 | |
| *** nikhil_k has joined #openstack-infra | 18:11 | |
| clarkb | or a shade regression | 18:11 |
| *** flepied1 has joined #openstack-infra | 18:11 | |
| clarkb | except shade isn't used for that yet | 18:11 |
| clarkb | ya no shade in the traceback | 18:12 |
| AJaeger | ok, two patches up for the removal, let's see how this works out and then I'll do the rest... | 18:12 |
| greghaynes | yep, just novaclient, but also in the nova-api logs it looks like the instance is made | 18:12 |
| *** zeih has quit IRC | 18:12 | |
| greghaynes | clarkb: http://logs.openstack.org/03/273703/8/check/gate-dsvm-nodepool/b04c2d3/logs/screen-n-api.txt.gz#_2016-02-05_05_17_28_219 | 18:13 |
| *** annegentle has joined #openstack-infra | 18:13 | |
| greghaynes | clarkb: that is some debugging nova added when they were tracking down a race issue with nova-net which had this same symptom | 18:13 |
| *** nikhil has quit IRC | 18:14 | |
| *** flepied has quit IRC | 18:14 | |
| clarkb | there is a network_info but not a nw_info | 18:15 |
| *** dahoo has joined #openstack-infra | 18:15 | |
| clarkb | maybe it is just a key mismatch, but if that was the case I would expect all the neutron jobs with tempest to fail too | 18:15 |
| *** dansmith is now known as superdan | 18:16 | |
| greghaynes | clarkb: it grabs network_info as nw_info | 18:17 |
| greghaynes | clarkb: I think the logging is just unclear | 18:17 |
| clarkb | :| | 18:17 |
| *** annegentle has quit IRC | 18:17 | |
| clarkb | probably the error message hardcoding a key that isn't the same key used in code | 18:17 |
| clarkb | to the grep | 18:17 |
| *** achanda has joined #openstack-infra | 18:18 | |
| greghaynes | http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/utils.py#n346 | 18:18 |
| *** kzaitsev_mb has quit IRC | 18:18 | |
| greghaynes | http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/floating_ips.py#n197 | 18:19 |
| greghaynes | that last one is where the logging comes from | 18:19 |
| clarkb | yup so hydration isn't having the desired side effect of populating that instance's data | 18:20 |
| *** jkraj has joined #openstack-infra | 18:22 | |
| clarkb | or the info_cache is missing the network info entirely (the logging implies otherwise?) and bubbling up an info cache without that data | 18:23 |
| greghaynes | clarkb: hrm, network_info exists but is empty in that log | 18:23 |
| greghaynes | clarkb: I am not sure if that validates as False | 18:24 |
| clarkb | greghaynes: it would because NetworkInfo subclasses list | 18:24 |
| *** sputnik13 has joined #openstack-infra | 18:25 | |
| *** stevemar has joined #openstack-infra | 18:25 | |
| *** piet has quit IRC | 18:26 | |
| *** dizquierdo has quit IRC | 18:26 | |
| *** ybathia has joined #openstack-infra | 18:27 | |
| *** ybathia_ has joined #openstack-infra | 18:28 | |
| *** sridhar_ram has joined #openstack-infra | 18:29 | |
| *** dims is now known as dimsum__ | 18:30 | |
| jeblair | clarkb, krotscheck: http://logs.openstack.org/15/272215/4/check/gate-tempest-dsvm-centos7/bd7c7ec/ seems to have passed | 18:31 |
| jeblair | (that's centos7, so should not be using wheel mirror) | 18:31 |
| *** auggy is now known as frauggy | 18:32 | |
| *** ybathia has quit IRC | 18:32 | |
| *** ybathia_ is now known as ybathia | 18:32 | |
| *** jtomasek__ has joined #openstack-infra | 18:32 | |
| *** annegentle has joined #openstack-infra | 18:34 | |
| *** dizquierdo has joined #openstack-infra | 18:34 | |
| *** kushal has quit IRC | 18:35 | |
| jeblair | krotscheck: and a wheel build has started | 18:35 |
| jeblair | so we should see that go through the entire build/release process | 18:36 |
| *** sdake_ has quit IRC | 18:36 | |
| *** esikachev has joined #openstack-infra | 18:36 | |
| clarkb | greghaynes: reading the nova neutron api stuff http://git.openstack.org/cgit/openstack/nova/tree/nova/network/neutronv2/api.py#n558 could possibly be related if there was a config change that removed networks or didn't add them by default | 18:36 |
| *** ociuhandu has quit IRC | 18:36 | |
| *** mrmartin has joined #openstack-infra | 18:36 | |
| krotscheck | jeblair: Neat! | 18:38 |
| krotscheck | Oh man, if this works I'm going to squee. | 18:38 |
| jeblair | krotscheck: http://logs.openstack.org/15/272215/4/check/gate-grenade-dsvm-multinode/f69f15f//logs/grenade.sh.txt.gz#_2016-02-05_18_02_33_227 | 18:38 |
| jeblair | krotscheck: i think it just worked | 18:38 |
| krotscheck | Downloading http://mirror.dfw.rax.openstack.org/pypi/packages/source/l/libvirt-python/libvirt-python-1.3.0.tar.gz | 18:38 |
| *** annegentle has quit IRC | 18:38 | |
| * jeblair dons protective gear | 18:38 | |
| krotscheck | Wait | 18:39 |
| krotscheck | No | 18:39 |
| krotscheck | That's pypi | 18:39 |
| *** Swami has joined #openstack-infra | 18:39 | |
| *** haleyb has quit IRC | 18:39 | |
| krotscheck | http://mirror.dfw.rax.openstack.org/wheel/ubuntu-14.04-x86_64/extras/extras-0.0.3-py2-none-any.whl | 18:39 |
| krotscheck | There we go | 18:39 |
| krotscheck | Nice | 18:39 |
| *** sbalukoff has quit IRC | 18:39 | |
| krotscheck | jeblair: graphite has the average times of those builds, yes? | 18:40 |
| *** sdake has joined #openstack-infra | 18:41 | |
| jeblair | krotscheck: yeah, i have to run ... but a quick look at other recent builds suggests something like 20 minutes saved | 18:41 |
| jeblair | i'm seeing about 1h in rax, and that was 43m | 18:41 |
| krotscheck | Yeah. | 18:41 |
| jeblair | sdague: ^ | 18:41 |
| krotscheck | That's not bad. | 18:41 |
| clarkb | you can compare the devstack run time too | 18:41 |
| krotscheck | 30%? | 18:41 |
| clarkb | which is logged iirc | 18:42 |
| *** fitoduarte has quit IRC | 18:42 | |
| clarkb | greghaynes: theory, we need to explicitly set the network name/uuid now for some reason | 18:42 |
| clarkb | greghaynes: going to push a patch atop your stack that does that to see | 18:42 |
| jeblair | i'm going to approve https://review.openstack.org/275997 | 18:42 |
| jeblair | and hopefully we can restart nodepool today with that... | 18:43 |
| clarkb | jeblair: ok, trying to get somewhere with the stack that handles disconnects | 18:43 |
| jeblair | clarkb: ++ | 18:43 |
| * nibalizer yawns | 18:43 | |
| clarkb | but this nova/neutron floating IP fail is :( | 18:43 |
| nibalizer | good morning everyone | 18:43 |
| *** mrmartin has quit IRC | 18:43 | |
| jeblair | clarkb: well, we did make it non-voting for a reason... so i'm in favor of using judgement there if we need to ignore those results for a bit | 18:44 |
| jeblair | (understand judgement might be "no, we really need dsvm results for this") | 18:44 |
| *** mrmartin has joined #openstack-infra | 18:45 | |
| jeblair | anyway, i have to run for a bit | 18:45 |
| jeblair | but -- 2 new mirrors in 2 weeks :) we're on a roll | 18:45 |
| *** naresht has joined #openstack-infra | 18:45 | |
| clarkb | I would feel much more comfortable with working dsvm as that exercises the whole thing. If my hunch is right we should hvae a workaround while we figure out what changed in nova/neutron to make it not work by default | 18:46 |
| AJaeger | Woot! Congrats! | 18:46 |
| *** sputnik13 has quit IRC | 18:47 | |
| *** kushal has joined #openstack-infra | 18:49 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool: Explicitly set network in devstack plugin https://review.openstack.org/276869 | 18:49 |
| *** Hal has joined #openstack-infra | 18:49 | |
| *** mrmartin has quit IRC | 18:49 | |
| *** Hal is now known as Guest89885 | 18:50 | |
| openstackgerrit | Merged openstack-infra/nodepool: Increase gearman timeouts https://review.openstack.org/275997 | 18:50 |
| *** e0ne has joined #openstack-infra | 18:50 | |
| *** esikachev has quit IRC | 18:50 | |
| *** jtomasek__ has quit IRC | 18:51 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Add docs tox target https://review.openstack.org/276870 | 18:52 |
| *** haleyb has joined #openstack-infra | 18:53 | |
| *** esikachev has joined #openstack-infra | 18:54 | |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-zuul: Allow configuration of X-Account-Meta-Temp-Url-Key https://review.openstack.org/276871 | 18:54 |
| *** vgridnev has joined #openstack-infra | 18:55 | |
| AJaeger | have a great weekend everybody - a review of the DEVSTACK_GATE_TIMEOUT removals would be nice... 276850 and 276859 | 18:56 |
| dimsum__ | AJaeger : you too! | 18:56 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add bifrost manifest https://review.openstack.org/231757 | 18:56 |
| *** trown|lunch is now known as trown | 18:57 | |
| *** shardy has quit IRC | 18:59 | |
| *** ociuhandu has joined #openstack-infra | 19:01 | |
| *** sputnik13 has joined #openstack-infra | 19:03 | |
| *** thorst has quit IRC | 19:03 | |
| *** rbrndt_ has quit IRC | 19:04 | |
| *** thorst has joined #openstack-infra | 19:05 | |
| *** abregman|afk has quit IRC | 19:06 | |
| clarkb | greghaynes: https://review.openstack.org/276869 I think the regression may be that nova/neutron no longer just pick the only one if thats the case | 19:06 |
| *** vgridnev has quit IRC | 19:06 | |
| clarkb | greghaynes: that change should confirm it | 19:06 |
| *** tonytan4ever has joined #openstack-infra | 19:07 | |
| *** jpr has joined #openstack-infra | 19:09 | |
| *** anthonyper has quit IRC | 19:11 | |
| *** anthonyper has joined #openstack-infra | 19:12 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Clarify Munch object usage in documentation https://review.openstack.org/276878 | 19:12 |
| Shrews | SpamapS: that change ^^^ is based on the conversation we had. let me know what you think. | 19:13 |
| *** eil397 has joined #openstack-infra | 19:13 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:14 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/puppet-infracloud: Add element to bring up bridge on infra cloud https://review.openstack.org/276292 | 19:14 |
| *** esikachev has quit IRC | 19:16 | |
| SpamapS | Shrews: damnit. So now the thing we said was deprecated.. is the main way, and dict access is "alternatively"? | 19:17 |
| Shrews | SpamapS: isn't that what we said? | 19:18 |
| Shrews | SpamapS: don't think we agreed on a "main way" | 19:18 |
| Shrews | it's all horribly confusing and wish we had never used Munches | 19:18 |
| SpamapS | Shrews: right, so making the not-munch way "alternatively" is really just saying this is never going away. | 19:19 |
| SpamapS | not in 2.x | 19:19 |
| SpamapS | not ever | 19:19 |
| *** stevemar has quit IRC | 19:20 | |
| SpamapS | Shrews: I thought we did munch, and basically were like "if there's ever a 2.x, it's gone" | 19:20 |
| SpamapS | but your text suggests it's the opposite. dict access is alternative. | 19:20 |
| *** stevemar has joined #openstack-infra | 19:20 | |
| Shrews | SpamapS: poorly worded then | 19:20 |
| *** daemontool has quit IRC | 19:21 | |
| *** dizquierdo has quit IRC | 19:22 | |
| Shrews | SpamapS: but you're thinking was what i had intended to express... lemme try again | 19:22 |
| openstackgerrit | Merged openstack-infra/system-config: Add AFS apt mirror https://review.openstack.org/272587 | 19:22 |
| *** Sukhdev has joined #openstack-infra | 19:22 | |
| *** salv-orlando has quit IRC | 19:24 | |
| *** stevemar has quit IRC | 19:24 | |
| *** thorst has quit IRC | 19:24 | |
| *** daemontool has joined #openstack-infra | 19:25 | |
| *** jpr has quit IRC | 19:26 | |
| *** zeih has joined #openstack-infra | 19:27 | |
| *** pradk has quit IRC | 19:27 | |
| *** bauzas is now known as bauwser | 19:27 | |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for lbaas v2 dashboard https://review.openstack.org/275394 | 19:28 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Adding integration test support for app catalog horizon plugin https://review.openstack.org/276440 | 19:29 |
| *** fedexo has joined #openstack-infra | 19:31 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/shade: Clarify Munch object usage in documentation https://review.openstack.org/276878 | 19:31 |
| Shrews | SpamapS: more clearerer?? ^^^^ | 19:32 |
| *** ybathia has quit IRC | 19:32 | |
| *** stevemar has joined #openstack-infra | 19:35 | |
| *** sridhar_ram1 has joined #openstack-infra | 19:36 | |
| clarkb | greghaynes: we had jobs pass on the 2nd | 19:36 |
| greghaynes | clarkb: awesome | 19:36 |
| greghaynes | clarkb: thats an annoying behavior change | 19:37 |
| clarkb | looking at commits between now and then and nova I don't see anything jump out so on to neutron and devstack | 19:37 |
| *** tonytan4ever has quit IRC | 19:37 | |
| clarkb | greghaynes: but the nova code explcility lists networks from neutron so it should get something back | 19:37 |
| *** Daisy has joined #openstack-infra | 19:38 | |
| *** sridhar_ram has quit IRC | 19:38 | |
| *** tonytan4ever has joined #openstack-infra | 19:39 | |
| *** krotscheck is now known as krotscheck_dcm | 19:39 | |
| *** vgridnev has joined #openstack-infra | 19:39 | |
| SpamapS | Shrews: much | 19:39 |
| prometheanfire | the pillow limitation in upper-constraints needs to be lifted | 19:40 |
| openstackgerrit | Hongbin Lu proposed openstack-infra/elastic-recheck: Add query for Magnum bug 1542386 https://review.openstack.org/276885 | 19:41 |
| openstack | bug 1542386 in Magnum "functional-api job causes the jenkins slave agent to die" [Critical,New] https://launchpad.net/bugs/1542386 | 19:41 |
| prometheanfire | https://bugs.gentoo.org/show_bug.cgi?id=573958 | 19:41 |
| openstack | bugs.gentoo.org bug 573958 in Vulnerabilities "<dev-python/pillow-3.1.1 - Buffer overflow in TiffDecode.c (CVE-2016-{0740,0775})" [Normal,Confirmed] - Assigned to security | 19:41 |
| prometheanfire | CVE-2016-0740 CVE-2016-0775 | 19:42 |
| clarkb | prometheanfire: feel free to propose the chang ethen... | 19:42 |
| prometheanfire | ya, will be | 19:42 |
| clarkb | though upper constraints is updated daily | 19:42 |
| *** Daisy has quit IRC | 19:42 | |
| *** _nadya_ has joined #openstack-infra | 19:42 | |
| clarkb | so as long as the requirements line for it works there should be an automagic update | 19:42 |
| prometheanfire | yep | 19:42 |
| *** jtomasek__ has joined #openstack-infra | 19:43 | |
| prometheanfire | what project is that under? | 19:43 |
| clarkb | openstack/requirements | 19:43 |
| prometheanfire | ya | 19:43 |
| prometheanfire | for bug reporting | 19:43 |
| clarkb | TIL neutron functioanl tests clone ovs, compile it and modprobe it | 19:44 |
| prometheanfire | that sounds fast | 19:44 |
| *** electrofelix has quit IRC | 19:46 | |
| *** _nadya_ has quit IRC | 19:46 | |
| *** ZZelle_ has joined #openstack-infra | 19:46 | |
| *** sbalukoff has joined #openstack-infra | 19:47 | |
| *** hongbin has joined #openstack-infra | 19:47 | |
| *** rbrndt_ has joined #openstack-infra | 19:48 | |
| clarkb | greghaynes: ok I figured it out | 19:48 |
| clarkb | greghaynes: devstack is making a second user called alt_demo | 19:48 |
| *** zeih has quit IRC | 19:48 | |
| clarkb | doesn't appear to have networks | 19:48 |
| clarkb | nodepool uses that for some reason (yay clouds.yaml?) | 19:48 |
| *** jpr has joined #openstack-infra | 19:49 | |
| clarkb | sdague: ^ I think your alt_demo user may need the same setup as the demo user as far as networking goes | 19:49 |
| clarkb | sdague: and possibly clouds.yaml needs to be tweaked to use demo | 19:49 |
| hongbin | Hi, want to find someone to discuss a magnum gate issue https://bugs.launchpad.net/magnum/+bug/1542386 | 19:49 |
| openstack | Launchpad bug 1542386 in Magnum "functional-api job causes the jenkins slave agent to die" [Critical,New] | 19:49 |
| clarkb | hongbin: sure | 19:50 |
| clarkb | hongbin: the test is ocmpletely crashing the slve from what we can see | 19:50 |
| clarkb | hongbin: I checked ssh this morning on a slave that was borked last night and still can't get in | 19:50 |
| fungi | and not on rackspace, iirc? | 19:50 |
| clarkb | fungi: correct | 19:50 |
| *** annegentle has joined #openstack-infra | 19:50 | |
| hongbin | clarkb: I used elastic-check to see this error: The first day it happened is Jan 26 | 19:50 |
| clarkb | my theory is difference in behavior between swap device and sparse swap file | 19:51 |
| fungi | so may be swapfile vs swappart related | 19:51 |
| fungi | that, yeah | 19:51 |
| hongbin | clarkb: I wonder if anything happen at that date. Any chance? | 19:51 |
| clarkb | hongbin: no idea I would have to check the git logs (which anyone else can do ) | 19:51 |
| fungi | magnum functional tests could try to swapoff -a? | 19:51 |
| hongbin | clarkb: ack | 19:52 |
| fungi | that might expose a different error, or solve it | 19:52 |
| openstackgerrit | Matthew Thode proposed openstack/requirements: update constraints on pillow for CVE-2016-{0740,0775} https://review.openstack.org/276892 | 19:52 |
| clarkb | fungi: well they are the test that hits swap the hardest | 19:52 |
| clarkb | so likely still fail but maybe we can avoid crashing the slave | 19:52 |
| *** Guest89885 has quit IRC | 19:52 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:52 | |
| sdague | clarkb: sorry, I'm not sure I understand | 19:53 |
| fungi | also, as you mentioned last night, safe to drop our artifical memory limits at boot now that hpcloud is rip | 19:53 |
| sdague | can you summarize the concern | 19:54 |
| clarkb | sdague: the nodepool test started using the alt_demo user that was just added to devstack. This fails because there is no network available to the alt_demo user | 19:54 |
| *** thorst has joined #openstack-infra | 19:54 | |
| *** thorst has quit IRC | 19:54 | |
| clarkb | sdague: devstack should create networks for the alt_demo user and should default to demo not alt_demo in clouds.yaml | 19:54 |
| *** sdake has quit IRC | 19:54 | |
| fungi | i guess the devstack-supplied clouds.yaml now says to use alt_demo? | 19:54 |
| clarkb | I am trying to figure out how that second thing functions in openstack client config now | 19:54 |
| fungi | right | 19:54 |
| openstackgerrit | Matthew Thode proposed openstack/requirements: update constraints on pillow for CVE-2016-{0740,0775} https://review.openstack.org/276895 | 19:55 |
| sdague | clarkb: damn, I screwed up the cloud name | 19:55 |
| *** flepied has joined #openstack-infra | 19:56 | |
| clarkb | yes looks like the share a cloud name so the second one wins? | 19:56 |
| fungi | oh, this was supposed to be a second cloud? | 19:56 |
| sdague | clarkb: also, where is the networks thing that I'm supposed to do? | 19:56 |
| sdague | fungi: yeh | 19:56 |
| sdague | well, for clouds.yaml purposes | 19:56 |
| clarkb | sdague: devstack has a create initial networks setting that runs neutron client to setup the private network for the tenant | 19:56 |
| fungi | that makes more sense then. i can see it would have been an easy copy/paste error | 19:56 |
| prometheanfire | https://review.openstack.org/#/c/276891/ https://review.openstack.org/#/c/276892/ https://review.openstack.org/#/c/276895/ | 19:56 |
| prometheanfire | oh, the links were dropped, just delayed | 19:57 |
| clarkb | sdague: but that currently only does so for the demo user | 19:57 |
| *** thorst has joined #openstack-infra | 19:57 | |
| *** austin81 has left #openstack-infra | 19:57 | |
| *** notnownikki has quit IRC | 19:58 | |
| jeblair | mordred: i'm guessing you aren't back yet... so i'm going to join your screen session which it looks like you set up for this purpose and kick off the first mirror update | 19:58 |
| clarkb | prometheanfire: I want to say that 3.x may require work in the projects to be functional? you might check the git log for why it was pinned | 19:58 |
| clarkb | sdague: should I push a patch to devstack to change the cloud name to alt_devstack? | 19:58 |
| prometheanfire | clarkb: I will, be | 19:58 |
| *** flepied1 has quit IRC | 19:58 | |
| sdague | clarkb: sure | 19:58 |
| sdague | I'm looking at create_neutron_initial_network ... and kind of wanting to bean myself with a lemon peel wrapped around a brick | 19:59 |
| *** ifarkas_ has quit IRC | 19:59 | |
| fungi | sdague: have a pan-galactic gargle blaster instead. after all, it's friday | 19:59 |
| clarkb | looks like the name in the comment is devstack-alt so will use that | 19:59 |
| prometheanfire | clarkb: was added as a doc depend for stevedore | 19:59 |
| sdague | clarkb: yeh, that was the intent | 19:59 |
| sdague | I just screwed up the patch | 20:00 |
| sdague | the alt_demo user is almost always used to negative test acls | 20:00 |
| sdague | so, the network issue is probably fine to punt for now | 20:00 |
| jeblair | anyone used reprepro? | 20:02 |
| jeblair | Error opening config file './conf/distributions': No such file or directory(2) | 20:02 |
| jeblair | it doesn't seem to be using the /etc/reprepro/... dirs we set up | 20:03 |
| *** vgridnev has quit IRC | 20:03 | |
| *** sridhar_ram1 is now known as sridhar_ram | 20:03 | |
| clarkb | sdague: ya the network issue is less pressing with the clouds.yaml fix. But may confuse devstack users if they expect networks there | 20:04 |
| clarkb | sdague: in theory we can use the same IP range since tenants are isolated? | 20:05 |
| *** doug-fis_ has joined #openstack-infra | 20:05 | |
| sdague | clarkb: it might, but I'm ok delaying for now | 20:05 |
| openstackgerrit | Mark Vanderwiel proposed openstack-infra/project-config: Add JavaScript gate jobs to LBaaS dashboard https://review.openstack.org/272771 | 20:05 |
| sdague | ok, got to drop, have a good weekend folks | 20:05 |
| jeblair | mordred: ^ it kind of looks to me like that config should be inside the repo, not in /etc; and then we should run it with cwd=the repo | 20:06 |
| fungi | looks like you can do -b/--basedir in reprepro | 20:07 |
| fungi | or separately set --outdir and --confdir | 20:07 |
| *** fedexo has quit IRC | 20:08 | |
| fungi | (and --distdir and lots of othes) | 20:08 |
| *** doug-fish has quit IRC | 20:08 | |
| fungi | seems to be highly configurable about letting you keep configuration, package pools, indices et cetera wherever you want | 20:08 |
| fungi | though i've never used it | 20:09 |
| cody-somerville | I'm a reprepro expert. | 20:09 |
| *** doug-fis_ has quit IRC | 20:10 | |
| jeblair | cody-somerville, fungi: looks like if we add --confdir /etc/reprepro, that should be what we need | 20:10 |
| jeblair | cause the config files in there set the others | 20:10 |
| jeblair | cody-somerville: here's the patch: https://review.openstack.org/272587 | 20:11 |
| cody-somerville | jeblair: Is there an options file in your reprepro conf directory? | 20:11 |
| jeblair | cody-somerville: yes | 20:12 |
| cody-somerville | ah, yes. So it is possible that mordred intends for REPREPRO_BASE_DIR envvar to be set and then it'll read /etc/reprepro/options to set outdir and the like. | 20:13 |
| jeblair | k5start -t -f /etc/reprepro.keytab service/reprepro -- reprepro --confdir /etc/reprepro update | 20:13 |
| jeblair | that worked | 20:13 |
| jeblair | but | 20:13 |
| jeblair | /etc/reprepro/options:4: unknown option 'export=changed'! | 20:13 |
| jeblair | so i think mordred had a different version | 20:13 |
| *** ldnunes has quit IRC | 20:14 | |
| *** dimtruck is now known as zz_dimtruck | 20:14 | |
| jeblair | no... changed is mentioned in the options file | 20:14 |
| jeblair | cody-somerville: mabye it can only be on the command line and not in the options file? | 20:14 |
| openstackgerrit | Fausto Marzi proposed openstack-infra/project-config: Add python-jobs and python3-jobs to freezer* https://review.openstack.org/269138 | 20:14 |
| *** zz_dimtruck is now known as dimtruck | 20:15 | |
| cody-somerville | jeblair: what version of reprepro do you have installed? | 20:15 |
| jeblair | ii reprepro 4.13.1-1build1 amd64 Debian package repository producer | 20:15 |
| jeblair | this is on trusty | 20:15 |
| *** jtomasek__ has quit IRC | 20:16 | |
| *** annegentle has quit IRC | 20:16 | |
| jeblair | i'll try removing that from options and adding it to cmdline | 20:17 |
| openstackgerrit | Mikhail S Medvedev proposed openstack-infra/puppet-openstackci: Add pass through for swift_account_temp_key https://review.openstack.org/276902 | 20:18 |
| jeblair | that gets us to Error: unknown key '437D05B5'! | 20:18 |
| jeblair | ii ubuntu-keyring 2012.05.19 all GnuPG keys of the Ubuntu archive | 20:19 |
| jeblair | that's installed | 20:19 |
| jeblair | cody-somerville: ^ do you think i need to manually fetch a key? | 20:19 |
| cody-somerville | jeblair: Yea, import the key to your local gpg db for the user | 20:20 |
| cody-somerville | jeblair: you can also configure reprepro to use separate gnupg files if you want to keep it separate from your personal user's db | 20:21 |
| *** zeih_ has joined #openstack-infra | 20:21 | |
| *** _nadya_ has joined #openstack-infra | 20:22 | |
| jeblair | ok, i'll do the manual import for now, and when we fix this up in puppet we'll probably want to do it that way | 20:22 |
| *** sdake has joined #openstack-infra | 20:23 | |
| jeblair | woot, this is running now: | 20:23 |
| jeblair | k5start -t -f /etc/reprepro.keytab service/reprepro -- reprepro --export=changed --confdir /etc/reprepro update | 20:23 |
| cody-somerville | strange that the export=changed doesn't work in the options file. | 20:23 |
| *** _nadya_ has quit IRC | 20:23 | |
| clarkb | jeblair: there is a puppet resource to trust a key | 20:23 |
| clarkb | not sure if it can do so per user | 20:23 |
| clarkb | may just be root? | 20:24 |
| *** Sukhdev has quit IRC | 20:24 | |
| nibalizer | does anyone have the hiera lock? | 20:24 |
| nibalizer | I would like to take it if possible | 20:24 |
| *** zeih_ has quit IRC | 20:24 | |
| *** salv-orlando has joined #openstack-infra | 20:24 | |
| *** weshay_xchat has quit IRC | 20:25 | |
| *** stevemar has quit IRC | 20:26 | |
| jeblair | clarkb: cool | 20:26 |
| *** stevemar has joined #openstack-infra | 20:26 | |
| jeblair | cody-somerville: any chance you know if we can convince reprepro to skip the 'enough disk space' check? | 20:26 |
| cody-somerville | ah, you need to remove the = and replace it with space | 20:26 |
| *** annegentle has joined #openstack-infra | 20:27 | |
| cody-somerville | jeblair: --spacecheck=none | 20:27 |
| *** sridhar_ram1 has joined #openstack-infra | 20:27 | |
| jeblair | cause with afs, the amount of free space for the afs filesystem is infinity... but it reports 9000000 blocks. | 20:27 |
| cody-somerville | jeblair: or "spacecheck none" in conf/options | 20:27 |
| jeblair | which was probably a lot of blocks 30 years ago | 20:27 |
| *** weshay_xchat has joined #openstack-infra | 20:28 | |
| clarkb | nibalizer: I think that means the lock is yours | 20:28 |
| *** Sukhdev has joined #openstack-infra | 20:28 | |
| nibalizer | fantastic | 20:29 |
| nibalizer | infra-root taking the hiera lock | 20:29 |
| *** cloudtrainme has joined #openstack-infra | 20:29 | |
| *** annegentle has quit IRC | 20:29 | |
| nibalizer | done with the lock | 20:30 |
| *** sridhar_ram has quit IRC | 20:30 | |
| *** Sukhdev has quit IRC | 20:30 | |
| *** jtomasek__ has joined #openstack-infra | 20:31 | |
| jeblair | cody-somerville: cool, it seems to have gotten further and is working now | 20:32 |
| cody-somerville | \o/ | 20:32 |
| jeblair | cody-somerville, mordred: i left inline comments https://review.openstack.org/272587 on what we need to do to catch puppet up | 20:33 |
| *** apoorvad has quit IRC | 20:33 | |
| cody-somerville | jeblair: btw, does AFS support hardlinking? | 20:33 |
| *** Sukhdev has joined #openstack-infra | 20:35 | |
| cody-somerville | jeblair: mordred: and hmm... I potentially see problems with this configuration. You're mirroring Ubuntu and Debian into the same pool. Since Ubuntu rebuilds Debian binary packages, they're going to conflict for packages that end up in the same component (e.g. main). | 20:36 |
| *** ccrouch has quit IRC | 20:37 | |
| *** ccrouch1 has joined #openstack-infra | 20:37 | |
| *** david-lyle has joined #openstack-infra | 20:39 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config: Import ansible-roles-jenkins(-plugins) projects https://review.openstack.org/276910 | 20:41 |
| rbradfor | I'mI'm getting a "Job gate-oslo.log-src-grenade-dsvm not defined" error proposing a zuul/layout.yaml change yet the review is specifically defining this. Anybody able to help me debug. error: http://logs.openstack.org/76/276476/1/check/gate-project-config-layout/989ff83/console.html#_2016-02-04_23_34_20_646 review: https://review.openstack.org/#/c/276476/1/zuul/layout.yaml | 20:44 |
| *** achanda has quit IRC | 20:45 | |
| pabelanger | rbradfor: you'll also need to make sure it is defined in jenkins/jobs/projects.yaml | 20:45 |
| dimsum__ | rbradfor : need an entry in project.yaml too | 20:45 |
| dimsum__ | yep what pabelanger said :) | 20:45 |
| rbradfor | pabelanger, it is to my understanding, it's effectively cloning an existing job in another olso project '{pipeline}-{name}-src-grenade-dsvm{job-suffix} | 20:47 |
| fungi | cody-somerville: yeah, i'm mildly concerned about that, but suspect that if the debs have the exact same filenames they're directly imported from debian and not rebuilt? you probably know more about ubuntu package policy than i do | 20:47 |
| fungi | i thought they usually ended up with a local package version suffix appended if ubuntu had modified them in some way | 20:47 |
| cody-somerville | fungi: no, they rebuild the binary packages when they import. | 20:48 |
| fungi | oh, so not treated like a binnmu would be in debian | 20:49 |
| pabelanger | rbradfor: look at projects.yaml and see how oslo.messaging is doing it. Basically, you don't have the job created in Jenkins and zuul is complaining | 20:49 |
| cody-somerville | fungi: Correct. | 20:49 |
| fungi | cody-somerville: jeblair: mordred: then i agree, we likely need separate debian and ubuntu package pools | 20:49 |
| rbradfor | pabelanger, dimsum__ thanks for your insight | 20:50 |
| pabelanger | rbradfor: line 4331 for the syntax you need to add to oslo.log | 20:50 |
| rbradfor | pabelanger, thanks | 20:50 |
| fungi | ultimately, the number of exact filename overlaps between debian and ubuntu is minimal enough that there's not going to be a ton of space savings from having them share a pool anyway? | 20:50 |
| *** trown is now known as trown|brb | 20:51 | |
| cody-somerville | fungi: it would be huge savings actually. most packages in Debian are in main and are untouched in Ubuntu IIRC. | 20:51 |
| openstackgerrit | Merged openstack-infra/release-tools: fix repository name detection https://review.openstack.org/276847 | 20:52 |
| cody-somerville | fungi: and by huge savings, that is of course relative - it isn't really that much more data for a whole separate mirror of Debian. | 20:52 |
| fungi | cody-somerville: yeah, i guess there is a lot of non-churn in the majority of debian packages between stable releases, so whatever ubuntu's pulling from testing at any given checkpoint is likely unchanged | 20:53 |
| Clint | cody-somerville: the sources are untouched but the .debs are rebuilt | 20:53 |
| cody-somerville | Clint: Correct. | 20:53 |
| clarkb | it is also only configured to mirror trusty right now | 20:53 |
| clarkb | so don't have to solve it this second | 20:53 |
| fungi | that's double-confirmation then. we can't safely reuse the same package pool between distros so will need to namespace them before we add a second | 20:53 |
| cody-somerville | clarkb: That isn't true. See what merged. | 20:53 |
| clarkb | cody-somerville: https://review.openstack.org/#/c/272587/6 ? is trusty only | 20:54 |
| clarkb | https://review.openstack.org/#/c/272587/6/modules/openstack_project/manifests/mirror_update.pp line 62 | 20:54 |
| *** david-lyle has quit IRC | 20:55 | |
| *** david-lyle_ has joined #openstack-infra | 20:55 | |
| cody-somerville | clarkb: ah, was just looking for that. Yea, just trusty for now since Debian releases is set to empty string. | 20:55 |
| cody-somerville | clarkb: good catch, thanks. | 20:56 |
| *** apoorvad has joined #openstack-infra | 20:56 | |
| pabelanger | Since we are talking about APT mirrors, is anybody looking at yum/dnf mirrors currently? I'd be interested in looking into it if needed. Curious about the whole afs setup | 20:56 |
| *** imcsk8|vsTheGvt is now known as imcsk8 | 20:57 | |
| clarkb | for afs its just writing the mirror contents to afs which is then read and served by local web servers that can perform caching of hot files | 20:57 |
| fungi | pabelanger: i think the answer is "we want them and don't know what the rpm-specific implementation looks like yet (unless ianw has started in on that)" | 20:58 |
| fungi | but yeah, the afs part is solved a couple times over now | 20:58 |
| *** annegentle has joined #openstack-infra | 20:58 | |
| pabelanger | fungi: sure, I'll follow up with ianw and see what it might look like | 20:58 |
| clarkb | the two differences with afs are that you run under k5start for write perms and you want to publish the changes whenever tehy get synced | 20:58 |
| fungi | several times over (python sdist, python wheel, now deb, soon npm/node?) | 20:59 |
| fungi | pabelanger: think of it as an explicit flush/sync | 20:59 |
| clarkb | in theory the explicit publish step will avoid the out of sync apt mirror problems that have plagued us | 20:59 |
| *** achanda has joined #openstack-infra | 20:59 | |
| *** david-lyle_ is now known as david-lyle | 21:00 | |
| pabelanger | fungi: sure | 21:00 |
| openstackgerrit | Ronald Bradford proposed openstack-infra/project-config: Add grenade testing for oslo.log https://review.openstack.org/276476 | 21:00 |
| clarkb | but reprepro may solve that on its own | 21:00 |
| fungi | also delayed deletes, as pointed out earlier today, help keep us from removing something as or right before a machine is about to download it because it has a stale index | 21:00 |
| clarkb | I want to say someone said it does its own consistency checking/enforcement | 21:00 |
| *** tongli has quit IRC | 21:00 | |
| fungi | clarkb: reprepro isn't a traditional "ftp mirror" and instead regenerates its indices from the packages available | 21:01 |
| *** jsavak has quit IRC | 21:01 | |
| fungi | afaik | 21:01 |
| *** annegentle has quit IRC | 21:01 | |
| *** jsavak has joined #openstack-infra | 21:01 | |
| fungi | so in theory you should never get into that "bad mirror" state where you've copied indices which are mismatched from the available packages themselves | 21:02 |
| *** fedexo has joined #openstack-infra | 21:02 | |
| *** thiagop has quit IRC | 21:03 | |
| pabelanger | Ya, I found reprepro pretty flexible back in the day. Been a while since I used it however | 21:04 |
| fungi | though if that's the case, we're going to find ourselves needing to solve gpg key management for secure apt? | 21:04 |
| fungi | maybe i need to take a closer read | 21:04 |
| clarkb | fungi: that was one of the things that jeblair noted. There is a key resource in the puppet apt module | 21:05 |
| clarkb | not sure if it is sufficient for reprepro's needs | 21:05 |
| *** esikachev has joined #openstack-infra | 21:05 | |
| fungi | yeah, manual indicates it does indeed generate and sign its own indices, which if you're going to support subsets it would have to anyway | 21:06 |
| *** achanda has quit IRC | 21:06 | |
| fungi | we'll likely want to create a gpg key for repository signing and puppet that onto the reprepro worker, and then add the public key for it to apt-key on all the machines using our mirror | 21:07 |
| fungi | i need to start reviewing the change i guess | 21:07 |
| *** gordc has quit IRC | 21:09 | |
| *** trown|brb is now known as trown | 21:10 | |
| *** infra-red has joined #openstack-infra | 21:10 | |
| phschwartz | What is the command to force delete a node in nodepool (I always for get) | 21:12 |
| *** achanda has joined #openstack-infra | 21:12 | |
| fungi | phschwartz: sudo -H -u nodepool nodepool delete --now XXXXXX | 21:12 |
| fungi | pretty sure | 21:12 |
| *** rlandy has quit IRC | 21:13 | |
| fungi | also nodepool has command-line help and documentation which should mention that | 21:13 |
| * dimsum__ eagerly waiting for fresh nodepool images :) | 21:13 | |
| phschwartz | fungi: hmm, I didn't see --now in the cmd line help | 21:14 |
| phschwartz | fungi: but that is what I was looking for | 21:14 |
| fungi | nodepool delete --help doesn't mention it? | 21:14 |
| clarkb | fungi: it does | 21:14 |
| clarkb | I just checked to make sure that wasn't buggy | 21:14 |
| fungi | yep: --now delete the node in the foreground | 21:15 |
| fungi | nodepool --help doesn't, of course, and isn't really clear that there are sublevel --help outputs | 21:15 |
| fungi | so i can see how it might get overlooked | 21:16 |
| *** sdake has quit IRC | 21:16 | |
| clarkb | that is an argparseism | 21:16 |
| *** e0ne has quit IRC | 21:20 | |
| clarkb | looks like selenium 2.51.0 is broken | 21:21 |
| clarkb | python setup.py egg_info fails | 21:22 |
| *** tonytan4ever has quit IRC | 21:22 | |
| *** boris-42 has joined #openstack-infra | 21:23 | |
| clarkb | yup the README.rst is missing | 21:23 |
| *** _nadya_ has joined #openstack-infra | 21:24 | |
| *** aysyd has quit IRC | 21:24 | |
| clarkb | I will file a bug upstream | 21:24 |
| fungi | wow | 21:24 |
| *** esikachev has quit IRC | 21:24 | |
| fungi | horizon's having a _very_ bad week | 21:24 |
| clarkb | it is in their manifest.in but not in the sdist so dunno why it broke | 21:25 |
| *** stevemar has quit IRC | 21:27 | |
| *** _nadya_ has quit IRC | 21:28 | |
| *** stevemar has joined #openstack-infra | 21:28 | |
| *** kgiusti has left #openstack-infra | 21:29 | |
| clarkb | https://github.com/SeleniumHQ/selenium/issues/1594 | 21:31 |
| nibalizer | clarkb: for the bifrost change, 231757 do we need to put the ipmi passwords into hiera somewere? | 21:31 |
| *** achanda has quit IRC | 21:31 | |
| nibalizer | I remember SpamapS and you and me comming up with something sortof clever where the bifrost node would copy them back to hiera or somethinng but now I forget | 21:32 |
| *** stevemar_ has joined #openstack-infra | 21:32 | |
| clarkb | the MANIFEST.in in the sdist shows the include is for README | 21:32 |
| *** jtomasek__ has quit IRC | 21:32 | |
| SpamapS | nibalizer: I just templated the inventory file in ansible | 21:33 |
| SpamapS | err | 21:33 |
| SpamapS | in Puppet | 21:33 |
| clarkb | oh looks like they may have just fixed it? | 21:33 |
| *** stevemar has quit IRC | 21:33 | |
| nibalizer | right | 21:33 |
| nibalizer | I think we're missing a node definition that would instantiate that class, though maybe its in a patch I have not reviewed yet | 21:34 |
| *** trown has quit IRC | 21:34 | |
| crinkle | nibalizer: what are you looking for? | 21:35 |
| crinkle | nibalizer: ipmi passwords need to be in secret hiera and get used like this https://review.openstack.org/#/c/231757/32/templates/bifrost/inventory.hpuswest.json.erb | 21:35 |
| nibalizer | crinkle: ya I'm wondering what I need to put in hiera | 21:35 |
| nibalizer | is there an exmaple in the local hiera on controller00? | 21:35 |
| crinkle | nibalizer: they should be in that hiera yes | 21:35 |
| *** trown has joined #openstack-infra | 21:36 | |
| nibalizer | ok | 21:36 |
| nibalizer | is controller00 the bifrost node? | 21:36 |
| crinkle | no | 21:36 |
| nibalizer | oh | 21:36 |
| nibalizer | which node is? | 21:36 |
| crinkle | bifrost node is 15.184.52.2 or .3 | 21:36 |
| crinkle | let me check | 21:36 |
| crinkle | nibalizer: .2 | 21:37 |
| *** esikachev has joined #openstack-infra | 21:38 | |
| nibalizer | question: where does /etc/ansible/hosts/emergency~ come from? | 21:39 |
| nibalizer | is filename~ like the emacs version of .swp? | 21:39 |
| nibalizer | crinkle: thinaks | 21:39 |
| clarkb | nibalizer: yes | 21:39 |
| *** jsavak has quit IRC | 21:40 | |
| *** achanda has joined #openstack-infra | 21:41 | |
| *** jsavak has joined #openstack-infra | 21:41 | |
| nibalizer | ah okay then I will ignore | 21:41 |
| nibalizer | i also tested a bit and it looks like ansible doesn't just suck that in | 21:41 |
| nibalizer | which is nice | 21:41 |
| *** esikachev has quit IRC | 21:46 | |
| *** infra-red has quit IRC | 21:48 | |
| *** shashank_hegde has quit IRC | 21:49 | |
| *** xyang1 has quit IRC | 21:50 | |
| *** tpsilva has quit IRC | 21:56 | |
| *** zeih_ has joined #openstack-infra | 21:58 | |
| *** thorst has quit IRC | 22:02 | |
| *** rossella_ has quit IRC | 22:03 | |
| *** sridhar_ram has joined #openstack-infra | 22:03 | |
| *** rossella_ has joined #openstack-infra | 22:03 | |
| *** sridhar_ram1 has quit IRC | 22:04 | |
| *** dims has joined #openstack-infra | 22:04 | |
| *** dimsum__ has quit IRC | 22:04 | |
| *** thorst has joined #openstack-infra | 22:04 | |
| clarkb | jeblair: the version of nodepool installed on nodepool.o.o includes that timeout increase | 22:04 |
| clarkb | jeblair: devstack fix is slow to get through so maybe we go ahead and restart nodepool now for timeouts then restart again monday for disconnect handling | 22:05 |
| clarkb | jeblair: I can do that and remove the building images from the db | 22:05 |
| *** trown is now known as trown|outttypeww | 22:05 | |
| *** diana_clarke has quit IRC | 22:07 | |
| *** diana_clarke has joined #openstack-infra | 22:07 | |
| openstackgerrit | Fausto Marzi proposed openstack-infra/irc-meetings: Add Freezer team meeting https://review.openstack.org/241039 | 22:07 |
| greghaynes | clarkb: ++ | 22:08 |
| *** thorst has quit IRC | 22:09 | |
| *** jsavak has quit IRC | 22:11 | |
| anteaya | sorry I missed the day, a morning errand extended to an afternoon errand and became an all day errand | 22:16 |
| *** Sukhdev has quit IRC | 22:19 | |
| *** gyee has quit IRC | 22:21 | |
| *** tonytan4ever has joined #openstack-infra | 22:22 | |
| clarkb | greghaynes: jeblair: will the builer need restarting as well? I guess I should just do it for completeness | 22:24 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/puppet-infracloud: Add bifrost manifest https://review.openstack.org/231757 | 22:24 |
| *** lascii is now known as alaski | 22:24 | |
| *** stevemar_ has quit IRC | 22:24 | |
| *** stevemar has joined #openstack-infra | 22:25 | |
| greghaynes | clarkb: I think the change was just nodepool side | 22:26 |
| greghaynes | clarkb: also I havent seen any disconnects from the builder | 22:26 |
| greghaynes | but it doesn't hurt to restart the builder | 22:26 |
| *** rfolco has quit IRC | 22:27 | |
| *** kzaitsev_mb has joined #openstack-infra | 22:29 | |
| *** daemontool has quit IRC | 22:30 | |
| *** ccrouch1 has quit IRC | 22:31 | |
| *** shashank_hegde has joined #openstack-infra | 22:32 | |
| *** Sukhdev has joined #openstack-infra | 22:33 | |
| *** arxcruz has quit IRC | 22:33 | |
| *** ccrouch has joined #openstack-infra | 22:34 | |
| *** gyee has joined #openstack-infra | 22:34 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add baremetal hpuswest node definition https://review.openstack.org/208751 | 22:37 |
| *** salv-orl_ has joined #openstack-infra | 22:40 | |
| *** baoli has quit IRC | 22:41 | |
| *** jpr has quit IRC | 22:41 | |
| *** salv-orlando has quit IRC | 22:43 | |
| *** cbader has quit IRC | 22:43 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run puppet apply on infracloud hosts https://review.openstack.org/276943 | 22:47 |
| *** amitgandhinz has quit IRC | 22:47 | |
| *** zeih_ has quit IRC | 22:47 | |
| clarkb | well I cleared up the old "building" images and a new devstack-centos7 started so will wait for that to complete | 22:48 |
| clarkb | ianw: ^ nd the logs for that are going to the correct file | 22:48 |
| anteaya | jeblair: AJaeger rebased and it lost your +2, are you available to review? https://review.openstack.org/#/c/276850/3 | 22:51 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Run puppet apply on infracloud hosts https://review.openstack.org/276943 | 22:53 |
| nibalizer | mordred: interested in your thoughts on 276943 | 22:54 |
| greghaynes | clarkb: is that tcpdump still running? | 22:54 |
| *** sabeen3 has quit IRC | 22:54 | |
| clarkb | greghaynes: yup | 22:54 |
| clarkb | I should actually make sure its ok | 22:54 |
| greghaynes | clarkb: has there been any disconnects? | 22:55 |
| clarkb | oh awit no I set it to go for 24 hours | 22:55 |
| greghaynes | heh | 22:55 |
| clarkb | but let me chek if there was a disconnect in that 24 hours | 22:55 |
| clarkb | gah there was one but after the 24 hours | 22:56 |
| clarkb | that'll learn me | 22:56 |
| *** erlon has quit IRC | 22:56 | |
| greghaynes | clarkb: I think there is a way to make tcpdump rotate files where itll eventually overwrite a previous file | 22:57 |
| greghaynes | clarkb: if we used that then we could always have the most recent N amount of time while capping max disk usage... | 22:57 |
| clarkb | ya if you do it file size based I think it does that | 22:57 |
| clarkb | but I went timestamp to make it easier to determine where packets were | 22:57 |
| greghaynes | ah | 22:57 |
| clarkb | I will look into restarting it with a circular buffer | 22:57 |
| greghaynes | timestamps are recorded in the pcap's I am pretty sure | 22:58 |
| greghaynes | youll have to find the correct file but its doable | 22:58 |
| *** sridhar_ram has quit IRC | 22:58 | |
| clarkb | yup -W works differently I will just do it that way | 22:58 |
| *** haleyb has quit IRC | 22:59 | |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition https://review.openstack.org/213980 | 22:59 |
| *** sridhar_ram has joined #openstack-infra | 23:01 | |
| clarkb | tcpdump: gearman.pcap00: Permission denied but I am root why | 23:01 |
| openstackgerrit | Colleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node https://review.openstack.org/209698 | 23:02 |
| clarkb | I an touch a file | 23:03 |
| greghaynes | clarkb: is it expecting a dir? | 23:03 |
| clarkb | no apparently tcpdump drops its privs | 23:05 |
| clarkb | so I just need to make the dir world writeable and yolo | 23:05 |
| greghaynes | huh | 23:05 |
| *** ccrouch has quit IRC | 23:06 | |
| clarkb | but that didn't work either | 23:06 |
| *** ccrouch has joined #openstack-infra | 23:06 | |
| *** dims_ has joined #openstack-infra | 23:06 | |
| *** krotscheck1 has joined #openstack-infra | 23:08 | |
| clarkb | zomg it is apparmor | 23:09 |
| clarkb | the only time apparmor has ever done anything | 23:10 |
| greghaynes | hahaaha | 23:10 |
| nibalizer | haha | 23:10 |
| *** dims has quit IRC | 23:10 | |
| nibalizer | ofc | 23:10 |
| *** krotscheck1 has quit IRC | 23:10 | |
| jeblair | wow | 23:10 |
| *** nmagnezi_ has joined #openstack-infra | 23:10 | |
| clarkb | apparmor="DENIED" operation="mknod" profile="/usr/sbin/tcpdump" name="/opt/net_captures/gearman.pcap00" | 23:11 |
| jeblair | cody-somerville: aiui afs supports hard links only within the same directory; if you try to hard-link across dirs, you get a 'cannot hard link across devices' error | 23:11 |
| jeblair | cody-somerville: (unless that's changed recently) | 23:11 |
| jeblair | clarkb: re restarts - cool | 23:11 |
| *** tonytan4ever has quit IRC | 23:11 | |
| *** edmondsw has quit IRC | 23:13 | |
| clarkb | I am going to remove the apparmor stuff for tcpdump | 23:13 |
| clarkb | because writing rotating pcap files and gzipping them is functionality that should work | 23:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Add mirror-update to cacti https://review.openstack.org/276949 | 23:14 |
| nibalizer | jeblair: 275300 looks fine to me, would you like a +2a or +2 ? | 23:14 |
| *** dimtruck is now known as zz_dimtruck | 23:14 | |
| clarkb | tcpdump is running, capture 24x200MB rotating files | 23:15 |
| jeblair | nibalizer: go for it | 23:15 |
| clarkb | that is roughly one days worth | 23:15 |
| clarkb | jeblair: ok, the dvestack-centos7 build is to the qemu image converting so should be done in a bit | 23:15 |
| jeblair | clarkb: what's up with devstack-trusty? | 23:16 |
| jeblair | clarkb: cause when that lands we start using wheels | 23:16 |
| clarkb | jeblair: it isn't currently building it got lost in a previous disconnect | 23:16 |
| jeblair | clarkb: oh i thought you cleaned them up | 23:16 |
| clarkb | jeblair: we can manually trigger that after the restart | 23:16 |
| jeblair | oh that hasn't happened yet? | 23:16 |
| clarkb | jeblair: I did, but since we have existing images it doesn't automagically start a new one | 23:16 |
| jeblair | clarkb: gotcha | 23:17 |
| clarkb | devstack-centos7 didn't have any built images so that one got started | 23:17 |
| jeblair | clarkb: are we restarting again then? | 23:17 |
| clarkb | havne't restarted yet | 23:17 |
| clarkb | will do so once this build finishes so we don't lose it | 23:17 |
| clarkb | I didn't restart ealrier I just cleaned up db | 23:17 |
| jeblair | aaaaah | 23:17 |
| jeblair | okay sorry | 23:17 |
| clarkb | ya I did it in the wrong order | 23:17 |
| clarkb | that'll learn me | 23:18 |
| jeblair | yeah, thing's we shouldn't have to learn :) | 23:18 |
| *** matrohon has quit IRC | 23:20 | |
| jeblair | Volume Name Quota Used %Used Partition | 23:20 |
| jeblair | mirror.apt 286.1G 57.4G 20% 30% | 23:20 |
| jeblair | based on mordred's estimate earlier, we're about 2/3 there | 23:20 |
| jeblair | hrm | 23:20 |
| jeblair | i see no reason not to go ahead and do a vos release? | 23:21 |
| clarkb | nothing is using it yet | 23:21 |
| clarkb | so should be fine to have half a mirror published | 23:21 |
| jeblair | it'll be completely inconsistent, but we'll get a head start on transferring data to the replica | 23:21 |
| openstackgerrit | Merged openstack-infra/project-config: Remove DEVSTACK_GATE_TIMEOUT [part 1] https://review.openstack.org/276850 | 23:22 |
| jeblair | starting that... | 23:22 |
| jeblair | and it's running | 23:22 |
| jeblair | mordred: ^ fyi | 23:22 |
| *** bauwser is now known as bauzas | 23:23 | |
| openstackgerrit | Merged openstack-infra/system-config: Revert "Temporarily stop releasing after bandersnatch" https://review.openstack.org/275300 | 23:23 |
| *** flepied has quit IRC | 23:23 | |
| jeblair | krotscheck_dcm: it looks like the wheel build and release jobs have both triggered and succeeded a few times now | 23:24 |
| krotscheck_dcm | jeblair: That seems unreasonably stable. | 23:25 |
| * krotscheck_dcm is expecting something big to break. | 23:25 | |
| *** regXboi has quit IRC | 23:25 | |
| jeblair | krotscheck_dcm: probably next pycon | 23:25 |
| *** nmagnezi_ has quit IRC | 23:25 | |
| krotscheck_dcm | jeblair: Probably. | 23:25 |
| krotscheck_dcm | jeblair: So... we're done? | 23:25 |
| krotscheck_dcm | jeblair: I noticed that your DNM job broke for a perfectly normal reason. | 23:25 |
| *** cody-somerville has quit IRC | 23:26 | |
| jeblair | krotscheck_dcm: ah good i just started looking at that | 23:26 |
| *** cody-somerville has joined #openstack-infra | 23:26 | |
| krotscheck_dcm | jeblair: I only looked at the voting one though. | 23:26 |
| jeblair | 2016-02-05 20:38:24.741 | Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. | 23:27 |
| jeblair | that doesn't seem particularly wheel related; i'll recheck it just for kicks though | 23:27 |
| krotscheck_dcm | Yeah, might as well | 23:27 |
| krotscheck_dcm | How's nodepool feeling? | 23:28 |
| jeblair | krotscheck_dcm: clarkb thinks it might be up to building some images here in a bit | 23:29 |
| krotscheck_dcm | Good for nodepool! I'm happy to let it do its thing :) | 23:29 |
| *** jtomasek_ has quit IRC | 23:31 | |
| jeblair | clarkb, krotscheck_dcm: earlier clarkb suggested checking the devstack pip time: old and busted: "pip_install - 1534 secs | 23:31 |
| jeblair | clarkb, krotscheck_dcm: new hotness: pip_install - 227 secs | 23:31 |
| krotscheck_dcm | That'd be the 20 minutes we saw | 23:32 |
| jeblair | yeh, seems to corroborate | 23:32 |
| krotscheck_dcm | Amazing how much of a benefit that provided. | 23:32 |
| anteaya | nice work | 23:33 |
| jeblair | though, now i'm wondering what it's doing for 4 minutes :) | 23:34 |
| krotscheck_dcm | jeblair: network traffic probably. | 23:34 |
| * krotscheck_dcm is off to see notmorgan about a beer. | 23:35 | |
| jeblair | or installing files at python-speed | 23:35 |
| jeblair | krotscheck_dcm: prost | 23:35 |
| anteaya | krotscheck_dcm: enjoy beer | 23:35 |
| *** geekinutah has joined #openstack-infra | 23:36 | |
| *** cloudtrainme has quit IRC | 23:36 | |
| *** flepied has joined #openstack-infra | 23:37 | |
| *** chlong has joined #openstack-infra | 23:38 | |
| *** cody-somerville has quit IRC | 23:41 | |
| *** cody-somerville has joined #openstack-infra | 23:41 | |
| clarkb | centos7 is almost done fwiw | 23:43 |
| clarkb | I will restart then queue a devstack-trusty build | 23:43 |
| *** diana_clarke has quit IRC | 23:48 | |
| jeblair | ++ | 23:50 |
| clarkb | restarting now | 23:52 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix unicode regressions https://review.openstack.org/275879 | 23:53 |
| openstackgerrit | James E. Blair proposed openstack/gertty: Fix typo in expire-age setting https://review.openstack.org/275880 | 23:53 |
| clarkb | thats done | 23:54 |
| clarkb | jeblair: krotscheck_dcm devstack-trusty dib image is building now | 23:55 |
| clarkb | that is still a snapshot image on rax if you want me to trigger those three builds now | 23:55 |
| clarkb | I guess I should | 23:56 |
| jeblair | clarkb: yeah, didn't take loo long last time i did it -- oh, make sure you do all this in screen :) | 23:56 |
| jeblair | clarkb: i think i use the 'all' feature | 23:56 |
| jeblair | clarkb: so i had one screen window for each image type and then did the rebuild of that image type specifying 'all' for the provider | 23:57 |
| jeblair | clarkb: though, i guess we only urgently need trusty images atm | 23:57 |
| clarkb | yup using screen and because its split between dib and snapshot all gets weird, but I hvae all three rax's going | 23:57 |
| clarkb | and once the dib image is done building I will upload it to all the other plcaes | 23:58 |
| openstackgerrit | Merged openstack/gertty: Support fetching more than 1k changes post 2.8 https://review.openstack.org/274260 | 23:58 |
| openstackgerrit | Merged openstack/gertty: Fix unicode regressions https://review.openstack.org/275879 | 23:58 |
| openstackgerrit | Merged openstack/gertty: Fix typo in expire-age setting https://review.openstack.org/275880 | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!