| jeblair | i can not actually parse the sentence, so i don't know what | 00:00 |
|---|---|---|
| jeblair | clarkb: i don't expect anything on those graphs to change | 00:00 |
| clarkb | jeblair: the lower one should get better right? maybe I am misreading that one | 00:00 |
| clarkb | jeblair: how does throughput increase if time per server remains constant? | 00:00 |
| jeblair | clarkb: i don't know. i don't know how that graph is built. | 00:01 |
| jeblair | it was not done through code review. it is opaque to me. | 00:01 |
| *** markvoelker_ has quit IRC | 00:02 | |
| tsg_ | SpamapS, sdague: I believe John D discussed this earlier here today - we have a FFE request for bumping a Swift requirement in global-requirements.txt. Should the patch be submitted against master or stable/kilo? | 00:03 |
| tsg_ | trying to figure out where to do rc style patches in general .. | 00:03 |
| jeblair | (we could add that graph to our status pages, i just thought we wanted to avoid publicizing those sorts of metrics.) | 00:04 |
| jeblair | clarkb: but the individual tasks should take the same amount of time. we will just execute some (more) of them in parallel. | 00:05 |
| clarkb | jeblair: gotcha | 00:05 |
| SpamapS | tsg_: Not sure why you'd ask me. :) | 00:05 |
| clarkb | tsg_: both, you would bump master and backport to stable/kilo | 00:05 |
| jeblair | clarkb: overall throughput should increase, and the time to create a server should decrease. i think we track that... lemme see. | 00:05 |
| tsg_ | clarkb: thank you :) | 00:05 |
| *** zz_dimtruck is now known as dimtruck | 00:05 | |
| tsg_ | (SpamapS: because it had the term FFE in it ;) ) | 00:06 |
| clarkb | jeblair: in general I don't think we should publish comparitive numbers, but having a first derivative line of that data may be helpful | 00:06 |
| jeblair | clarkb: i think it's in stats.timers.nodepool.launch.provider.... | 00:06 |
| tsg_ | SpamapS: kidding .. couldn't figure out who was the best person to ask :) | 00:06 |
| *** mahito has joined #openstack-infra | 00:07 | |
| fungi | tsg_: you merely ask the aether, and the aether then responds | 00:07 |
| jeblair | clarkb: so starting with something like http://tinyurl.com/lbvc5ak | 00:08 |
| mordred | clarkb, jeblair: it's also comparative - but -it's plain graphite: http://inaugust.com/task-report.html | 00:08 |
| mordred | clarkb: should be very easy to edit and make non-comparitive | 00:08 |
| *** yamahata has quit IRC | 00:10 | |
| SpamapS | jeblair: https://code.launchpad.net/~clint-fewbar/gearmand/document-protocol/+merge/253140 has my prose added now | 00:11 |
| clarkb | in other news, zm04 may be having trouble. `git fetch http://zm04.openstack.org/p/openstack/nova refs/zuul/master/Z8d7b073a40be4f4db56ba22e76e5bc4a` has been hanging for 5 minutes in a job | 00:11 |
| clarkb | looking into it now | 00:11 |
| *** achanda_ has joined #openstack-infra | 00:11 | |
| jeblair | fungi: there are a number of ready nodes assigned to jenkins07 | 00:12 |
| mordred | clarkb, jeblair: problem is - the times are so radically different, without provider comparisons it's hard to see change in behavior | 00:12 |
| jeblair | fungi: what's your status there? maybe i should nodepool delete those? | 00:12 |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=2040&rra_id=all appears to be the high level culprit | 00:12 |
| jeblair | mordred: yeah, i'm all for making the graphs we need to run our systems. i just want to make sure we're not abusing the access we have. | 00:13 |
| fungi | jeblair: i just started quiescing 07, 06 has one job left before i restart | 00:13 |
| mordred | jeblair: totally | 00:13 |
| fungi | i'll clean up the ready nodes on both | 00:13 |
| clarkb | I am not getting a shell very quickly via ssh | 00:13 |
| jeblair | mordred: as clarkb points out, first derivative might be good enough... | 00:13 |
| clarkb | load average of ~120 | 00:13 |
| jeblair | mordred: and if not, maybe we make per-provider dashboards, but try not to put the providers side-by side. ;) | 00:14 |
| mordred | jeblair: in this case, I'm starting to lean towards us having a split-out graph, not for "comparison" - but because looking at rax numbers or hp numbers is super helpful sometimes in diagnosing thigns with them | 00:14 |
| clarkb | we are swapping | 00:14 |
| mordred | jeblair: that's a good idea | 00:14 |
| SpamapS | ok, and I just sent a message to the gearman mailing list asking if anybody is actually reviewing code anymore | 00:14 |
| mordred | jeblair: different graphs - perhaps with different scales so taht we're seeing shapes | 00:14 |
| *** yamamoto has quit IRC | 00:15 | |
| jeblair | fungi: i'll delete ready jenkins07 nodes | 00:15 |
| fungi | jeblair: i've already got that delete loop underway | 00:15 |
| jeblair | nm then :) | 00:16 |
| jeblair | i will look for something else to do! :) | 00:16 |
| *** achanda_ has quit IRC | 00:16 | |
| *** achanda has quit IRC | 00:16 | |
| fungi | though if you know a faster mechanism than going through the cli which doesn't miss subnodes, i'm thrilled for some details | 00:16 |
| jeblair | fungi: is it slow? | 00:16 |
| fungi | i'd guess a join would get me that | 00:16 |
| fungi | not particularly, i'm just doing it serially so it takes a few minutes to complete | 00:17 |
| fungi | there were about 50 ready nodes in total assigned to either jenkins06 or 07 | 00:17 |
| jeblair | ok; looks like it tries to talk to all the jenkins masters, we can probably fix that in the delete command (it doesn't need to do that; it should just be a database op) | 00:18 |
| fungi | clarkb: check-tempest-dsvm-multinode-full is slower than i expected | 00:18 |
| fungi | looks like it runs in the check pipeline for nova now | 00:19 |
| clarkb | fungi: it does, I think some of the slowness is related to zm04 and affecting a lot more jobs | 00:19 |
| fungi | oh, okay | 00:19 |
| clarkb | fungi: and my change to parallelize some of the setup speeds things up quite a bit | 00:19 |
| clarkb | its about ~8 minutes at least | 00:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Do not set up managers on CLI delete https://review.openstack.org/174163 | 00:20 |
| jeblair | fungi: ^ | 00:20 |
| jeblair | fungi: we could cherry-pick that into production and you should be able to start using it immediately | 00:20 |
| fungi | jeblair: yep, looks like it'll help a lot--thanks! | 00:20 |
| jeblair | fungi: i will do that | 00:20 |
| clarkb | jeblair: it needs the jenkins manager to delete it from jenkisn right? | 00:21 |
| fungi | clarkb: not on async | 00:21 |
| fungi | just on sync (i.e. --now) | 00:21 |
| clarkb | oh right | 00:21 |
| jogo | lifeless: I am happy to see you diving into the dependency issue | 00:21 |
| jeblair | fungi: installed. your next invocation should either be fast or show us what's wrong with that approach. :) | 00:21 |
| *** tkelsey has joined #openstack-infra | 00:21 | |
| clarkb | jeblair: fungi: I think we should consider larger zuul mergers or more of them | 00:22 |
| fungi | jeblair: appreciated. i'll be trying it out again here as soon as i can stop 06 | 00:22 |
| fungi | whenever this multinode job completes | 00:22 |
| clarkb | zm04 seems ot have fallen by a death of a thousand cuts. Not process is using more than 100MB of memory but there are many git operations and they add up | 00:22 |
| clarkb | s/Not/No/ | 00:22 |
| *** Rockyg has quit IRC | 00:22 | |
| jeblair | clarkb: the others look idle. that seems weird. | 00:23 |
| clarkb | oh, I didn't look at the others, maybe this one ended up with the majority of the work | 00:23 |
| clarkb | which is odd since gearman should naturally load balance based on it getting work done | 00:24 |
| jeblair | clarkb: zm07 has a spike, the others look essentially idle | 00:24 |
| *** signed8bit_ZZZzz has quit IRC | 00:25 | |
| *** tkelsey has quit IRC | 00:26 | |
| clarkb | 04 has done roughly twice the number of merge jobs according to the debug log than most of the others | 00:27 |
| clarkb | 07 hsa done about 50% more | 00:27 |
| jeblair | zm01 appears stuck | 00:27 |
| jeblair | since 00:10 utc | 00:28 |
| clarkb | 02/03/05/06 as well | 00:28 |
| jeblair | clarkb: yep | 00:29 |
| fungi | no wonder 04 is panting | 00:29 |
| clarkb | they seem to have successfully finished their last jobs then have stopped | 00:30 |
| fungi | jeblair: just did another delete pass now to catch 14 new straggler ready nodes on 06/07 and it went blazingly fast compared to before | 00:30 |
| jeblair | clarkb: yeah. strace shows them gearmaning | 00:31 |
| jeblair | at least zm01 | 00:31 |
| jeblair | is zm04 just winning too much? | 00:31 |
| fungi | okay, the last job on 06 just finished. cleaning it up and restarting now | 00:32 |
| clarkb | jeblair: maybe? 02 strace shows it doing REQ and getting back RES | 00:33 |
| jeblair | clarkb: zm04 seems to have recovered | 00:33 |
| *** sarob has quit IRC | 00:33 | |
| jeblair | clarkb: yeah, there are no pending merge jobs | 00:33 |
| lifeless | jogo: heh | 00:34 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: New repos for Akanda https://review.openstack.org/174104 | 00:34 |
| lifeless | jogo: its all frustration | 00:35 |
| clarkb | jeblair: and 04 is pretty consistently winning | 00:35 |
| *** adreznec has quit IRC | 00:36 | |
| *** r-daneel has quit IRC | 00:37 | |
| *** adreznec has joined #openstack-infra | 00:37 | |
| *** reazem_ has joined #openstack-infra | 00:37 | |
| jogo | lifeless: yup | 00:38 |
| jogo | your plan sounds good though | 00:38 |
| openstackgerrit | Merged openstack-infra/project-config: Temporarily copy proposed/* ACLs to stable/kilo https://review.openstack.org/174074 | 00:39 |
| *** reazem has quit IRC | 00:39 | |
| openstackgerrit | Tushar Gohad proposed openstack/requirements: Bump PyECLib version from 1.0.3 to 1.0.7 https://review.openstack.org/174167 | 00:39 |
| *** dims_ has quit IRC | 00:40 | |
| jeblair | clarkb: wow those servers are busy | 00:42 |
| clarkb | jeblair: looks like the other mergers are picking up some of the slack | 00:42 |
| clarkb | jeblair: yes, I think the new dependency stuff needlessly merges all things in a stack that is already stacked | 00:43 |
| clarkb | so you do 1 + 2 + 3 + 4 + 5 + 6 + 7 etc merges per stack | 00:43 |
| clarkb | its technically correct but inefficient | 00:43 |
| openstackgerrit | Merged openstack-infra/project-config: Configure stable branch ACLs for non-Oslo libs https://review.openstack.org/173892 | 00:43 |
| jeblair | clarkb: yeah, i wrote a change for that. it needs more work. but i don't think that's at issue here | 00:43 |
| jeblair | clarkb: (that actually _slows down_ a merger's ability to consume lots of work from zuul, and should cause the other mergers a chance to pick it up) | 00:44 |
| jeblair | clarkb: here, i think we're looking at a single merger servicing a lot of jobs for some reason | 00:45 |
| clarkb | jeblair: ya | 00:45 |
| clarkb | though the others are doing more of the work now | 00:45 |
| davidlenwell | clark.. So you are saying use http git urls and not git@ correct? | 00:46 |
| davidlenwell | clarkb: ^ | 00:46 |
| davidlenwell | re: https://review.openstack.org/#/c/174104/4/gerrit/projects.yaml | 00:46 |
| clarkb | davidlenwell: I would use https | 00:46 |
| clarkb | davidlenwell: because git@ means do ssh | 00:46 |
| clarkb | davidlenwell: but jeepyb (I may have said nodepool for some reason) isn't going to be able to ssh to github | 00:47 |
| davidlenwell | got it .. fixing | 00:47 |
| fungi | yeah, jeepyb can do git:// and http:// and https:// but not so much git@ | 00:47 |
| davidlenwell | doh | 00:47 |
| *** asettle has joined #openstack-infra | 00:47 | |
| fungi | since that last one requires authentication to github | 00:48 |
| jeblair | clarkb: i'm not seeing a pattern for why this might be the case. but things seem to be settling out, so i think i might file this away for another time | 00:48 |
| clarkb | jeblair: ok | 00:49 |
| *** mbacchi1 has quit IRC | 00:49 | |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: New repos for Akanda https://review.openstack.org/174104 | 00:50 |
| jeblair | i have to attend to food now | 00:51 |
| davidlenwell | I thought you fed introveniously while you code.. or was that wine? | 00:51 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/devstack-gate: ***DO NOT MERGE*** Flip DVR bit to collect experimental run datapoints https://review.openstack.org/171809 | 00:51 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add fingerprint for bug 1442272 https://review.openstack.org/173591 | 00:52 |
| openstack | bug 1442272 in neutron "Ovs monitor tests fail with eventlet timeout" [High,In progress] https://launchpad.net/bugs/1442272 - Assigned to Maru Newby (maru) | 00:52 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: New repos for Akanda https://review.openstack.org/174104 | 00:53 |
| *** esker has quit IRC | 00:54 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add fingerprint for bug 1439371 https://review.openstack.org/173595 | 00:54 |
| openstack | bug 1439371 in Cinder "Volume creation from image fails for UEC and Glance API version 2" [High,Fix committed] https://launchpad.net/bugs/1439371 - Assigned to Jon Bernard (jbernard) | 00:54 |
| *** sputnik13 has quit IRC | 00:55 | |
| *** tsg_ has quit IRC | 00:56 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net runs https://review.openstack.org/174139 | 00:56 |
| *** aswadr has joined #openstack-infra | 00:58 | |
| *** tiswanso has joined #openstack-infra | 00:59 | |
| clarkb | davidlenwell: do you mind marking that change WIP until you have irc ironed out? | 00:59 |
| *** eantyshev has quit IRC | 01:00 | |
| davidlenwell | what do we need to finish irc? the channels is on.. and its never had an op.. so the bot can make itself op.. | 01:00 |
| davidlenwell | or am I thinking wrong? | 01:00 |
| *** tiswanso has quit IRC | 01:01 | |
| clarkb | davidlenwell: I left a comment, you need to register the channel then give the bot the permissions I listed | 01:01 |
| davidlenwell | okay | 01:01 |
| clarkb | davidlenwell: then edit the file that enforces those perms via the bot | 01:01 |
| *** tiswanso has joined #openstack-infra | 01:01 | |
| *** MarkAtwood has quit IRC | 01:03 | |
| *** FallenPegasus is now known as MarkAtwood | 01:03 | |
| *** MarkAtwood has quit IRC | 01:03 | |
| mordred | davidlenwell: accessbot/channels.yaml | 01:05 |
| *** marun_ has joined #openstack-infra | 01:06 | |
| openstackgerrit | Merged openstack-infra/nodepool: Don't try to delete nonexistent jenkins nodes https://review.openstack.org/174131 | 01:07 |
| *** tqtran has quit IRC | 01:07 | |
| *** markvoelker has joined #openstack-infra | 01:07 | |
| *** davideagnello has quit IRC | 01:08 | |
| *** marun has quit IRC | 01:08 | |
| davidlenwell | thanks | 01:09 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 01:11 | |
| *** ivar-laz_ has joined #openstack-infra | 01:15 | |
| *** baoli has joined #openstack-infra | 01:17 | |
| *** mriedem has joined #openstack-infra | 01:18 | |
| fungi | hrm, 174074 finally just applied to gerrit | 01:18 |
| *** julim has joined #openstack-infra | 01:18 | |
| *** ivar-lazzaro has quit IRC | 01:19 | |
| fungi | sp stable/kilo branches _should_ be locked down to release teams now | 01:19 |
| *** ivar-laz_ has quit IRC | 01:19 | |
| *** baoli has quit IRC | 01:22 | |
| fungi | zuul is catching up with its waiting backlog quickly | 01:23 |
| davidlenwell | clarkb: I marked it wip.. everyone in that channel is afk and I cannot claim op until they all vacate.. nobody knows who started it. | 01:24 |
| davidlenwell | thanks for looking at that so fast .. I'll ping when its ready to go | 01:24 |
| *** jogo has quit IRC | 01:24 | |
| fungi | last job just wrapped up on jenkins07. cleaning up and restarting now | 01:25 |
| *** crc32 has joined #openstack-infra | 01:25 | |
| *** armax has quit IRC | 01:28 | |
| *** markvoelker has quit IRC | 01:30 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net runs https://review.openstack.org/174139 | 01:32 |
| *** jerryz has joined #openstack-infra | 01:35 | |
| *** mtanino has quit IRC | 01:37 | |
| *** tnovacik has quit IRC | 01:38 | |
| *** fedexo has joined #openstack-infra | 01:39 | |
| *** dims has joined #openstack-infra | 01:40 | |
| *** sarob has joined #openstack-infra | 01:41 | |
| *** SumitNaiksatam has quit IRC | 01:41 | |
| *** trey has quit IRC | 01:42 | |
| *** trey has joined #openstack-infra | 01:46 | |
| *** sarob has quit IRC | 01:47 | |
| *** dims has quit IRC | 01:47 | |
| *** Daisy has joined #openstack-infra | 01:48 | |
| *** Daisy_ has joined #openstack-infra | 01:54 | |
| *** Daisy has quit IRC | 01:55 | |
| *** Daisy_ is now known as Daisy | 01:56 | |
| *** kaisers1 has joined #openstack-infra | 01:56 | |
| *** kaisers has quit IRC | 01:59 | |
| *** rfolco has joined #openstack-infra | 02:02 | |
| *** jogo has joined #openstack-infra | 02:05 | |
| *** harlowja is now known as harlowja_away | 02:05 | |
| *** esker has joined #openstack-infra | 02:08 | |
| *** thingee has quit IRC | 02:09 | |
| *** weshay has quit IRC | 02:12 | |
| *** sdake_ has joined #openstack-infra | 02:16 | |
| *** unicell has quit IRC | 02:16 | |
| *** patrickeast has quit IRC | 02:18 | |
| tchaypo | lifeless: so looking at the first comments on https://review.openstack.org/#/c/161047/4/specs/stable-requirements.rst | 02:18 |
| *** sdake__ has joined #openstack-infra | 02:19 | |
| *** sdake has quit IRC | 02:19 | |
| tchaypo | I think a clearer way to express the conflict would be to explicitly say that “pip install --upgrade foo” will result in bar==2.1 being installed, while “pip install —upgrade -r requirements.txt” will result in bar==2.0 installed | 02:19 |
| *** sdake_ has quit IRC | 02:22 | |
| lifeless | tchaypo: huh | 02:25 |
| tchaypo | *tests more* | 02:26 |
| tchaypo | oh, there’s a different issue. | 02:26 |
| tchaypo | If requirements.txt specifies bar<=2.0, but foo requires bar>=2.1, bar==2.0 gets installed rather than bar==2.1 | 02:27 |
| *** baoli has joined #openstack-infra | 02:27 | |
| lifeless | whats foo? | 02:27 |
| ianw | is there a reason why we're so far behind with kibana on logstash? | 02:27 |
| clarkb | ianw: yes actually | 02:27 |
| ianw | we seem to be cloning it from a repo that hasn't updated in two years | 02:28 |
| tchaypo | requirements.txt specifies foo and bar<=2.0; but the latest foo specifies bar>=2.1 | 02:28 |
| clarkb | ianw: latest kibana is ridiculous and expects write access to the elasticsearch cluster | 02:28 |
| clarkb | ianw: at one point I had it mostly working (with only one minor annoying warning) with that disabled | 02:28 |
| lifeless | tchaypo: that is a case of genuinely incompatible requirements | 02:28 |
| clarkb | ianw: then they went and changed everything on me | 02:28 |
| lifeless | tchaypo: that spec's example, and thus IIRC the comment you're referring to is about compatible requirements that pip doesn't handle | 02:28 |
| lifeless | tchaypo: I don't think your rephrasing is the same thing at all? | 02:29 |
| clarkb | ianw: I haven't looked at it since, but you should be able to run the js locally against our cluster and see what breaks | 02:29 |
| tchaypo | lifeless: in my testing, “pip install -r requirements.txt” installs bar==2.0, which conflicts with what foo requires | 02:29 |
| tchaypo | lifeless: if my rephrasing is wrong it’s probably because I haven’t understood the problem | 02:29 |
| clarkb | ianw: http://logstash.openstack.org/elasticsearch is the es REST api root | 02:29 |
| ianw | clarkb: hmm, maybe i'd be better just writing a visualisation of a query directly | 02:30 |
| ianw | clarkb: i'm trying to figure out how to get decent stats on memory usage of jobs | 02:30 |
| lifeless | tchaypo: ok so firstly we need to make sure we're talking about the same thing | 02:30 |
| lifeless | tchaypo: there is no package foo inopenstack | 02:31 |
| ianw | clarkb: seeing as hosts don't have access to push into graphite, storing a file in a known format in logstash seems the best way forward | 02:31 |
| lifeless | tchaypo: so either you've made some test data up, in which case I need to see it, because there are several hidden variables, such as the use of pbr or not. | 02:31 |
| clarkb | ianw: it may be though we only keep that indexed for a short period of time | 02:31 |
| lifeless | tchaypo: or, you're replacing actual package names with foo and bar to make clear its generic, but at the cost of the same hidden data. | 02:31 |
| *** mahito has quit IRC | 02:31 | |
| *** asettle has quit IRC | 02:32 | |
| clarkb | ianw: I will try to play with running newer kibana against the cluster some time soon | 02:32 |
| ianw | clarkb: yeah; the other option seems to be to do was subunit2sql did, and create a special path that maybe reads a log file from devstack/devstack-gate in a known format, and pushes arbitrary stats into graphite | 02:32 |
| clarkb | I also need to upgrade the cluster but its been pretty solid on 0.90.9 and 1.x has been full of security issues | 02:32 |
| *** liusheng has joined #openstack-infra | 02:33 | |
| *** mahito has joined #openstack-infra | 02:34 | |
| tchaypo | lifeless: one second while I push some repos | 02:35 |
| *** SumitNaiksatam has joined #openstack-infra | 02:35 | |
| *** dannywilson has quit IRC | 02:36 | |
| *** wenlock has quit IRC | 02:42 | |
| *** crc32 has quit IRC | 02:43 | |
| *** teran_ has quit IRC | 02:43 | |
| mordred | clarkb, ianw: fwiw, graphite is pretty low overhead - perhaps we should run a second graphite that can be used to shove test run metrics in to if we have some legitimate uses for doing such a thing | 02:45 |
| mordred | I mean, my original thought would have been that due to the wildly variable nature of things in the gate, that it would be garbage data | 02:45 |
| *** tqtran has joined #openstack-infra | 02:46 | |
| mordred | but if there are things like that that would be useful ... maybe a case can be made that it's not a crazy thing to have? | 02:46 |
| * mordred thinking out loud - may be stupd | 02:46 | |
| clarkb | jogo: so my dhcp tcpdump is only seeing the requests I think we may have asymetric routing | 02:47 |
| *** dims has joined #openstack-infra | 02:48 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net runs https://review.openstack.org/174139 | 02:48 |
| ianw | mordred: my first thought is around tracking memory usage -- https://review.openstack.org/#/c/171919/ | 02:49 |
| ianw | mordred: keeping stats for successful verify pipeline jobs ... i.e. for what we merge | 02:49 |
| *** tqtran has quit IRC | 02:50 | |
| mordred | ianw: nod. well - honestly, if it's something like _that_ then it wouldn't be too hard to get it into the current graphite | 02:51 |
| mordred | if you put the numbers into a file | 02:51 |
| lifeless | what numbers are we interested in? | 02:51 |
| tchaypo | lifeless: okay - https://github.com/jamezpolley/161047-foo, https://github.com/jamezpolley/161047-bar, and https://github.com/jamezpolley/161047-requirements | 02:51 |
| ianw | mordred: yeah, but who pushes it in? | 02:51 |
| mordred | and then we have something similar to the logstash processor workers that notice jobs are done, grab the file and inject the data | 02:52 |
| ianw | lifeless: as a very first number, peak memory usage. but things like runtime, disk usage, timing breakdown, more specific memory numbers for subprojects, etc should all be considered | 02:52 |
| clarkb | ya having another set of workers that operate in the same post job data processing cluster would be fine | 02:53 |
| mordred | ianw: somethign that isn't the job- so publish the dstat file to logs.o.o as an artifact like you're collecting there - then it's just a matter of grabbing the files associated with successful gate jobs, reading them and writing them in | 02:53 |
| *** dims has quit IRC | 02:53 | |
| mordred | I think it's actually a good idea and should be pretty simple to do | 02:53 |
| clarkb | and that could talk to the existing graphite service problably | 02:53 |
| mordred | yup | 02:53 |
| lifeless | ianw: some of those seem like things we should capture in the subunit test stream | 02:53 |
| ianw | mordred: right, so i think as i said above basically copying the subunit2sql path, but for dstat | 02:53 |
| lifeless | ianw: e.g. per-test CPU use could be captured | 02:53 |
| mordred | yah | 02:53 |
| clarkb | I am not sure the subunit stream helps us here | 02:54 |
| lifeless | ianw: other stuff, separate gather and store makes sense to me | 02:54 |
| mordred | ianw: but the subunit2sql straem runs at the wrong time | 02:54 |
| clarkb | tempests view of the world is very small particularly as we build out multinode test envs | 02:54 |
| mordred | but conceptually, yes | 02:54 |
| lifeless | clarkb: sure, for that it doesn't fit super well. OTOH you could gather it in the test, and interleave it into the stream. | 02:54 |
| lifeless | clarkb: that is actually one of the reasons subunit has the structure it has | 02:55 |
| *** Longgeek has joined #openstack-infra | 02:57 | |
| ianw | lifeless: so i'm thinking global things like time spent running openstack-client commands, similar to the tracing sdague did recently, we could potentially want to track. | 02:57 |
| ianw | lifeless: i think the subunit results are great when you're drilling down | 02:58 |
| lifeless | ianw: so- openstack-client command time is definitely something we should gather within tempest | 02:58 |
| lifeless | ianw: I'd attach that per test as a JSON attachment | 02:58 |
| lifeless | ianw: reporting over a whole run is then a simple sum | 02:59 |
| clarkb | lifeless: tempest isn't touching it | 02:59 |
| ianw | lifeless: yeah, but for outside that ... for devstack itself | 02:59 |
| lifeless | clarkb: doesn't tempest run openstack-client within various tests? | 02:59 |
| clarkb | lifeless: it does not | 02:59 |
| lifeless | clarkb: ah, ok | 02:59 |
| lifeless | ianw: oh right, devstack time. Sure. | 02:59 |
| lifeless | ianw: there is BTW a command line tool for generating subunit, if you did want to encapsulate it | 02:59 |
| lifeless | I'm not saying you should :) | 03:00 |
| *** jerryz has quit IRC | 03:00 | |
| ianw | yeah, and when centos7 started running out of memory, it would have been great to look at a graph of peak memory usage and i would have seen a jump after i committed the changes to use all pip libraries | 03:00 |
| lifeless | I'm a huge +1 on gathering more data and making it available | 03:00 |
| lifeless | I hope i'm not distracting with my mild kibbitzing | 03:01 |
| lifeless | tchaypo: ok, so I've got those | 03:01 |
| ianw | lifeless: i think the complexity has risen to a point where i'll write something up to get buy-in before i just start throwing reviews out | 03:02 |
| lifeless | tchaypo: you are using requirements.txt in that example the way pip intends it to be used. Not the way pbr does. | 03:02 |
| lifeless | tchaypo: to reproduce, you need to replace requirements.txt with a third package 'ourthing' that depends on foo and bar<=2.0 | 03:03 |
| tchaypo | and then pip install ourthing, and I should see breakage because of the conflict between foo->bar>=2.1 and ourthing->bar<=2.0 | 03:04 |
| *** spzala has quit IRC | 03:06 | |
| lifeless | no | 03:07 |
| lifeless | there should be no dep >=2.1 to reconstruct the example in the spec | 03:07 |
| *** marun_ has quit IRC | 03:07 | |
| lifeless | foo's dep should be on 'bar' | 03:07 |
| lifeless | what you're trying to trigger is the evaluation of <=2.0 after the evaluation of 'any' | 03:10 |
| lifeless | for bar | 03:10 |
| lifeless | it may require a slight tweaking of your test data - the example in the spec while a genuine example of 2a may be a little tricky to trigger | 03:10 |
| *** armax has joined #openstack-infra | 03:15 | |
| *** asettle has joined #openstack-infra | 03:21 | |
| *** mahito has quit IRC | 03:22 | |
| *** mahito has joined #openstack-infra | 03:23 | |
| *** baoli has quit IRC | 03:26 | |
| openstackgerrit | Merged openstack-dev/pbr: Fixup semver https://review.openstack.org/124651 | 03:28 |
| lifeless | \o/ | 03:30 |
| *** marun_ has joined #openstack-infra | 03:31 | |
| *** mahito has quit IRC | 03:31 | |
| *** achanda has joined #openstack-infra | 03:33 | |
| *** thingee has joined #openstack-infra | 03:36 | |
| *** mrmartin has joined #openstack-infra | 03:39 | |
| *** yamamoto has joined #openstack-infra | 03:41 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Use envassert for integration tests https://review.openstack.org/173592 | 03:43 |
| *** mahito has joined #openstack-infra | 03:44 | |
| *** marun_ has quit IRC | 03:44 | |
| *** Daisy has quit IRC | 03:45 | |
| *** mahito has quit IRC | 03:48 | |
| *** mriedem has quit IRC | 03:57 | |
| *** garyk has quit IRC | 03:58 | |
| *** mriedem has joined #openstack-infra | 03:59 | |
| *** mahito has joined #openstack-infra | 04:03 | |
| *** mriedem has quit IRC | 04:04 | |
| *** Longgeek_ has joined #openstack-infra | 04:05 | |
| *** Longgeek has quit IRC | 04:06 | |
| *** amotoki has joined #openstack-infra | 04:07 | |
| *** mrmartin has quit IRC | 04:20 | |
| *** tkelsey has joined #openstack-infra | 04:23 | |
| *** Daisy has joined #openstack-infra | 04:25 | |
| *** garyk has joined #openstack-infra | 04:28 | |
| *** tkelsey has quit IRC | 04:29 | |
| *** achanda has quit IRC | 04:30 | |
| *** frickler_ is now known as frickler | 04:43 | |
| *** yfried has quit IRC | 04:44 | |
| *** dhritishikhar has joined #openstack-infra | 04:44 | |
| *** afazekas has quit IRC | 04:51 | |
| *** otter768 has quit IRC | 04:59 | |
| *** tsg has joined #openstack-infra | 05:00 | |
| *** achanda has joined #openstack-infra | 05:00 | |
| *** stevemar has joined #openstack-infra | 05:02 | |
| *** fedexo has quit IRC | 05:04 | |
| *** dhritishikhar_ has joined #openstack-infra | 05:06 | |
| *** fedexo has joined #openstack-infra | 05:08 | |
| *** sks has joined #openstack-infra | 05:09 | |
| *** dhritishikhar has quit IRC | 05:10 | |
| *** mrmartin has joined #openstack-infra | 05:12 | |
| *** mrmartin has quit IRC | 05:15 | |
| *** mahito has quit IRC | 05:15 | |
| *** Kennan has quit IRC | 05:17 | |
| *** timfreund has quit IRC | 05:17 | |
| *** Kennan has joined #openstack-infra | 05:17 | |
| *** Krinkle is now known as Krinkle|detached | 05:18 | |
| *** dimtruck is now known as zz_dimtruck | 05:18 | |
| *** mahito has joined #openstack-infra | 05:20 | |
| *** wwwbukolaycom has joined #openstack-infra | 05:25 | |
| *** sc` has left #openstack-infra | 05:29 | |
| * wwwbukolaycom Click http://www.bukolay.com I Come. | 05:30 | |
| *** stevemar has quit IRC | 05:40 | |
| *** stevemar has joined #openstack-infra | 05:40 | |
| *** yfried has joined #openstack-infra | 05:44 | |
| *** dims has joined #openstack-infra | 05:45 | |
| *** hdd has joined #openstack-infra | 05:50 | |
| *** dims has quit IRC | 05:50 | |
| *** pal has joined #openstack-infra | 05:52 | |
| *** yfried is now known as yfried|afk | 05:54 | |
| *** yfried|afk is now known as yfried | 05:54 | |
| *** BharatK has quit IRC | 05:55 | |
| *** xyang1 has quit IRC | 06:05 | |
| *** teran has joined #openstack-infra | 06:16 | |
| *** mrunge has joined #openstack-infra | 06:17 | |
| *** BharatK has joined #openstack-infra | 06:18 | |
| *** bnemec has quit IRC | 06:18 | |
| *** jcoufal has joined #openstack-infra | 06:18 | |
| *** thingee has quit IRC | 06:19 | |
| *** armax_ has joined #openstack-infra | 06:22 | |
| *** armax has quit IRC | 06:23 | |
| *** armax_ is now known as armax | 06:23 | |
| openstackgerrit | Merged openstack-infra/project-config: enable ceilometerclient functional tests on master only https://review.openstack.org/174103 | 06:24 |
| *** stevemar has quit IRC | 06:25 | |
| *** timfreund has joined #openstack-infra | 06:28 | |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed https://review.openstack.org/173408 | 06:28 |
| openstack | bug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,New] https://launchpad.net/bugs/1444806 | 06:28 |
| mordred | lifeless: yay! | 06:29 |
| openstackgerrit | Bharat Kumar Kobagana proposed openstack-infra/project-config: Skip test_volume_boot_pattern for GlusterFS until bug 1444806 is fixed https://review.openstack.org/173408 | 06:29 |
| openstack | bug 1444806 in Cinder "test_volume_boot_pattern tempest test failure for glusterfs backend - Part 2" [Undecided,New] https://launchpad.net/bugs/1444806 | 06:29 |
| *** jcoufal has quit IRC | 06:30 | |
| *** zhiwei has joined #openstack-infra | 06:34 | |
| *** Kennan2 has joined #openstack-infra | 06:35 | |
| *** Kennan has quit IRC | 06:36 | |
| *** armax has quit IRC | 06:38 | |
| *** Ala has joined #openstack-infra | 06:39 | |
| *** fedexo has quit IRC | 06:41 | |
| *** asettle has quit IRC | 06:44 | |
| *** sergsh has joined #openstack-infra | 06:47 | |
| *** stevemar has joined #openstack-infra | 06:54 | |
| *** tnovacik has joined #openstack-infra | 06:55 | |
| *** tsg_ has joined #openstack-infra | 06:56 | |
| *** tsg has quit IRC | 06:59 | |
| *** otter768 has joined #openstack-infra | 07:00 | |
| *** stevemar has quit IRC | 07:00 | |
| *** bnemec has joined #openstack-infra | 07:00 | |
| *** achanda has quit IRC | 07:02 | |
| * wwwbukolaycom Please add my skype address. burculayd | 07:02 | |
| *** wwwbukolaycom has quit IRC | 07:03 | |
| *** dansmith has quit IRC | 07:04 | |
| *** achanda has joined #openstack-infra | 07:04 | |
| *** scheuran has joined #openstack-infra | 07:04 | |
| *** otter768 has quit IRC | 07:04 | |
| *** markus_z has joined #openstack-infra | 07:05 | |
| *** neoice has quit IRC | 07:05 | |
| *** yfried is now known as yfried|afk | 07:05 | |
| *** neoice has joined #openstack-infra | 07:05 | |
| *** praveenkumar has quit IRC | 07:06 | |
| *** sks has quit IRC | 07:06 | |
| *** BharatK has quit IRC | 07:07 | |
| *** dansmith has joined #openstack-infra | 07:08 | |
| *** dansmith is now known as Guest67672 | 07:08 | |
| *** yfried|afk is now known as yfried | 07:09 | |
| *** mpavone has joined #openstack-infra | 07:13 | |
| *** liusheng has quit IRC | 07:18 | |
| *** vdo_ has joined #openstack-infra | 07:19 | |
| *** dtantsur|afk is now known as dtantsur | 07:19 | |
| *** tsg_ has quit IRC | 07:24 | |
| *** achanda has quit IRC | 07:25 | |
| lifeless | mordred: indeed :) | 07:25 |
| *** luqas has joined #openstack-infra | 07:25 | |
| *** BharatK has joined #openstack-infra | 07:28 | |
| *** sks has joined #openstack-infra | 07:28 | |
| *** tnovacik has quit IRC | 07:28 | |
| *** _nadya_ has joined #openstack-infra | 07:30 | |
| *** praveenkumar has joined #openstack-infra | 07:30 | |
| *** esker has quit IRC | 07:30 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 07:34 |
| *** soren has joined #openstack-infra | 07:35 | |
| *** belmoreira has joined #openstack-infra | 07:37 | |
| *** hashar has joined #openstack-infra | 07:38 | |
| *** jlanoux has joined #openstack-infra | 07:39 | |
| *** chlong has quit IRC | 07:40 | |
| *** jistr has joined #openstack-infra | 07:43 | |
| *** vlaza has joined #openstack-infra | 07:44 | |
| *** dhritishikhar_ has quit IRC | 07:44 | |
| *** arxcruz has joined #openstack-infra | 07:46 | |
| *** dizquierdo has joined #openstack-infra | 07:47 | |
| *** devvesa has joined #openstack-infra | 07:47 | |
| *** MaxV has joined #openstack-infra | 07:48 | |
| *** devvesa has quit IRC | 07:50 | |
| openstackgerrit | Jan Provaznik proposed openstack/requirements: Add python-tuskarclient to requirements https://review.openstack.org/174233 | 07:50 |
| *** Hal has joined #openstack-infra | 07:55 | |
| *** Hal is now known as Guest87082 | 07:56 | |
| *** devvesa has joined #openstack-infra | 07:57 | |
| openstackgerrit | melanie witt proposed openstack-infra/project-config: Remove stable-compat-jobs from python-novaclient https://review.openstack.org/173547 | 07:58 |
| *** doude has quit IRC | 07:59 | |
| *** ihrachyshka has joined #openstack-infra | 07:59 | |
| *** esker has joined #openstack-infra | 08:01 | |
| *** dhritishikhar_ has joined #openstack-infra | 08:04 | |
| *** andreykurilin has joined #openstack-infra | 08:05 | |
| *** soren has quit IRC | 08:05 | |
| *** doude has joined #openstack-infra | 08:06 | |
| *** _nadya_ has quit IRC | 08:06 | |
| *** dalgaaf has joined #openstack-infra | 08:07 | |
| *** sergsh has quit IRC | 08:07 | |
| *** ociuhandu has joined #openstack-infra | 08:08 | |
| openstackgerrit | Ghe Rivero proposed openstack-infra/system-config: Fix doc. nodepool usage https://review.openstack.org/174238 | 08:10 |
| *** derekh has joined #openstack-infra | 08:19 | |
| openstackgerrit | Serg Melikyan proposed openstack-infra/project-config: [gate-murano-devstack-dsvm] Fix bug with neutron misconfiguration https://review.openstack.org/172447 | 08:19 |
| *** hichihara has quit IRC | 08:20 | |
| *** andrey-mp has joined #openstack-infra | 08:24 | |
| *** e0ne has joined #openstack-infra | 08:24 | |
| *** andreykurilin has quit IRC | 08:26 | |
| *** ildikov has joined #openstack-infra | 08:26 | |
| *** pal has quit IRC | 08:27 | |
| *** mrunge has quit IRC | 08:29 | |
| *** zz_johnthetubagu is now known as johnthetubaguy | 08:29 | |
| *** derekh is now known as derekh_afk | 08:30 | |
| *** ssam2 has joined #openstack-infra | 08:31 | |
| *** aderyugin has quit IRC | 08:32 | |
| *** skolekonov_ has quit IRC | 08:32 | |
| *** skolekonov has joined #openstack-infra | 08:34 | |
| *** aderyugin has joined #openstack-infra | 08:35 | |
| *** sdake__ has quit IRC | 08:36 | |
| andrey-mp | please review this - https://review.openstack.org/#/c/173945/ | 08:37 |
| *** _nadya_ has joined #openstack-infra | 08:38 | |
| *** sdake has joined #openstack-infra | 08:39 | |
| *** teran has quit IRC | 08:39 | |
| *** yfried_ has joined #openstack-infra | 08:39 | |
| *** viglesias has quit IRC | 08:39 | |
| *** viglesias has joined #openstack-infra | 08:40 | |
| *** jlibosva has joined #openstack-infra | 08:40 | |
| *** mestery has joined #openstack-infra | 08:40 | |
| *** yfried has quit IRC | 08:41 | |
| *** aderyugin has quit IRC | 08:43 | |
| *** skolekonov has quit IRC | 08:43 | |
| *** mestery_ has quit IRC | 08:43 | |
| *** mrunge has joined #openstack-infra | 08:43 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/system-config: Move server class call outside of zuul_*.pp classes https://review.openstack.org/174258 | 08:46 |
| *** pal has joined #openstack-infra | 08:47 | |
| *** e0ne has quit IRC | 08:48 | |
| *** notnownikki has joined #openstack-infra | 08:50 | |
| *** garyk has quit IRC | 08:51 | |
| *** garyk has joined #openstack-infra | 08:52 | |
| *** esker has quit IRC | 08:52 | |
| *** jyuso1 has joined #openstack-infra | 08:53 | |
| *** ilyashakhat has quit IRC | 08:55 | |
| *** skraynev has quit IRC | 08:55 | |
| *** vladiskuz has quit IRC | 08:55 | |
| *** freerunner has quit IRC | 08:56 | |
| *** otter768 has joined #openstack-infra | 09:01 | |
| *** vdo_ has quit IRC | 09:01 | |
| *** sergsh has joined #openstack-infra | 09:01 | |
| *** otter768 has quit IRC | 09:05 | |
| *** dtantsur is now known as dtantsur|brb | 09:06 | |
| *** freerunner has joined #openstack-infra | 09:06 | |
| *** e0ne has joined #openstack-infra | 09:09 | |
| *** jaypipes has quit IRC | 09:10 | |
| *** e0ne has quit IRC | 09:10 | |
| *** yfried__ has joined #openstack-infra | 09:10 | |
| *** ilyashakhat has joined #openstack-infra | 09:12 | |
| *** skolekonov has joined #openstack-infra | 09:12 | |
| *** skraynev has joined #openstack-infra | 09:12 | |
| *** yfried__ is now known as yfried|afk | 09:13 | |
| *** rm_work is now known as rm_work|away | 09:13 | |
| *** yfried_ has quit IRC | 09:14 | |
| *** esker has joined #openstack-infra | 09:17 | |
| *** andrey-mp has quit IRC | 09:19 | |
| *** dims has joined #openstack-infra | 09:19 | |
| *** yamamoto has quit IRC | 09:23 | |
| *** dims has quit IRC | 09:24 | |
| *** deepakcs has joined #openstack-infra | 09:25 | |
| *** aderyugin has joined #openstack-infra | 09:27 | |
| *** woodster_ has quit IRC | 09:30 | |
| *** ildikov has quit IRC | 09:32 | |
| *** mahito has quit IRC | 09:33 | |
| *** aswadr has quit IRC | 09:36 | |
| *** yfried|afk is now known as yfried__ | 09:37 | |
| *** esker has quit IRC | 09:38 | |
| *** teran has joined #openstack-infra | 09:39 | |
| *** vladiskuz has joined #openstack-infra | 09:39 | |
| *** teran has quit IRC | 09:44 | |
| *** yfried__ is now known as yfried|afk | 09:47 | |
| *** ildikov has joined #openstack-infra | 09:47 | |
| *** afazekas has joined #openstack-infra | 09:47 | |
| *** yamamoto has joined #openstack-infra | 09:49 | |
| *** soren has joined #openstack-infra | 09:55 | |
| *** teran has joined #openstack-infra | 09:55 | |
| *** teran has quit IRC | 09:56 | |
| *** teran has joined #openstack-infra | 09:56 | |
| *** yfried|afk is now known as yfried__ | 09:59 | |
| *** tqtran has joined #openstack-infra | 10:01 | |
| *** hashar is now known as hasharLunch | 10:03 | |
| *** dims has joined #openstack-infra | 10:06 | |
| *** jamespag` has quit IRC | 10:11 | |
| *** gnuoy` has quit IRC | 10:11 | |
| *** gnuoy has joined #openstack-infra | 10:11 | |
| *** jamespage has joined #openstack-infra | 10:11 | |
| *** tnovacik has joined #openstack-infra | 10:13 | |
| *** [HeOS] has quit IRC | 10:13 | |
| *** yfried__ is now known as yfried|afk | 10:15 | |
| *** spredzy|afk is now known as spredzy | 10:17 | |
| *** yamamoto has quit IRC | 10:19 | |
| *** asselin has quit IRC | 10:19 | |
| *** pelix has quit IRC | 10:19 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Move server outside puppetdb class https://review.openstack.org/174286 | 10:19 |
| *** pelix has joined #openstack-infra | 10:22 | |
| *** deepakcs has quit IRC | 10:23 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Move server outside puppetdb class https://review.openstack.org/174286 | 10:24 |
| *** vlaza1 has joined #openstack-infra | 10:27 | |
| *** vlaza has quit IRC | 10:27 | |
| *** tqtran has quit IRC | 10:27 | |
| *** luqas has quit IRC | 10:28 | |
| *** ajo_ has joined #openstack-infra | 10:31 | |
| *** vlaza1 is now known as vlaza | 10:31 | |
| *** luqas has joined #openstack-infra | 10:32 | |
| *** ajo has quit IRC | 10:32 | |
| *** luqas has quit IRC | 10:33 | |
| *** luqas has joined #openstack-infra | 10:34 | |
| *** yfried|afk is now known as yfried__ | 10:39 | |
| *** katyafervent is now known as katyafervent_awa | 10:42 | |
| *** katyafervent_awa is now known as katyafervent | 10:44 | |
| *** kmartin_ has quit IRC | 10:46 | |
| *** yamamoto has joined #openstack-infra | 10:50 | |
| *** pc_m has joined #openstack-infra | 10:52 | |
| *** derekh_afk is now known as derekh | 10:54 | |
| *** dtantsur|brb is now known as dtantsur | 10:56 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** yamamoto has joined #openstack-infra | 11:01 | |
| *** otter768 has joined #openstack-infra | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** otter768 has quit IRC | 11:06 | |
| *** erlon has joined #openstack-infra | 11:08 | |
| *** e0ne has joined #openstack-infra | 11:10 | |
| *** yamamoto has joined #openstack-infra | 11:12 | |
| *** amotoki_ has joined #openstack-infra | 11:15 | |
| *** marcusvrn1 has joined #openstack-infra | 11:15 | |
| *** marcusvrn has quit IRC | 11:18 | |
| *** claudiub has joined #openstack-infra | 11:20 | |
| *** cdent has joined #openstack-infra | 11:21 | |
| *** esker has joined #openstack-infra | 11:22 | |
| *** doug-fish has joined #openstack-infra | 11:23 | |
| *** HeOS has joined #openstack-infra | 11:26 | |
| *** esker has quit IRC | 11:27 | |
| *** hasharLunch is now known as hashar | 11:32 | |
| *** dizquierdo has quit IRC | 11:32 | |
| *** Daisy has quit IRC | 11:34 | |
| *** ldnunes has joined #openstack-infra | 11:38 | |
| *** pal has quit IRC | 11:38 | |
| *** jamielennox is now known as jamielennox|away | 11:47 | |
| *** fhubik has joined #openstack-infra | 11:49 | |
| *** weshay has joined #openstack-infra | 11:52 | |
| *** ajo_ is now known as ajo | 11:56 | |
| *** tiswanso has quit IRC | 12:00 | |
| *** luqas has quit IRC | 12:07 | |
| *** kgiusti has joined #openstack-infra | 12:09 | |
| *** isviridov_away is now known as isviridov | 12:14 | |
| *** _nadya_ has quit IRC | 12:14 | |
| *** dims has quit IRC | 12:14 | |
| *** dims__ has joined #openstack-infra | 12:17 | |
| *** zhiwei has quit IRC | 12:18 | |
| *** abregman has joined #openstack-infra | 12:18 | |
| *** radez_g0n3 is now known as radez | 12:19 | |
| *** dprince has joined #openstack-infra | 12:20 | |
| *** e0ne has quit IRC | 12:20 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** yfried__ is now known as yfried|afk | 12:21 | |
| *** stevemar has joined #openstack-infra | 12:21 | |
| *** julim has quit IRC | 12:22 | |
| *** sks has quit IRC | 12:22 | |
| *** esker has joined #openstack-infra | 12:23 | |
| *** mgedmin_ has joined #openstack-infra | 12:25 | |
| *** esker has quit IRC | 12:27 | |
| *** gordc has joined #openstack-infra | 12:28 | |
| *** fhubik is now known as fhubik_afk | 12:29 | |
| *** dkranz has joined #openstack-infra | 12:29 | |
| *** gordc has quit IRC | 12:31 | |
| *** pal has joined #openstack-infra | 12:32 | |
| *** mgedmin_ has quit IRC | 12:33 | |
| *** bswartz has quit IRC | 12:34 | |
| *** yfried|afk is now known as yfried__ | 12:37 | |
| *** gordc has joined #openstack-infra | 12:38 | |
| *** tiswanso has joined #openstack-infra | 12:43 | |
| *** rlandy has joined #openstack-infra | 12:43 | |
| *** dhritishikhar_ has quit IRC | 12:43 | |
| *** baoli has joined #openstack-infra | 12:44 | |
| *** aysyd has joined #openstack-infra | 12:46 | |
| *** asettle has joined #openstack-infra | 12:46 | |
| openstackgerrit | Anthony Michon proposed openstack-infra/project-config: Cerberus and Sticks: remove python3 tests https://review.openstack.org/174358 | 12:47 |
| *** andrey-mp has joined #openstack-infra | 12:47 | |
| *** julim has joined #openstack-infra | 12:47 | |
| *** bknudson has joined #openstack-infra | 12:52 | |
| *** tnovacik has quit IRC | 12:52 | |
| *** andrey-mp has quit IRC | 12:54 | |
| *** tnovacik has joined #openstack-infra | 12:54 | |
| *** ildikov has quit IRC | 12:55 | |
| *** dboik has joined #openstack-infra | 12:55 | |
| *** mriedem has joined #openstack-infra | 12:56 | |
| *** sarob has joined #openstack-infra | 12:57 | |
| *** rkukura has quit IRC | 12:57 | |
| *** Prophidys1 has joined #openstack-infra | 13:00 | |
| *** woodster_ has joined #openstack-infra | 13:00 | |
| *** eharney has quit IRC | 13:00 | |
| *** baoli has quit IRC | 13:01 | |
| *** mpaolino has joined #openstack-infra | 13:01 | |
| *** baoli has joined #openstack-infra | 13:02 | |
| *** fhubik_afk is now known as fhubik | 13:02 | |
| *** Guest87082 has quit IRC | 13:02 | |
| *** otter768 has joined #openstack-infra | 13:03 | |
| *** achuprin has joined #openstack-infra | 13:03 | |
| *** annegentle has joined #openstack-infra | 13:04 | |
| *** Hal has joined #openstack-infra | 13:05 | |
| *** Hal is now known as Guest9920 | 13:05 | |
| *** otter768 has quit IRC | 13:07 | |
| *** luqas has joined #openstack-infra | 13:08 | |
| openstackgerrit | Merged openstack-infra/system-config: Fix doc. nodepool usage https://review.openstack.org/174238 | 13:09 |
| *** zhiwei has joined #openstack-infra | 13:11 | |
| *** _nadya_ has joined #openstack-infra | 13:12 | |
| *** luqas has quit IRC | 13:13 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: remove lib-forward-testing jobs from libs with stable branches https://review.openstack.org/174371 | 13:13 |
| *** sputnik13 has joined #openstack-infra | 13:14 | |
| *** sputnik13 has quit IRC | 13:14 | |
| *** tiswanso has quit IRC | 13:14 | |
| *** wenlock has joined #openstack-infra | 13:15 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack-infra/devstack-gate: Run sahara in distributed mode https://review.openstack.org/172494 | 13:16 |
| *** bswartz has joined #openstack-infra | 13:17 | |
| *** peristeri has joined #openstack-infra | 13:18 | |
| *** fhubik is now known as fhubik_afk | 13:19 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update register_machine to use tasks https://review.openstack.org/171324 | 13:19 |
| *** dustins has joined #openstack-infra | 13:19 | |
| *** Longgeek_ has quit IRC | 13:20 | |
| *** fhubik_afk is now known as fhubik | 13:20 | |
| *** spzala has joined #openstack-infra | 13:20 | |
| openstackgerrit | Julia Kreger proposed openstack-infra/shade: Update unregister_machine to use tasks https://review.openstack.org/171325 | 13:21 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: remove stable-compat jobs from libs with stable branches https://review.openstack.org/174371 | 13:22 |
| *** soren has quit IRC | 13:23 | |
| *** sdake_ has joined #openstack-infra | 13:26 | |
| *** yamamoto has joined #openstack-infra | 13:28 | |
| *** ddieterly has joined #openstack-infra | 13:29 | |
| *** sdake has quit IRC | 13:29 | |
| *** markvoelker has joined #openstack-infra | 13:30 | |
| *** tnovacik has quit IRC | 13:32 | |
| *** luqas has joined #openstack-infra | 13:32 | |
| openstackgerrit | Doug Hellmann proposed openstack/requirements: Disable syncs to libs to unjam master requirements changes https://review.openstack.org/174383 | 13:35 |
| *** Guest67672 is now known as dansmith | 13:36 | |
| *** dansmith is now known as Guest14453 | 13:37 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Stop mkdir output on each job write https://review.openstack.org/167250 | 13:38 |
| *** esker has joined #openstack-infra | 13:39 | |
| *** signed8bit has joined #openstack-infra | 13:40 | |
| *** mwagner_lap has quit IRC | 13:40 | |
| *** Guest14453 is now known as dansmith_ | 13:43 | |
| openstackgerrit | Merged openstack-infra/jenkins-job-builder: Added support for JClouds Blobstore https://review.openstack.org/107515 | 13:43 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Allow gnocchi-release to create branch https://review.openstack.org/174387 | 13:43 |
| *** esker has quit IRC | 13:43 | |
| *** mattfarina has joined #openstack-infra | 13:47 | |
| *** fhubik is now known as fhubik_afk | 13:47 | |
| *** annegentle has quit IRC | 13:48 | |
| *** amitgandhinz has joined #openstack-infra | 13:51 | |
| *** Sukhdev has joined #openstack-infra | 13:52 | |
| openstackgerrit | Sergey Kraynev proposed openstack-infra/project-config: Enable ceilometer service on Heat gates https://review.openstack.org/174394 | 13:53 |
| *** tiswanso has joined #openstack-infra | 13:53 | |
| clarkb | jogo http://logs.openstack.org/39/174139/5/experimental/check-tempest-dsvm-multinode-full/3a68823/logs/dhcpdump.txt.gz I think the br_flat interface is not where the dhcpd responses are sent, any idea what interface that would be on? | 13:55 |
| *** signed8bit has quit IRC | 13:56 | |
| clarkb | zaro new gerrit 2.10.3 release today | 13:56 |
| *** jgrimm is now known as zz_jgrimm | 13:56 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 13:56 | |
| openstackgerrit | Merged openstack-infra/shade: Wrap ironicclient methods that leak objects https://review.openstack.org/173567 | 13:57 |
| clarkb | jogo I am out today(about to get in car for much driving) so feel free to update that change. we can use 'any' as the interface but that will catch hpclouds dhcp too | 13:57 |
| *** dansmith_ is now known as dansmith | 13:58 | |
| *** Krinkle|detached is now known as Krinkle | 14:02 | |
| *** fhubik_afk is now known as fhubik | 14:02 | |
| *** xyang1 has joined #openstack-infra | 14:02 | |
| *** soren has joined #openstack-infra | 14:03 | |
| *** sarob has quit IRC | 14:03 | |
| *** kaisers1 has quit IRC | 14:04 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** zhiwei has quit IRC | 14:05 | |
| anteaya | clarkb: safe travels | 14:05 |
| *** sarob has joined #openstack-infra | 14:06 | |
| *** Sukhdev has quit IRC | 14:06 | |
| openstackgerrit | Merged openstack-infra/shade: Add patch_machine method and operator unit test substrate https://review.openstack.org/171165 | 14:09 |
| *** ayoung has quit IRC | 14:09 | |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Allow gnocchi-release to create branch https://review.openstack.org/174387 | 14:09 |
| *** HenryG_ is now known as HenryG | 14:11 | |
| *** dboik has quit IRC | 14:12 | |
| *** dboik has joined #openstack-infra | 14:13 | |
| *** yfried__ is now known as yfried|afk | 14:15 | |
| *** yfried|afk is now known as yfried__ | 14:15 | |
| *** mwagner_lap has joined #openstack-infra | 14:16 | |
| *** markvoelker_ has joined #openstack-infra | 14:16 | |
| *** mrmartin has joined #openstack-infra | 14:16 | |
| *** sarob has quit IRC | 14:16 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Clean up unused stable-compat jobs configuration https://review.openstack.org/174403 | 14:17 |
| fungi | dhellmann: sdague: there's the remaining cleanup ^ | 14:18 |
| *** AJaeger has quit IRC | 14:18 | |
| *** annegentle has joined #openstack-infra | 14:18 | |
| *** ildikov has joined #openstack-infra | 14:18 | |
| *** markvoelker has quit IRC | 14:19 | |
| *** e0ne has joined #openstack-infra | 14:19 | |
| fungi | overlimit errors for our 600 instance quota in hpcloud. doing another alien cleanup pass now | 14:21 |
| *** emagana has joined #openstack-infra | 14:21 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 14:23 |
| mordred | fungi: 164264 installs a my.cnf file on the box | 14:23 |
| rcarrillocruz | weee | 14:23 |
| rcarrillocruz | thx for those approvals mordred, they were around on my queue for some time | 14:23 |
| *** tonytan4ever has joined #openstack-infra | 14:23 | |
| mordred | fungi: it's a TERRIBLE file if someone is actually running a mysql | 14:23 |
| *** markvoelker has joined #openstack-infra | 14:23 | |
| rcarrillocruz | mordred: ^ | 14:23 |
| *** gaelL has quit IRC | 14:23 | |
| mordred | rcarrillocruz: (I'm trying to work through my review backlog :) ) | 14:23 |
| rcarrillocruz | much appreciated :-) | 14:24 |
| mordred | fungi: how much do we care about what this will or won't do for people who are running things differently from us | 14:24 |
| *** dizquierdo has joined #openstack-infra | 14:24 | |
| mordred | fungi: oh - wait - I see your comment | 14:24 |
| fungi | mordred: agreed. there's a safeguard there to only install it if the gerrit module is not configured to provide a local mysql installation, but maybe there should be a better way to only install that when the client is present but the server is not | 14:24 |
| *** mattt has left #openstack-infra | 14:25 | |
| mordred | ++ | 14:25 |
| mordred | (in that case, will people kill me if I make a pedantic suggestion?) | 14:25 |
| mordred | we should remove the mysqld_safe and mysqld sections from it and only have the client config settings so that it's clear to someone that this is a config for client settings reasons | 14:26 |
| *** markvoelker_ has quit IRC | 14:26 | |
| *** mtanino has joined #openstack-infra | 14:27 | |
| russellb | fungi: not urgent at all, but could you do another connection count check on me today? I'm wondering if it calmed itself down. If not, I'll just kill it. | 14:28 |
| *** hdd has quit IRC | 14:28 | |
| russellb | fungi: gerrit ssh connections that is | 14:28 |
| *** mpaolino has quit IRC | 14:29 | |
| *** sdake has joined #openstack-infra | 14:29 | |
| *** eharney has joined #openstack-infra | 14:30 | |
| *** sarob has joined #openstack-infra | 14:31 | |
| *** Daisy has joined #openstack-infra | 14:32 | |
| *** sdake_ has quit IRC | 14:33 | |
| *** markvoelker has quit IRC | 14:34 | |
| fungi | russellb: sure, just a sec | 14:34 |
| openstackgerrit | Merged openstack-infra/shade: Deprecate use of cache in list_volumes https://review.openstack.org/172599 | 14:34 |
| fungi | mordred: that seems reasonable | 14:34 |
| *** abregman has quit IRC | 14:35 | |
| *** abregman1 has joined #openstack-infra | 14:35 | |
| openstackgerrit | Merged openstack-infra/shade: Add test for invalidation after delete https://review.openstack.org/172600 | 14:35 |
| fungi | russellb: still 63 | 14:36 |
| *** sergmelikyan has joined #openstack-infra | 14:36 | |
| *** sarob__ has joined #openstack-infra | 14:36 | |
| *** gaelL has joined #openstack-infra | 14:36 | |
| russellb | fungi: ok, weird, i'll kill it off for now | 14:36 |
| russellb | and sorry :) | 14:36 |
| *** MarkAtwood has joined #openstack-infra | 14:36 | |
| fungi | russellb: they're all originating from the same ipv6 address so presumably all the same application | 14:37 |
| russellb | yep, it's a box running the reviewstats apps | 14:37 |
| *** abregman1 has quit IRC | 14:37 | |
| *** sarob has quit IRC | 14:37 | |
| fungi | russellb: considered trying the rest api? | 14:37 |
| *** jaypipes has joined #openstack-infra | 14:39 | |
| *** soren has quit IRC | 14:40 | |
| *** ssam2 has quit IRC | 14:41 | |
| *** hashar is now known as Jenkins_ | 14:41 | |
| openstackgerrit | Merged openstack-infra/project-config: remove stable-compat jobs from libs with stable branches https://review.openstack.org/174371 | 14:41 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: Enable websockets transport on Etherpad https://review.openstack.org/151652 | 14:42 |
| openstackgerrit | Merged openstack-infra/puppet-etherpad_lite: Add support for Apache mod_proxy_wstunnel https://review.openstack.org/154842 | 14:43 |
| *** garyk has quit IRC | 14:43 | |
| anteaya | russellb: was looking at your patch you linked to the tc mailing list, realizing I have zero idea who someone named Egle is | 14:43 |
| anteaya | russellb: that person seems to be heavy with the -2 | 14:43 |
| russellb | she's co-chair of the defcore committee | 14:44 |
| *** tsg_ has joined #openstack-infra | 14:44 | |
| *** sarob__ has quit IRC | 14:44 | |
| russellb | elected to the board this year as an individual director, works for rackspace | 14:44 |
| *** zz_dimtruck is now known as dimtruck | 14:44 | |
| rcarrillocruz | egle = eglute me thinks | 14:44 |
| russellb | i think it's a misunderstanding / unfamiliarity with how -2 and workflow-1 is generally used | 14:44 |
| *** Jenkins_ is now known as hasharAway | 14:44 | |
| eglute | yes. sorry about that | 14:45 |
| anteaya | russellb: ah okay, I'll adopt that perspective for a bit | 14:45 |
| russellb | no problem | 14:45 |
| *** mrmartin has quit IRC | 14:45 | |
| anteaya | eglute: hi, just looking at the patch, generally we tend to use -1 for I happen to disagree with this direction or how it is worded and here is why | 14:45 |
| yolanda | mordred, you are in review mode today! | 14:46 |
| anteaya | eglute: one -2 especially with a policy as difficult as defcore might be interpreted as 'I will fight to the death to keep this out' | 14:46 |
| *** e0ne has quit IRC | 14:46 | |
| anteaya | eglute: I personally have never seen both a -2 review and a -1 workflow on anything I don't think | 14:46 |
| eglute | I realize that now, i definitely did not want to send that message. I am new to gerrit, and still learning. appreciate all the help | 14:47 |
| anteaya | eglute: happy to help, glad to hear it is just a newness thing | 14:47 |
| eglute | i do hope that you comment on the defcore/technical leadership | 14:47 |
| anteaya | eglute: if you would like to change your votes on any patch at any time, open a new comment and you have the option to select new radio buttons | 14:48 |
| eglute | i did change it to just a -1, | 14:48 |
| anteaya | eglute: great, thanks | 14:48 |
| *** mrmartin has joined #openstack-infra | 14:48 | |
| eglute | thank you :) | 14:48 |
| anteaya | :) | 14:48 |
| anteaya | eglute: as for commenting I do have a patch up to attempt to get a defintion for Community into the lexicon, since I don't yet see one | 14:49 |
| *** mrunge has quit IRC | 14:49 | |
| anteaya | eglute: mostly my patch is a placeholder since I have no idea what wording should go there | 14:49 |
| *** nikhil_k is now known as flaper87-2 | 14:49 | |
| eglute | good point- please submit it | 14:49 |
| anteaya | it is submitted | 14:50 |
| *** flaper87-2 is now known as nikhil_k | 14:50 | |
| anteaya | https://review.openstack.org/#/c/173510/ | 14:50 |
| *** smccully has joined #openstack-infra | 14:50 | |
| anteaya | eglute: if you have any thoughts on what definition should be there please add as a comment and I can include that in a new patchset | 14:50 |
| *** zz_jgrimm is now known as jgrimm | 14:51 | |
| fungi | mordred: do we have the apache vhost on ci.openstack.org (a.k.a. old-wiki.openstack.org) puppeted anywhere i'm not finding, or did you just edit it by hand to add the shade alias there? | 14:53 |
| *** ildikov has quit IRC | 14:53 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/reviewstats: Handle reconnections more appropriately. https://review.openstack.org/174031 | 14:53 |
| *** mjordan has joined #openstack-infra | 14:53 | |
| fungi | i'm feeling like the time is nigh to bite the bullet and move all that content to static.openstack.org | 14:54 |
| mordred | fungi: I just added the shade alias by hand | 14:54 |
| fungi | because that server really needs to go away | 14:54 |
| fungi | k, thanks | 14:54 |
| mordred | fungi: it's not puppeted AT ALL | 14:54 |
| fungi | i'll start working on getting that moved | 14:54 |
| mordred | and yes - I'm very much ready for that server to go away | 14:54 |
| mordred | I think the 2 main things that still exist there | 14:54 |
| mordred | are ci.o.o and tarballs.o.o iirc | 14:54 |
| fungi | nope, tarballs moved to static.o.o long ago | 14:55 |
| mordred | oh - so it's literally just ci.o.o ? | 14:55 |
| *** mrmartin has quit IRC | 14:55 | |
| fungi | ttx still has one not-puppeted release-related app hosted on it, and we have some old docs redirects for things like nova.openstack.org | 14:55 |
| *** pal has quit IRC | 14:56 | |
| fungi | and the old moin wiki is still running there in case we need to copy anything else from it which got missed in the mediawiki migration, but i think the utility of that is now being outweighed by the danger of people stumbling across really old content and thinking it's current in any way | 14:56 |
| *** fhubik is now known as fhubik_afk | 14:56 | |
| StevenK | fungi: Annouce its death to a bunch of cheering? Probably mostly from mordred and ttx | 14:57 |
| fungi | well, ttx will want to relocate his thing he's still running there | 14:57 |
| StevenK | fungi: I meant old-wiki.o.o only in that case. | 14:57 |
| *** ildikov has joined #openstack-infra | 14:58 | |
| fungi | but beyond that and ci.o.o, yes the whole server could go away today and i think nobody would notice | 14:58 |
| *** sarob has joined #openstack-infra | 14:58 | |
| eglute | anteaya thank you, I will take a look | 14:59 |
| mordred | fungi: and ci.o.o should really be a redirect to docs.o.o/infra prolly | 14:59 |
| mordred | fungi: which means we should change the infra things that publish to ci.o.o to publish using infra-docs-jobs, no? | 14:59 |
| *** dizquierdo has quit IRC | 15:00 | |
| *** fhubik_afk is now known as fhubik | 15:00 | |
| jeblair | yep | 15:00 |
| jeblair | but we need to figure out the hierarchy there | 15:00 |
| fungi | right | 15:00 |
| mordred | nod | 15:00 |
| *** pal has joined #openstack-infra | 15:00 | |
| *** baoli has quit IRC | 15:01 | |
| fungi | anyway, http://ci.openstack.org/openstackid/ is live now | 15:01 |
| *** prad has joined #openstack-infra | 15:01 | |
| anteaya | eglute: thanks | 15:01 |
| *** ajmiller has joined #openstack-infra | 15:01 | |
| *** tiswanso has quit IRC | 15:01 | |
| *** baoli has joined #openstack-infra | 15:01 | |
| anteaya | eglute: anytime you want to quickly view how many open patches are against the defcore repo you can use this link: https://review.openstack.org/#/q/project:openstack/defcore+status:open,n,z | 15:02 |
| *** dboik_ has joined #openstack-infra | 15:02 | |
| *** mrmartin has joined #openstack-infra | 15:02 | |
| fungi | in theory we could start "migrating" by just adding new docs jobs for those projects currently publishing to ci.o.o and then once we're comfortable they're all current in the subtree we want, we can switch dns over and remove the old jobs at our leisure | 15:02 |
| *** tiswanso_ has joined #openstack-infra | 15:02 | |
| *** scheuran has quit IRC | 15:03 | |
| fungi | and if any of those are so low-churn that we get impatient, there's always rsync | 15:03 |
| openstackgerrit | Merged openstack-infra/puppet-gerrit: Fix Gerrit mimetype setting for direct downloading of files https://review.openstack.org/172722 | 15:03 |
| *** otter768 has joined #openstack-infra | 15:03 | |
| openstackgerrit | Russell Bryant proposed openstack-infra/reviewstats: Handle (dis|re)connect more appropriately. https://review.openstack.org/174031 | 15:04 |
| *** annegentle has quit IRC | 15:04 | |
| *** markvoelker has joined #openstack-infra | 15:04 | |
| *** dboik has quit IRC | 15:05 | |
| *** marun_ has joined #openstack-infra | 15:05 | |
| mordred | fungi: yah | 15:05 |
| jeblair | https://etherpad.openstack.org/p/infra-docs-move | 15:06 |
| jeblair | hrm. unhappy etherpad | 15:06 |
| *** amitgandhinz has quit IRC | 15:07 | |
| *** sergsh has quit IRC | 15:07 | |
| jeblair | http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=113&rra_id=5&view_type=&graph_start=1429182452&graph_end=1429196852 | 15:07 |
| *** Daisy_ has joined #openstack-infra | 15:07 | |
| mordred | jeblair: that doesn't seem like a happy graph | 15:08 |
| anteaya | like as of right now | 15:08 |
| *** otter768 has quit IRC | 15:08 | |
| anteaya | didn't we have a graph like that when someone tried to upload some sort of document on time in the past | 15:09 |
| anteaya | one time in the past | 15:09 |
| *** dannywilson has joined #openstack-infra | 15:09 | |
| fungi | wow, bam | 15:09 |
| *** Daisy has quit IRC | 15:09 | |
| *** Daisy_ is now known as Daisy | 15:10 | |
| fungi | and yeah, i think abiword runs wild on the server sometimes processing imports/exports/something | 15:10 |
| fungi | checking | 15:10 |
| *** armax has joined #openstack-infra | 15:10 | |
| fungi | whatever it was, it's subsided | 15:10 |
| *** ssam2 has joined #openstack-infra | 15:11 | |
| fungi | 5-minute load is well under 1 now | 15:11 |
| jeblair | fungi: well, i just reloaded my pad, let's see if it spikes again | 15:11 |
| *** markvoelker has quit IRC | 15:11 | |
| fungi | still falling | 15:11 |
| *** sputnik13 has joined #openstack-infra | 15:11 | |
| mordred | loading that pad works for me? | 15:11 |
| *** vlaza has quit IRC | 15:11 | |
| fungi | it does for me too | 15:11 |
| fungi | the load average is still falling | 15:12 |
| mordred | oh - still _falling_ | 15:12 |
| jeblair | seeing my typing? | 15:12 |
| mordred | I thought you said still _failing_ | 15:12 |
| fungi | i see nothing | 15:12 |
| jeblair | and i just got disconnected | 15:12 |
| fungi | typing-wise | 15:12 |
| mordred | nope. I see nothing | 15:12 |
| jeblair | fungi, mordred: why don't you type on that pad and make sure it's not me | 15:13 |
| fungi | i've typed things into that pad now | 15:13 |
| mordred | I have too | 15:13 |
| mordred | I have also seen fungi's things | 15:13 |
| fungi | yeah, mordred and i are seeing each others stuff | 15:13 |
| jeblair | okay, i will double click on my control panel and restart my browser and stuff | 15:14 |
| *** tsg_ has quit IRC | 15:14 | |
| mordred | jeblair: have you tried turning it off and on again? | 15:14 |
| *** amitgandhinz has joined #openstack-infra | 15:14 | |
| mordred | jeblair: I'm blaming systemd | 15:14 |
| fungi | or channeling roy | 15:14 |
| fungi | though your irish accent is pretty lacking, i must say | 15:14 |
| jeblair | 19879 corvus 20 0 1802120 357984 4480 S 0.0 1.1 611:47.34 systemd-browser | 15:15 |
| jeblair | mordred: yep that's the problem ^ | 15:15 |
| mordred | jeblair: please tell me that that's fake | 15:16 |
| mordred | because I'm pretty sure I don't know the difference with systemd anymore | 15:16 |
| jeblair | mordred: i think it's fake. but i'm not even sure now. | 15:17 |
| jeblair | ;) | 15:17 |
| jeblair | mordred, fungi: can you see my purple text? | 15:17 |
| *** jcoufal has joined #openstack-infra | 15:17 | |
| jroll | I definitely bought that for a moment | 15:17 |
| fungi | yep, loud and, um, purple | 15:17 |
| jeblair | okay, so erm, i guess i have entered a period of daily firefox restarts :/ | 15:17 |
| *** scheuran has joined #openstack-infra | 15:18 | |
| fungi | i blame everybody | 15:18 |
| *** rm_work|away is now known as rm_work | 15:18 | |
| fungi | let's abandon this web thing and go back to gopher | 15:18 |
| *** kmartin has joined #openstack-infra | 15:18 | |
| jeblair | fungi: i'll write this up and send you the afs path | 15:18 |
| *** dannywilson has quit IRC | 15:19 | |
| fungi | ha | 15:21 |
| mordred | jeblair: everything in that etherpad seems good to me | 15:21 |
| jeblair | so, docs.o.o/infra/system-config seem good then? | 15:21 |
| *** abhiram_moturi has joined #openstack-infra | 15:22 | |
| fungi | wfm. i guess we can live with the fact that we're also publishing other documentation to static.openstack.org (where we actually have control over apache) | 15:22 |
| jeblair | we did make 'infra-manual' special and put it in 'manual' | 15:22 |
| openstackgerrit | Julien Danjou proposed openstack-infra/project-config: Allow gnocchi-release to create branch https://review.openstack.org/174387 | 15:22 |
| jeblair | so if we wanted to make 'system-config' special and put it in 'systems' or something if we want | 15:22 |
| StevenK | docs.o.o already has a pretty index, do you guys mean docs.o.o/infra ? | 15:23 |
| jeblair | fungi: yeah, i mean, we started doing scp stuff when we thought we were going to move all the docs to that. but now, i think we have a new plan for docs (which i expect we'll start on soon) and it's going to be different, so i don't think there's value in keeping the scp publishing | 15:23 |
| fungi | i guess the documentation we're publishing to static.o.o isn't infra-specific anyway (governance, security, specs) | 15:24 |
| jeblair | StevenK: yes, thanks :) | 15:24 |
| *** alexpilotti has joined #openstack-infra | 15:24 | |
| *** ZZelle has joined #openstack-infra | 15:25 | |
| openstackgerrit | Henry Gessau proposed openstack-infra/system-config: Include completed blueprints in launchpad search https://review.openstack.org/174429 | 15:28 |
| fungi | i can go ahead and get started on the publisher macro for this if everyone's on board with docs.o.o/infra/{jenkins-job-builder,system-config,zuul,et cetera} | 15:28 |
| mordred | fungi: I'm half-way through a patch :) | 15:28 |
| fungi | have at it | 15:28 |
| *** arxcruz has quit IRC | 15:29 | |
| fungi | we'll also need to host a ci.o.o redirect somewhere. i can whip up one similar to what we have for devstack.org | 15:29 |
| fungi | then we can repoint dns for ci.o.o to static.o.o when we're good with the content on docs.o.o | 15:29 |
| fungi | it's unfortunate rackspace cloudsites is so inflexible in that regard, that we have to use an apache server to host redirects to our cloudsites content | 15:30 |
| *** tsg_ has joined #openstack-infra | 15:31 | |
| jeblair | fungi: yeah. but at least we have one around, and we _are_ still planning on moving off of cloudsites. | 15:31 |
| fungi | yep | 15:31 |
| fungi | temporary, temporary | 15:31 |
| openstackgerrit | Andrew Stangl proposed openstack-infra/jenkins-job-builder: Job DSL target needs to be plural, not singular https://review.openstack.org/174431 | 15:32 |
| *** russellb has quit IRC | 15:32 | |
| *** sabeen1 has joined #openstack-infra | 15:32 | |
| jeblair | yolanda: did you see we're running your patch in production? | 15:32 |
| yolanda | so i saw some comments | 15:32 |
| yolanda | but i wasn't sure | 15:33 |
| yolanda | and how is that performing? | 15:33 |
| *** spzala has quit IRC | 15:33 | |
| *** reed has quit IRC | 15:33 | |
| *** reed has joined #openstack-infra | 15:33 | |
| rcarrillocruz | *drumroll* | 15:33 |
| *** russellb has joined #openstack-infra | 15:35 | |
| *** yfried__ has quit IRC | 15:37 | |
| yolanda | jeblair, you have me waiting :) plese tell that performance is super now :) | 15:38 |
| dtantsur | hi infra! can someone have a look at an easy patch? https://review.openstack.org/#/c/156270/ | 15:38 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config: Publish infra docs to docs.o.o https://review.openstack.org/174435 | 15:38 |
| mordred | fungi: ^^ | 15:38 |
| jeblair | yolanda: i don't have good numbers for that. we had a lot of unrelated problems before, so i can't do a direct comparison. i'll try to get something together later | 15:38 |
| jeblair | yolanda: it's working fine though :) | 15:39 |
| yolanda | so you ran the version with the modified Task class? | 15:39 |
| mordred | jeblair, fungi and yay the patch removes more lines than it adds | 15:39 |
| rcarrillocruz | yolanda: rockstar :-) | 15:39 |
| jeblair | yolanda: yes | 15:39 |
| *** esker has joined #openstack-infra | 15:39 | |
| jeblair | yolanda: (functionally it was fine, just needed the api changes for shade forward-compatability -- your new version lgtm, btw) | 15:40 |
| mordred | jeblair: 170788 ? | 15:40 |
| yolanda | ok, i'm really willing that this speeds up things and we can adopt it | 15:40 |
| ttx | jeblair: so, I happen to be in a meeting with a Frecnh Public cloud provider and they would really really like to give Infra cloud resources | 15:40 |
| ttx | jeblair: can I answer that you could be interested ? | 15:40 |
| jeblair | ttx: absolutely! | 15:40 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add redirects from ci.o.o to docs.o.o/infra https://review.openstack.org/174437 | 15:40 |
| fungi | mordred: jeblair ^ | 15:40 |
| mordred | ttx: yes please | 15:40 |
| mordred | ttx: could you point me at them so I can sign up for a personal account and verify that shade and glance and whatnot all work? | 15:41 |
| ttx | I'll make intros soon | 15:41 |
| mordred | ttx: cool | 15:41 |
| ttx | jeblair: do you have a list of things you would like the most ? like typical RAM/cores, number of them, more pet or cattle style ? | 15:42 |
| jeblair | ttx: will email it to you now | 15:42 |
| ttx | cool thx | 15:42 |
| anteaya | ttx yay | 15:42 |
| mordred | ttx: I recently verified that RunAbove (france and quebec) and vexxhost (montreal) work well with our tools - so go french speaking people | 15:42 |
| *** erikmwilson is now known as Guest44089 | 15:43 | |
| *** erikmwil_ has joined #openstack-infra | 15:43 | |
| *** Guest44089 has quit IRC | 15:43 | |
| *** erikmwil_ is now known as erikmwilson | 15:43 | |
| *** erikmwilson_ has joined #openstack-infra | 15:43 | |
| ttx | mordred: that meeting I'm having would be RunAbove. | 15:43 |
| *** fhubik has quit IRC | 15:43 | |
| mordred | ttx: great. so - they work well with our tools, and I have already verified them :) | 15:43 |
| *** jcoufal has quit IRC | 15:43 | |
| fungi | they need ipv6 support | 15:43 |
| fungi | otherwise i'd have already moved my personal virtual machines there | 15:44 |
| *** peristeri has quit IRC | 15:44 | |
| mordred | ttx: you can tell them I added them to the latest release of os-client-config http://docs.openstack.org/developer/os-client-config/vendor-support.html#runabove | 15:44 |
| *** hdd has joined #openstack-infra | 15:45 | |
| fungi | oh wow, debian has gotten the all clear from their legal counsel to start including zfs support | 15:46 |
| *** annegentle has joined #openstack-infra | 15:46 | |
| jeblair | fungi: now i'm wondering if you are trolling us | 15:46 |
| *** dtantsur is now known as dtantsur|afk | 15:46 | |
| mordred | fungi: zfs as fuse? or as actual kernel modules? | 15:47 |
| fungi | also libdvdcss (now that's seriously historical... screw you dmca!) | 15:47 |
| fungi | mordred: not sure what the actual legal blockers were for zfs, assuming cddl/gpl incompatibility | 15:48 |
| fungi | though it's been so long i haven't followed it lately | 15:48 |
| sdague | jeblair: I was happily surprised I could Depends-On a stable/juno devstack patch in my grenade series | 15:48 |
| jeblair | sdague: yay! yeah, it's intentionally "depend on all changes with this change id regardless of project or branch" | 15:49 |
| sdague | yeh, I thought at one point we discussed it being branch restricted | 15:49 |
| sdague | but non branch restricted is totally what I need with grenade | 15:50 |
| sdague | so \o/ | 15:50 |
| fungi | i think we had determined that not restricting branches was desirable for exactly your use case | 15:50 |
| jeblair | sdague: i finally got back to reviewing topic:devstack-trove and agree it's ready to merge. i have ++d the patches that you updated | 15:51 |
| fungi | it just makes things complicated with our current backport policy reusing change-ids so there is some undefined behavior there | 15:51 |
| jeblair | so i think we're ready for someone from trove to kick that off with https://review.openstack.org/167047 | 15:51 |
| sdague | jeblair: cool | 15:52 |
| jeblair | fungi: well -- that _might_ be desired behavior too, depending on exactly what needs doing. :) | 15:52 |
| yolanda | mm , i may have dropped the name attribute accidentally, tests are failing | 15:52 |
| yolanda | going to check | 15:52 |
| sdague | jeblair: yeh, getting trove ptl to hit approve on that would be nice | 15:53 |
| *** bknudson has quit IRC | 15:53 | |
| sdague | now I just need someone to rub cheetah blood on rax and hp clouds so I can get test results back on this grenade stack faster :) | 15:54 |
| jeblair | SlickNik: the patches in https://review.openstack.org/#/q/status:open+topic:devstack-trove,n,z are ready to merge whenever you are | 15:54 |
| *** tonytan4ever has quit IRC | 15:54 | |
| *** mpavone has quit IRC | 15:56 | |
| *** russellb has quit IRC | 15:57 | |
| *** sabeen2 has joined #openstack-infra | 15:57 | |
| *** bknudson has joined #openstack-infra | 15:58 | |
| *** luqas has quit IRC | 15:59 | |
| *** garyk has joined #openstack-infra | 15:59 | |
| *** sabeen1 has quit IRC | 15:59 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/system-config: Symlink virtualenv in /usr/bin https://review.openstack.org/174441 | 16:00 |
| *** jlanoux has quit IRC | 16:00 | |
| *** mbacchi1 has joined #openstack-infra | 16:01 | |
| *** russellb has joined #openstack-infra | 16:01 | |
| *** thingee has joined #openstack-infra | 16:02 | |
| fungi | reviewing http://cacti.openstack.org/cacti/graph.php?local_graph_id=113&rra_id=all it looks like there must be some scheduled interaction happening to etherpad.o.o daily at 15:00 utc | 16:02 |
| * anteaya looks for cheetah blood | 16:02 | |
| *** amitgandhinz has quit IRC | 16:02 | |
| *** Sukhdev has joined #openstack-infra | 16:02 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 16:02 |
| yolanda | ok, resending nodepool, silly bug | 16:02 |
| fungi | whatever's triggering something at 15:00 utc there, it's not being run from cron | 16:04 |
| fungi | maybe someone has a document they're refreshing daily and it's large or fires up abiword to do the import | 16:05 |
| *** emagana has quit IRC | 16:05 | |
| fungi | or maybe someone has a pad they're exporting daily | 16:05 |
| *** gyee_ has joined #openstack-infra | 16:06 | |
| *** markvoelker has joined #openstack-infra | 16:08 | |
| *** jistr has quit IRC | 16:08 | |
| *** ociuhandu has quit IRC | 16:08 | |
| *** afazekas has quit IRC | 16:11 | |
| *** emagana has joined #openstack-infra | 16:13 | |
| *** scheuran has quit IRC | 16:13 | |
| *** markvoelker has quit IRC | 16:13 | |
| *** hdd has quit IRC | 16:14 | |
| mordred | fungi: image uploads to runabove are slowish, btw - but we only do that once a day | 16:14 |
| jeblair | fungi: not backup related? | 16:14 |
| *** ociuhandu has joined #openstack-infra | 16:14 | |
| *** marun_ is now known as marun | 16:14 | |
| jeblair | mordred: yeah, image _everything_ is slowish :) | 16:14 |
| *** Daisy has quit IRC | 16:15 | |
| mordred | jeblair: well, image uploads to dreamhost and vexxhost are actually pretty snappy | 16:15 |
| anteaya | jeblair: that is a pretty purple | 16:15 |
| openstackgerrit | Jeremy Stanley proposed openstack/requirements: Revert "Disable syncs to libs to unjam master requirements changes" https://review.openstack.org/174447 | 16:15 |
| fungi | dhellmann: ttx: i've approved https://review.openstack.org/174383 to temporarily remove libs from the requirements projects.txt file. also just uploaded a wip revert for it as https://review.openstack.org/174447 | 16:15 |
| *** tiswanso_ has quit IRC | 16:15 | |
| mordred | jeblair: like, when I run a test run of making an image and uploading it everywhere, runabove is noticably slower - but I do not think that actually matters, given how that pattern works | 16:15 |
| *** annegentle has quit IRC | 16:15 | |
| ttx | fungi: what's the idea here, remove them temporarily and add them again so that they don't block on themselves ? | 16:16 |
| ttx | i.e. add them all in one step ? | 16:16 |
| fungi | jeblair: not backup related afaict. mysqldump fires at 00:00 utc and bup at 05:37 utc | 16:17 |
| ttx | I think I got it | 16:17 |
| * ttx trying to keep up | 16:17 | |
| fungi | ttx: yeah, that was dhellmann's plan as far as i understood it | 16:17 |
| *** Guest9920 has quit IRC | 16:17 | |
| *** sarob has quit IRC | 16:18 | |
| *** asselin has joined #openstack-infra | 16:18 | |
| *** sushilkm has joined #openstack-infra | 16:18 | |
| *** sushilkm has left #openstack-infra | 16:18 | |
| fungi | jeblair: oh, nevermind. it looks like the daily spikes are at bup time. that spike today at 15:00 utc is out of cadence with the others | 16:19 |
| *** tiswanso_ has joined #openstack-infra | 16:19 | |
| *** baoli has quit IRC | 16:20 | |
| *** baoli has joined #openstack-infra | 16:24 | |
| *** baoli has quit IRC | 16:24 | |
| *** tqtran has joined #openstack-infra | 16:24 | |
| *** baoli has joined #openstack-infra | 16:27 | |
| zaro | morning | 16:27 |
| zaro | clarkb: i saw the gerrit release. we forked off stable-2.10 so we are ok because that includes everything that is in 1.10.3 | 16:27 |
| zaro | *1.10.3/2.10.3 | 16:28 |
| *** krtaylor has quit IRC | 16:28 | |
| anteaya | morning zaro | 16:30 |
| *** mriedem has quit IRC | 16:30 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/system-config: Add redirects from ci.o.o to docs.o.o/infra https://review.openstack.org/174437 | 16:31 |
| *** annegentle has joined #openstack-infra | 16:32 | |
| *** mriedem has joined #openstack-infra | 16:32 | |
| clarkb | fungi could the etherpad thing be daily db backups? | 16:36 |
| fungi | clarkb: yeah, see my later confirmation | 16:36 |
| fungi | clarkb: the spike today was unrelated to the others | 16:36 |
| clarkb | gotcha | 16:37 |
| *** markus_z has quit IRC | 16:38 | |
| *** derekh has quit IRC | 16:39 | |
| *** EmilienM is now known as EmilienM|afk | 16:39 | |
| yolanda | jeblair, change passed now, it was a silly typo | 16:40 |
| jeblair | yolanda: yay tests! :) | 16:40 |
| yolanda | glad they are there | 16:40 |
| clarkb | mt rainier is looking good today | 16:41 |
| anteaya | clarkb: glad you have nice weather | 16:42 |
| nibalizer | clarkb: you can see rainier from your house? | 16:43 |
| mordred | nibalizer: it's a big mountain | 16:43 |
| *** gyee_ has quit IRC | 16:43 | |
| * nibalizer impressed | 16:43 | |
| *** belmoreira has quit IRC | 16:43 | |
| *** claudiub has quit IRC | 16:43 | |
| *** patrickeast has joined #openstack-infra | 16:44 | |
| clarkb | no I am near olympia | 16:44 |
| *** davideagnello has joined #openstack-infra | 16:44 | |
| *** andreykurilin__ has joined #openstack-infra | 16:45 | |
| *** pelix has quit IRC | 16:45 | |
| *** pal_ has joined #openstack-infra | 16:46 | |
| *** afazekas has joined #openstack-infra | 16:47 | |
| *** pal has quit IRC | 16:47 | |
| sdague | ttx: I believe all libraries that released with caps, including client libraries, need to release a non cap version now as well, otherwise, brokeness | 16:48 |
| *** ivar-lazzaro has joined #openstack-infra | 16:49 | |
| *** dvorak is now known as clayton | 16:49 | |
| *** amotoki_ has quit IRC | 16:50 | |
| *** sks has joined #openstack-infra | 16:50 | |
| *** sushilkm has joined #openstack-infra | 16:50 | |
| *** sushilkm has left #openstack-infra | 16:50 | |
| *** pkarikh has quit IRC | 16:52 | |
| *** krtaylor has joined #openstack-infra | 16:53 | |
| mjordan | hi there - I'm seeing a lot of strange disconnects between Zuul (2.0.0) and Gerrit (2.9.1). When it occurs, Gerrit reports a timeout and a disconnect (http://pastebin.com/4iLyEFPZ), while Zuul apparently fails to catch the fact that it is no longer connected (http://pastebin.com/1FMjg1Fj). | 16:53 |
| *** russellb has quit IRC | 16:55 | |
| *** marcusvrn1 has quit IRC | 16:55 | |
| mjordan | thread dump appears to show Zuul waiting still on poll - http://pastebin.com/YztCNp7b | 16:56 |
| *** emagana has quit IRC | 16:56 | |
| *** bswartz has quit IRC | 16:58 | |
| *** marcusvrn has joined #openstack-infra | 16:58 | |
| *** harlowja_away is now known as harlowja | 16:58 | |
| anteaya | mjordan: we had to restart gerrit yesterday due to a thread issu | 16:58 |
| anteaya | e | 16:58 |
| *** andreykurilin__ has quit IRC | 16:58 | |
| anteaya | I am currently able to hear stream-events so it doesn't appear to be a reoccurance of what we saw yesterday | 16:59 |
| mjordan | anteaya: I've seen that post floating around as well, but the few instances of it that I found had errors that didn't appear to match up with anything in my Gerrit log | 16:59 |
| *** luqas has joined #openstack-infra | 16:59 | |
| *** mtanino has quit IRC | 16:59 | |
| mjordan | anteaya: interesting. In my case, I can barely keep a connection up from Zuul to Gerrit for more than an hour | 16:59 |
| mjordan | and these are two VMs sitting in the same rack :-) | 16:59 |
| *** russellb has joined #openstack-infra | 17:00 | |
| clarkb | zaro ^ wasnt that version of gerrit buggy? | 17:00 |
| anteaya | mjordan: has that been your experience the entire time you have run zuul? | 17:00 |
| clarkb | I want to say the mina sshd did weird stuff like that | 17:00 |
| mjordan | anteaya: yes | 17:00 |
| *** devvesa has quit IRC | 17:00 | |
| anteaya | mjordan: have you tried a different version of gerrit ever? | 17:01 |
| zaro | mjordan: 2.10.2 is no good | 17:01 |
| annegentle | hey reed | 17:01 |
| zaro | mjordan: it has issues with ssh | 17:01 |
| reed | hello annegentle | 17:01 |
| *** ivar-laz_ has joined #openstack-infra | 17:01 | |
| mjordan | anteaya: not yet - I took a look at the release notes and nothing jumped out at me (but that doesn't probably mean much). Our project just upgraded to Gerrit, so we're going through the growing pains and I haven't wanted to pull the rug out from under them with an upgrade | 17:02 |
| mjordan | zaro: good to know :-) | 17:02 |
| zaro | mjordan: https://groups.google.com/forum/#!topic/repo-discuss/u1nO6VCjWAg | 17:02 |
| *** rkukura has joined #openstack-infra | 17:02 | |
| *** Sukhdev has quit IRC | 17:02 | |
| *** unicell has joined #openstack-infra | 17:03 | |
| *** MaxV has quit IRC | 17:03 | |
| mjordan | zaro: thanks for the link; not sure if it is the same issue but sounds suspiciously similar. | 17:04 |
| zaro | mjordan: ohh yeah 2.9.1 has the same ssh issue i believe. 2.9.4 should be ok | 17:04 |
| *** luqas has quit IRC | 17:04 | |
| mjordan | zaro: which version are you running? :-) | 17:04 |
| zaro | mjordan: we running 2.8.4 | 17:04 |
| *** otter768 has joined #openstack-infra | 17:04 | |
| mjordan | ah. I'd jump back to that, but a number of plugins looked to be 2.9+ | 17:04 |
| zaro | mjordan: i've tested 2.9.4 and that looks good if you want to upgrade to that | 17:05 |
| mjordan | zaro: great, thanks - I'll give that a shot in a day or two | 17:05 |
| mjordan | appreciate the help | 17:05 |
| zaro | np | 17:05 |
| *** ivar-lazzaro has quit IRC | 17:05 | |
| *** emagana has joined #openstack-infra | 17:06 | |
| *** dannywilson has joined #openstack-infra | 17:06 | |
| zaro | mjordan: i believe this is the issue you are seeing: https://groups.google.com/d/msg/repo-discuss/4va1DH520to/pDGmY7pNUG4J | 17:07 |
| clarkb | can I bribe someone to update the tcpdump interface to 'any' in https://review.openstack.org/#/c/174139/5/devstack-vm-gate.sh ? then check experimental that change? I want to see the dhcp traffic | 17:08 |
| *** dannywilson has quit IRC | 17:08 | |
| *** BobBall is now known as BobBall_AWOL | 17:08 | |
| *** otter768 has quit IRC | 17:09 | |
| *** dannywilson has joined #openstack-infra | 17:09 | |
| *** annegentle has quit IRC | 17:09 | |
| *** dannywilson has quit IRC | 17:09 | |
| *** dannywilson has joined #openstack-infra | 17:10 | |
| *** dannywilson has quit IRC | 17:10 | |
| anteaya | mjordan: we are going to upgrade to 2.10 in about 3 weeks | 17:10 |
| *** dannywilson has joined #openstack-infra | 17:10 | |
| *** dannywilson has quit IRC | 17:10 | |
| anteaya | clarkb: I can update the patch if you tell me where the 'any' goes: sudo nohup tcpdump -i br_flat -e -n -w $BASE/logs/dhcpdump.pcap port 67 or port 68 & | 17:11 |
| *** tkelsey has joined #openstack-infra | 17:11 | |
| openstackgerrit | Joshua Harlow proposed openstack/requirements: Bump pymemcache to >= 1.2.9 https://review.openstack.org/174472 | 17:12 |
| *** markvoelker has joined #openstack-infra | 17:12 | |
| clarkb | anteaya replace br_flat with any | 17:13 |
| clarkb | anteaya tge -i is tge interface flag | 17:13 |
| anteaya | clarkb: thanks I will do | 17:14 |
| clarkb | thank you | 17:14 |
| *** annegentle has joined #openstack-infra | 17:14 | |
| anteaya | welcome | 17:16 |
| *** radez is now known as radez_g0n3 | 17:16 | |
| openstackgerrit | Merged openstack-infra/project-config: Clean up unused stable-compat jobs configuration https://review.openstack.org/174403 | 17:17 |
| openstackgerrit | Anita Kuno proposed openstack-infra/devstack-gate: Capture dhcp during multinode n-net runs https://review.openstack.org/174139 | 17:17 |
| *** peristeri has joined #openstack-infra | 17:17 | |
| *** dboik_ has quit IRC | 17:18 | |
| *** achanda has joined #openstack-infra | 17:18 | |
| mjordan | anteaya: ah, I see 2.10.3 is released | 17:18 |
| *** dboik has joined #openstack-infra | 17:19 | |
| anteaya | clarkb: it is in the experimental queue | 17:19 |
| clarkb | anteaya awesome thanks again | 17:20 |
| anteaya | clarkb: pleasure | 17:20 |
| anteaya | mjordan: yes, zaro is our gerrit guru, he tests all the things | 17:20 |
| *** andrey-mp has joined #openstack-infra | 17:23 | |
| *** pal_ has quit IRC | 17:23 | |
| andrey-mp | someone please review this - https://review.openstack.org/#/c/173945/ | 17:23 |
| zaro | warning: don't trust *all the things* from zaro | 17:23 |
| fungi | i've got the infra docs site index generator jobs almost finished writing, but need to head to an appointment. back in a while | 17:25 |
| *** tonytan4ever has joined #openstack-infra | 17:25 | |
| *** pal has joined #openstack-infra | 17:26 | |
| *** ildikov has quit IRC | 17:26 | |
| anteaya | fungi: happy appointment | 17:27 |
| *** jlibosva has quit IRC | 17:28 | |
| *** peristeri has quit IRC | 17:29 | |
| anteaya | but do take zaro's thoughts into account | 17:29 |
| *** yfried__ has joined #openstack-infra | 17:30 | |
| anteaya | I have a sudden urge to take a walk but I think my timing could be better, I'll wait until fungi re-appears | 17:30 |
| *** pal has quit IRC | 17:32 | |
| *** ssam2 has quit IRC | 17:32 | |
| openstackgerrit | Merged openstack-infra/project-config: Experimental Bandit job for Barbican https://review.openstack.org/173166 | 17:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade to get the client objects https://review.openstack.org/168603 | 17:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Add tests for shade cloud config parameter https://review.openstack.org/169339 | 17:35 |
| *** dizquierdo has joined #openstack-infra | 17:36 | |
| *** dboik_ has joined #openstack-infra | 17:37 | |
| *** tsg_ has quit IRC | 17:38 | |
| *** tiswanso_ has quit IRC | 17:38 | |
| openstackgerrit | Merged openstack-infra/shade: Basic test for meta method obj_list_to_dict https://review.openstack.org/173871 | 17:39 |
| *** tiswanso_ has joined #openstack-infra | 17:40 | |
| *** MaxV has joined #openstack-infra | 17:40 | |
| *** ihrachyshka has quit IRC | 17:40 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for image uploads https://review.openstack.org/168633 | 17:41 |
| mordred | greghaynes: I just rebased the first shade patches on top of your use-fakes patch - I'm not sure I fully did it right | 17:41 |
| *** dboik has quit IRC | 17:41 | |
| dhellmann | ttx, fungi: sdague pointed out that we can't unblock requirements in master until we have versions of the libs that don't include caps so the new libs can be installed together. Now that I'm finally back online I'm adding some details to https://etherpad.openstack.org/p/the-big-thaw | 17:41 |
| greghaynes | mordred: oh, that should have merged already | 17:41 |
| mordred | greghaynes: yes. your thing merged | 17:42 |
| greghaynes | ah, ok | 17:42 |
| mordred | greghaynes: which means my thing was in merge conflict | 17:42 |
| *** markvoelker has quit IRC | 17:43 | |
| *** EmilienM|afk is now known as EmilienM | 17:43 | |
| greghaynes | I should read what youall did for fakes | 17:44 |
| greghaynes | ETOOMANYTHINGS | 17:44 |
| greghaynes | (youall being shade) | 17:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Pass TaskManager in to shade https://review.openstack.org/168663 | 17:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade API for flavors https://review.openstack.org/168634 | 17:44 |
| mordred | greghaynes: :) | 17:44 |
| *** ociuhandu has quit IRC | 17:45 | |
| *** peristeri has joined #openstack-infra | 17:45 | |
| *** yamahata has joined #openstack-infra | 17:45 | |
| *** btran has joined #openstack-infra | 17:46 | |
| *** MaxV has quit IRC | 17:46 | |
| *** amitgandhinz has joined #openstack-infra | 17:47 | |
| *** alexpilotti has quit IRC | 17:48 | |
| *** tiswanso_ has quit IRC | 17:49 | |
| *** mrmartin has quit IRC | 17:49 | |
| *** HeOS has quit IRC | 17:49 | |
| *** vipul has quit IRC | 17:50 | |
| *** andrey-mp has quit IRC | 17:50 | |
| mordred | nibalizer: https://review.openstack.org/#/c/169075/ are the docs written to cover https://review.openstack.org/#/c/167625/ | 17:54 |
| *** hrubi has quit IRC | 17:55 | |
| *** EmilienM is now known as EmilienM|afk | 17:55 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-nodepool: Install additional depends for dib https://review.openstack.org/167625 | 17:55 |
| openstackgerrit | Merged openstack/requirements: Disable syncs to libs to unjam master requirements changes https://review.openstack.org/174383 | 17:56 |
| *** hrubi has joined #openstack-infra | 17:56 | |
| *** spzala has joined #openstack-infra | 17:57 | |
| *** dizquierdo has quit IRC | 17:58 | |
| *** sdake_ has joined #openstack-infra | 17:59 | |
| *** afazekas has quit IRC | 17:59 | |
| *** vipul has joined #openstack-infra | 17:59 | |
| nibalizer | mordred: cool | 17:59 |
| *** tiswanso has joined #openstack-infra | 18:01 | |
| *** jaypipes has quit IRC | 18:01 | |
| mordred | jeblair: gerrit seems maybe down | 18:01 |
| *** jaypipes has joined #openstack-infra | 18:01 | |
| mordred | oh - nevermind | 18:01 |
| mordred | jeblair: pebkac | 18:02 |
| *** sdake has quit IRC | 18:02 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-diskimage_builder: Fix .gitreview file https://review.openstack.org/174505 | 18:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-diskimage_builder: Consume vhd-util from infra vhd-util PPA https://review.openstack.org/174506 | 18:03 |
| *** ivar-laz_ has quit IRC | 18:03 | |
| david-lyle | where is the channel list for the statusbot maintained? | 18:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/puppet-nodepool: Consume puppet-diskimage_builder https://review.openstack.org/167610 | 18:03 |
| mordred | david-lyle: project-config repo | 18:03 |
| *** ivar-lazzaro has joined #openstack-infra | 18:03 | |
| mordred | david-lyle: openstack-infra/project-config | 18:03 |
| david-lyle | I looked in there, didn't see the config, will look again | 18:04 |
| mordred | david-lyle: one sec - I will get you link | 18:04 |
| nibalizer | http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests/eavesdrop.pp#n39 i think | 18:04 |
| mordred | yah. nibalizer is right | 18:04 |
| nibalizer | there are a couple things we dont have in project-config that are candidates for getting out | 18:05 |
| nibalizer | i think there is a list of channels in project-config too... so im not 100% sure which list is for what | 18:05 |
| *** mrmartin has joined #openstack-infra | 18:07 | |
| openstackgerrit | Khai Do proposed openstack-infra/jenkins-job-builder: Log info about updated and deleted jobs https://review.openstack.org/162293 | 18:08 |
| david-lyle | ok I looked there too, it references $statusbot_channels, which I now find in manifests/site.pp | 18:09 |
| david-lyle | thanks nibalizer, mordred | 18:09 |
| *** davideagnello has quit IRC | 18:09 | |
| *** davideagnello has joined #openstack-infra | 18:09 | |
| *** markmcclain has quit IRC | 18:09 | |
| *** johnthetubaguy is now known as zz_johnthetubagu | 18:11 | |
| openstackgerrit | David Lyle proposed openstack-infra/system-config: Adding openstack-horizon to statusbot channels https://review.openstack.org/174526 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Pass TaskManager in to shade https://review.openstack.org/168663 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use correct diskimage when building diskimages https://review.openstack.org/169474 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Support per provider image build settings https://review.openstack.org/126747 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for image uploads https://review.openstack.org/168633 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade to get the client objects https://review.openstack.org/168603 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Just set qcow2 version https://review.openstack.org/169383 | 18:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade API for flavors https://review.openstack.org/168634 | 18:12 |
| mordred | jhesketh: I addressed your comment in https://review.openstack.org/169383 above | 18:12 |
| *** sdake has joined #openstack-infra | 18:13 | |
| cody-somerville | Is infra nodepool seeing any problems with hpcloud at the moment? | 18:15 |
| *** _nadya_ has quit IRC | 18:16 | |
| *** andreykurilin__ has joined #openstack-infra | 18:17 | |
| anteaya | yes, I don't know what they are since I don't have server access | 18:17 |
| anteaya | but my graph shows some unhappiness | 18:17 |
| *** sdake_ has quit IRC | 18:17 | |
| anteaya | cody-somerville: fungi is at an appointment and should be back later today (I don't know when), clarkb is driving and I don't know what task jim is doing | 18:18 |
| anteaya | mordred: are you able to help cody-somerville get some details about the hpcloud errors we are experiencing? | 18:18 |
| *** baoli has quit IRC | 18:18 | |
| mordred | cody-somerville, anteaya: it certainly doesn't seem happy | 18:18 |
| openstackgerrit | Khai Do proposed openstack-infra/system-config: Allow puppet to manage replication keys on review-dev.o.o https://review.openstack.org/174133 | 18:18 |
| *** sushilkm has joined #openstack-infra | 18:18 | |
| dhellmann | fungi, ttx, sdague : here's the series of patches to uncap libs: https://review.openstack.org/#/q/topic:uncap-requirements,n,z | 18:19 |
| anteaya | mordred: agreed | 18:19 |
| *** sushilkm has left #openstack-infra | 18:19 | |
| *** achanda has quit IRC | 18:20 | |
| *** tkelsey has quit IRC | 18:20 | |
| openstackgerrit | Merged openstack-infra/bindep: Add a --file option https://review.openstack.org/164835 | 18:21 |
| openstackgerrit | Merged openstack-infra/bindep: Add a --brief option https://review.openstack.org/164854 | 18:21 |
| jeblair | ReadTimeout: HTTPSConnectionPool(host='region-b.geo-1.compute.hpcloudsvc.com', port=443): Read timed out. (read timeout=60.0) | 18:22 |
| anteaya | ah here's jeblair | 18:22 |
| jeblair | cody-somerville, mordred: ceate server tasks are taking 40+ seconds for us. we time out after 60s | 18:22 |
| cody-somerville | We're seeing that as well. | 18:23 |
| openstackgerrit | Andreas Jaeger proposed openstack/requirements: Mark documentation repositories as soft https://review.openstack.org/173201 | 18:23 |
| *** wwwBUKOLAYcom has joined #openstack-infra | 18:23 | |
| *** teran has quit IRC | 18:23 | |
| jeblair | (so even when one eventually succeeds, we lose track of it and leak it) | 18:23 |
| *** teran has joined #openstack-infra | 18:23 | |
| *** radez_g0n3 is now known as radez | 18:24 | |
| *** Sukhdev has joined #openstack-infra | 18:24 | |
| openstackgerrit | Michael Krotscheck proposed openstack/requirements: Updated oslo.config to 1.10.0 https://review.openstack.org/173449 | 18:24 |
| *** mhu has quit IRC | 18:26 | |
| *** teran has quit IRC | 18:28 | |
| *** achanda has joined #openstack-infra | 18:28 | |
| *** mhu has joined #openstack-infra | 18:30 | |
| jesusaurus | jeblair: yeah, we're seeing nodes turn up as aliens as well | 18:30 |
| *** bswartz has joined #openstack-infra | 18:30 | |
| dhellmann | ttx, fungi, sdague : hmm, I guess being in the release team doesn't automatically grant +2, so I can't force some of those requirements changes through for libs no managed by oslo | 18:30 |
| mordred | jeblair: I don't seem to be getting counters in graphite for rax after midnight ish - only hp remain | 18:31 |
| jeblair | mordred: which metric? | 18:32 |
| mordred | jeblair: CreateServers | 18:32 |
| mordred | (actually, any of the task metrics | 18:32 |
| jeblair | mordred: because they're now under _main and _second | 18:32 |
| *** ArxCruz has joined #openstack-infra | 18:33 | |
| mordred | ah - I missed that | 18:33 |
| jeblair | mordred: i think we should hide that, so we may want to ask yolanda to update her change or make a followup | 18:33 |
| mordred | o - and I'm getting lines for hp because of the * for matching azs | 18:33 |
| *** EmilienM|afk is now known as EmilienM | 18:33 | |
| jeblair | also, while we're at it, i think _main and _slow might be better names | 18:33 |
| mordred | yah | 18:34 |
| yolanda | ah, stats changed now with the taskworker names... | 18:34 |
| yolanda | i didn't consider that | 18:34 |
| *** dprince has quit IRC | 18:35 | |
| jeblair | yolanda: you want to fix that in your patch? (also, i think changing _second to _slow might be more descriptive if you have an extra minute) | 18:35 |
| yolanda | sure | 18:35 |
| yolanda | _horribly_slow :) | 18:36 |
| jeblair | heh | 18:36 |
| dhellmann | fungi, jeblair : doesn't adding depends-on to a commit message cause that patch to be checked out on the node where the tests are being run? that doesn't seem to be working for removing the requirements caps | 18:37 |
| *** dboik_ has quit IRC | 18:37 | |
| *** ayoung has joined #openstack-infra | 18:37 | |
| *** dboik has joined #openstack-infra | 18:39 | |
| jeblair | dhellmann: yes. link? | 18:39 |
| dhellmann | jeblair: https://jenkins07.openstack.org/job/gate-oslo.concurrency-requirements/34/console is the log | 18:40 |
| dhellmann | the patch is https://review.openstack.org/#/c/174486/ | 18:40 |
| dhellmann | the depends-on patch just merged, so I may be able to recheck my other patches | 18:41 |
| *** yamahata has quit IRC | 18:42 | |
| *** markvoelker has joined #openstack-infra | 18:42 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/nodepool: Different queues for slow tasks. https://review.openstack.org/170788 | 18:43 |
| yolanda | ok, quick fix sent | 18:43 |
| yolanda | i prefered passing an stats_name param, rather than go manipulating strings and splitting names | 18:43 |
| *** sks has quit IRC | 18:44 | |
| *** ArxCruz has quit IRC | 18:44 | |
| jeblair | dhellmann: 2015-04-16 18:04:43.355 | requirements git sha: 517f2174b025a4b7859e63308c3b58f4b4130e79 | 18:44 |
| jeblair | it appears to have checked out the correct git sha | 18:45 |
| dhellmann | I wonder if that job doesn't look at the project list. :-/ | 18:45 |
| dhellmann | like maybe we need to turn off that check job separately | 18:45 |
| *** ArxCruz has joined #openstack-infra | 18:45 | |
| jeblair | dhellmann: ooooo.... i thought you were indicating that the values did not look correct. but instead, those values are indeed what's in openstack/requiremests. | 18:48 |
| dhellmann | oh, are they? I wonder if I picked the wrong patch | 18:48 |
| dhellmann | ugh | 18:48 |
| jeblair | dhellmann: you pointed to a patch that removes things from the project list, but does not change requirements | 18:49 |
| jeblair | https://review.openstack.org/#/c/174383/1/projects.txt | 18:49 |
| dhellmann | jeblair: bah | 18:49 |
| jeblair | dhellmann: we need "Depends-On: that patch i really meant" | 18:49 |
| dhellmann | jeblair: I need a script to fix these :-) | 18:50 |
| *** hasharAway is now known as hashar | 18:50 | |
| *** achanda has quit IRC | 18:52 | |
| anteaya | clarkb: zuul estimates 20 minutes left on the experimental tests | 18:53 |
| *** doude has quit IRC | 18:54 | |
| *** doude_ has joined #openstack-infra | 18:54 | |
| fungi | back. now to see what i've been missing | 18:57 |
| *** ArxCruz has quit IRC | 18:58 | |
| *** tsg has joined #openstack-infra | 18:58 | |
| fungi | david-lyle: don't listen to mordred and nibalizer. the statusbot channel list is https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n267 | 19:01 |
| fungi | clearly we still have irc bot channel lists scattered in way too many places | 19:01 |
| *** tonytan4ever has quit IRC | 19:01 | |
| *** tonytan4ever has joined #openstack-infra | 19:02 | |
| anteaya | fungi: should pleia2 be listed https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n268 as well? | 19:04 |
| fungi | anteaya: she certainly _can_ be. she should feel free to propose that patch ;) | 19:05 |
| *** otter768 has joined #openstack-infra | 19:05 | |
| fungi | dhellmann: just caught up. so you have now discovered that the zuul/layout.yaml in openstack-infra/project-config is running gate-{name}-requirements jobs on projects which you didn't want getting requirements updates... is there a patch to stop running those i should be reviewing? | 19:06 |
| fungi | e.g. we merged the patch which stops proposing requirements updates to those projects, but we did not merge a patch which stops gating those projects on whether or not their requirements match the global list | 19:07 |
| fungi | (er, i.e., not e.g.) | 19:07 |
| dhellmann | fungi: if you could put it together, that would be great. I just had a meetup mini-emergency arise | 19:07 |
| * dhellmann is having an unusually frantic day | 19:08 | |
| fungi | dhellmann: you've got it. will do | 19:08 |
| dhellmann | fungi: you're awesome, thanks | 19:10 |
| *** otter768 has quit IRC | 19:10 | |
| * dhellmann goes back to figuring out how to fit 100 people in a 45 person space | 19:10 | |
| fungi | dhellmann: petroleum jelly | 19:11 |
| *** achanda has joined #openstack-infra | 19:12 | |
| dhellmann | fungi: I'll stop by the drug store on the way over to the venue | 19:12 |
| dhellmann | fungi: the other thing I did wrong was depend on the wrong patch, so I'm not sure we need to turn the jobs off | 19:14 |
| dhellmann | we might just have to make those patches depend on the one that actually removes the requirements | 19:14 |
| *** markvoelker has quit IRC | 19:14 | |
| dhellmann | hrm, except we can't land that patch until there are new versions of all of the libs, can we? | 19:15 |
| *** hdd has joined #openstack-infra | 19:15 | |
| dhellmann | so yeah, I think we need the job turned off to land these fixes, and then turned back on after we have the releases out | 19:15 |
| fungi | yeah, i think the jobs which try to coinstall those libs (aside from devstack-based jobs) are going to choke because they want different versions of dependencies | 19:15 |
| *** peristeri has quit IRC | 19:16 | |
| dhellmann | fungi: I'm making notes on the steps we need to follow under "unblocking requirements" on https://etherpad.openstack.org/p/the-big-thaw | 19:18 |
| mordred | sdague, jogo: reading nova api docs ... novaclient doesnt' seem to implement "/v2/{tenant_id}/servers/{server_id}/ips" anywhere | 19:19 |
| *** radez is now known as radez_g0n3 | 19:19 | |
| *** isviridov is now known as isviridov_away | 19:20 | |
| *** mpavone has joined #openstack-infra | 19:21 | |
| ttx | ok, available now, catching up with backlog | 19:23 |
| jogo | mordred: I am not surprised | 19:23 |
| ttx | that was a hell of a meeting but I think I got you 600 extra test nodes | 19:23 |
| jogo | mordred: novaclient is a pretty terrible | 19:23 |
| mordred | ttx: WOOT! | 19:23 |
| ttx | so probably worth it | 19:24 |
| mordred | jogo: ok. so - it's not that /ips is not a real thing | 19:24 |
| jogo | whoa 600 more! | 19:24 |
| jogo | mordred: do the docs say it is? | 19:24 |
| mordred | ttx: I shall buy you something inappropriate | 19:24 |
| jogo | ttx: where did they come from? | 19:24 |
| mordred | jogo: the docs say it is | 19:24 |
| ttx | mordred: I'll drink when I see them, but I got the local Mark Shuttleworth go ahead | 19:24 |
| jogo | mordred: sigh the docs are also really bad :/ | 19:24 |
| jogo | mordred: wan to file a bug for that | 19:25 |
| mordred | ttx: sabdfl is involved? | 19:25 |
| ttx | jogo: RunAbove public cloud (OVH) | 19:25 |
| jogo | ttx: neat! | 19:25 |
| ttx | mordred: no. But OVH have their own sabdfl | 19:25 |
| mordred | ttx: ah. gotcha | 19:25 |
| mordred | ttx: well, I've alraedy added runabove support to os-client-config - so we're all set | 19:25 |
| annegentle | what's, uh, sabdfl (dare I ask?) | 19:25 |
| ttx | mordred: I'll send you a more detailed report when fire is off | 19:26 |
| ttx | annegentle: Mark Shuttleworth, self-appointed benevolent dictator for life | 19:26 |
| annegentle | ttx: whew on that f-word | 19:26 |
| annegentle | :) | 19:26 |
| ttx | ((sabdfl is his IRC nick) | 19:26 |
| * jogo gets ready to board his flight | 19:26 | |
| anteaya | ttx wooooo | 19:26 |
| ttx | dhellmann: sorry I couldn't be more available today | 19:26 |
| anteaya | jogo: safe travels | 19:26 |
| mordred | ttx: you earned being away from IRC - you got us a new cloud | 19:27 |
| dhellmann | dhellmann: np, I have spent most of today in the car anyway | 19:27 |
| fungi | annegentle: no comment on whether sabdfl might sort of be an f-word anyway | 19:27 |
| annegentle | fungi: hehe | 19:27 |
| *** HeOS has joined #openstack-infra | 19:27 | |
| fungi | dhellmann: talking to yourself--that's awesome | 19:27 |
| spredzy | If any core around could take few mins to review https://review.openstack.org/#/c/173255/ , missing a +2. Thank you. | 19:27 |
| dhellmann | ttx: I'm updating the big thaw etherpad with the steps to uncap requirements | 19:27 |
| dhellmann | fungi: ugh, man, I'm going back to be | 19:28 |
| dhellmann | bed | 19:28 |
| anteaya | the lost 100 have to fend for themselves? | 19:28 |
| ttx | dhellmann: I tried to expand under point 9 earlier | 19:28 |
| dhellmann | ttx: I'll move my stuff up there | 19:29 |
| ttx | I'm available for the ext hour(s) though, and just freed my tomorrow morning | 19:30 |
| *** spzala has quit IRC | 19:32 | |
| mordred | fungi: hi! | 19:32 |
| ttx | dhellmann: wow, that's more complex than I thought | 19:32 |
| mordred | jeblair: ping | 19:32 |
| dhellmann | ttx: ok, cool. I'm actually traveling tomorrow, and I have a meetup in a bit, so I only have a couple more hours to work on this. I'm trying to make sure everyone else understands the state of things before I disappear | 19:32 |
| dhellmann | ttx: yeah | 19:32 |
| fungi | mordred: ohai | 19:32 |
| mordred | fungi: HP is asking us to turn off nodepool | 19:33 |
| fungi | woo! okie-dokey | 19:33 |
| anteaya | clarkb: https://review.openstack.org/#/c/174139/ finished | 19:33 |
| fungi | mordred: sounds like we could use that runabove offer right about now ;) | 19:33 |
| ttx | dhellmann: what I need to know now is... (1) can we do RC2s (i.e. can we push stuff to stable/kilo) and (2) can we unfreeze master req | 19:34 |
| ttx | I think (2) needs to wait at this point if I read the status correctly | 19:34 |
| mordred | fungi: yah | 19:34 |
| dhellmann | ttx: I've been focusing on unfreezing master because I assumed 1 was done | 19:34 |
| dhellmann | that is, I assumed we were ok there | 19:34 |
| ttx | right, if (1) was done I can start opening RC2 windows tomorrow | 19:34 |
| ttx | or tonight | 19:35 |
| * ttx doublechecks that | 19:35 | |
| *** dustins has quit IRC | 19:35 | |
| ttx | dhellmann: IIRC some of your open-stable-kilo failed tests | 19:35 |
| dhellmann | ttx: we may need to recheck some of those kilo jobs, let me see if my patch to turn off some of the bad jobs is landed | 19:35 |
| dhellmann | ttx: https://review.openstack.org/#/c/174371/ has merged, let me recheck a couple of those patches | 19:36 |
| ttx | dhellmann: looks like all the ones I urgently care about (RC2 wanabees) got their open-stable-kilo change landed | 19:36 |
| *** annegentle has quit IRC | 19:36 | |
| ttx | there are a few libs we need to push, let me see if tose are cleared too | 19:37 |
| fungi | hrm, openstackgerrit has gone silent. i hope we're not blocked on stream-events again | 19:37 |
| *** teran has joined #openstack-infra | 19:38 | |
| anteaya | fungi: stream-events is listening but no response yet for me | 19:38 |
| dhellmann | ttx: I'm reverifying changes in python-saharaclient and python-heatclient to see if those can land now | 19:38 |
| anteaya | fungi: still nothing, it is a possiblity | 19:39 |
| ttx | I think all the point releases "from left field" got their open-stable-kilo change merged too | 19:39 |
| mordred | fungi: puppet is stopped on nodepool - any problem with me editing quotas live? | 19:39 |
| anteaya | mordred: stand by we might have to restart gerrit | 19:39 |
| *** dustins has joined #openstack-infra | 19:39 | |
| mordred | ok | 19:39 |
| ttx | dhellmann: i was wondering if we can clear them to release their stable/kilo point release | 19:39 |
| fungi | mordred: nope. i suspect it's been disabled since last night when we were restarting it and getting config updates into it | 19:40 |
| ttx | python-glanceclient 0.17.1 | 19:40 |
| ttx | keystoneclient 1.3.1 | 19:40 |
| ttx | keystonemiddleware 1.5.1 | 19:40 |
| ttx | python-ceilometerclient 1.0.14 | 19:40 |
| dhellmann | ttx: I'm not sure what that means? | 19:40 |
| anteaya | stream-events still has picked up no events for me | 19:40 |
| dhellmann | ttx: should we move to relmgr since they're having a gerrit issue? | 19:40 |
| ttx | dhellmann: sure | 19:40 |
| fungi | jeblair: it looks like your patch to show accounts associated with stream-events tasks in the show-queue output is not having any affect | 19:41 |
| fungi | however we do seem to have a bunch of stream-events tasks waiting | 19:41 |
| mordred | ok. I've set the hp cloud quotas to 0 in nodepool | 19:41 |
| mordred | fungi: anything I can do to help you chase gerrit? | 19:43 |
| fungi | yep, last event zuul saw on the stream was 19:14:32 utc, 30 minutes ago | 19:43 |
| fungi | i'm restarting gerrit now | 19:43 |
| mordred | fungi: I'm looking forward to that not being a thing anymore | 19:44 |
| fungi | jeblair: for the record, the entries still look like: | 19:44 |
| fungi | abc88ab1 waiting .... 19:14:32.406 com.google.gerrit.sshd.commands.StreamEvents$3@11a5f89 | 19:44 |
| jeblair | fungi: well, phooey. | 19:45 |
| jeblair | mordred: pong | 19:45 |
| *** andreykurilin__ has quit IRC | 19:45 | |
| fungi | #status notice gerrit has been restarted to clear a problem with its event stream. any gerrit changes updated or approved between 19:14 and 19:46 utc will need to be rechecked or have their approval reapplied for zuul to pick them up | 19:46 |
| openstackstatus | fungi: sending notice | 19:46 |
| anteaya | stream-events is picking up events again for me | 19:46 |
| -openstackstatus- NOTICE: gerrit has been restarted to clear a problem with its event stream. any gerrit changes updated or approved between 19:14 and 19:46 utc will need to be rechecked or have their approval reapplied for zuul to pick them up | 19:47 | |
| mordred | jeblair: was pinging about the hp/nodepool thing | 19:48 |
| *** julim has quit IRC | 19:48 | |
| openstackstatus | fungi: finished sending notice | 19:48 |
| jeblair | fungi: how important do you think getting that to work is? | 19:48 |
| anteaya | mordred: have you a patch to reflect that so it can be tracked like? | 19:48 |
| jeblair | fungi: i figured it was the next step to figure out if there's a pattern to what's causing the stuck connections | 19:48 |
| jeblair | fungi: any other ideas? | 19:48 |
| jeblair | mordred: did you make a change to turn quotas to 0? | 19:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Turn off HP in nodepool https://review.openstack.org/174578 | 19:49 |
| fungi | jeblair: maybe a packet sniffer, though i don't think we'd have a lot of luck telling those connections apart from others on that same port | 19:49 |
| mordred | jeblair: I did | 19:49 |
| mordred | jeblair: I did it live on the box since puppet is off - the corresponding patch is there ^^ | 19:49 |
| lifeless | Daviey: ping | 19:49 |
| anteaya | mordred: should puppet be on again at some point? | 19:50 |
| jeblair | fungi: yeah, a stuck connection will be hard to sniff | 19:51 |
| mordred | anteaya: yes - but we have it off right now testing a patch from yolanda | 19:51 |
| anteaya | mordred: ah | 19:51 |
| jeblair | mordred: we don't need it off anymore | 19:51 |
| mordred | jeblair: oh, great | 19:51 |
| jeblair | mordred: i was about to turn it back on | 19:51 |
| jeblair | but now i'll wait for that to land | 19:51 |
| fungi | jeblair: though if it's a bunch of inbound connections from one source triggering it, then maybe we can correlate. i can refresh my memory on how to make tcpdump rotate a capture so we don't run out of space, and then leave it going until the next incident | 19:51 |
| sdague | mordred: looking at the nova api samples, I'm not sure ips is a thing | 19:51 |
| *** yamahata has joined #openstack-infra | 19:52 | |
| mordred | sdague: excellent | 19:52 |
| mordred | sdague: that's good to know | 19:52 |
| sdague | mordred: which cloud is returning it? | 19:52 |
| fungi | jeblair: long-shot though | 19:52 |
| mordred | sdague: nobody | 19:52 |
| jeblair | fungi: i'm not sure that will help | 19:52 |
| sdague | mordred: ok | 19:52 |
| mordred | sdague: it's here: http://developer.openstack.org/api-ref-compute-v2.html | 19:53 |
| jeblair | fungi: i don't know how we would associate a connection with a stuck thread | 19:53 |
| mordred | http://developer.openstack.org/api-ref-compute-v2.html#compute_server-addresses | 19:53 |
| mordred | sdague: ^^ specifically | 19:53 |
| fungi | jeblair: if it's a bunch of new connections at the moment everything hangs, then the source on the syn(without ack) packets | 19:53 |
| fungi | jeblair: loose correlation | 19:53 |
| sdague | mordred: yeh, let me look at code | 19:53 |
| mordred | sdague: (I was brainstorming ways the nodepool api sequence could be nicer to the nova db) | 19:53 |
| jeblair | fungi: i don't have a reason to think that's happening though. these are long-lived connections, which get passed around to threads, and occasionally a thread gets stuck when writing to it. | 19:54 |
| anteaya | does the show connections command give anything useful? | 19:54 |
| fungi | anteaya: lots of connections from lots of places | 19:54 |
| anteaya | :( | 19:54 |
| jeblair | fungi: so i think we'd have to look for a connection that we thought should be there but wasn't having data transmitted | 19:54 |
| sdague | mordred: oh, now it's there... it's just not in the examples.... hm | 19:55 |
| jeblair | fungi: i'll try to find a way to test that patch | 19:55 |
| fungi | jeblair: okay. i'm similarly unsure what we'll be able to tell from the accounts being handled by the waiting stream-events threads, but more data can't hurt | 19:56 |
| *** radez_g0n3 is now known as radez | 19:56 | |
| anteaya | show connections has an idle setting: https://gerrit-documentation.storage.googleapis.com/Documentation/2.10/cmd-show-connections.html | 19:56 |
| jeblair | fungi: i'm hopping it will tell us whether the stuck threads are related to specific accounts or not | 19:57 |
| *** dustins has quit IRC | 19:57 | |
| jeblair | fungi: or specific connections | 19:57 |
| *** dboik has quit IRC | 19:58 | |
| *** dboik has joined #openstack-infra | 19:58 | |
| *** tiswanso has quit IRC | 19:58 | |
| fungi | jeblair: yep, if they're predominantly for one account then that would be a clue | 20:00 |
| *** sabeen has joined #openstack-infra | 20:00 | |
| *** ildikov has joined #openstack-infra | 20:00 | |
| *** kgiusti has left #openstack-infra | 20:00 | |
| fungi | anteaya: it's possible we might be able to correlate the idle time on open connections to the waiting since timestamp for running tasks. i'll compare them next time for any additional possibilties | 20:00 |
| fungi | er, possibilities | 20:00 |
| *** sabeen2 has quit IRC | 20:00 | |
| *** erikmwil_ has joined #openstack-infra | 20:01 | |
| *** erikmwilson has quit IRC | 20:01 | |
| *** erikmwil_ is now known as erikmwilson | 20:01 | |
| jeblair | ++ | 20:01 |
| anteaya | fungi: thanks, might not be useful if the keep-alive is all that we are seeing but might as well grasp at that straw | 20:01 |
| *** annegentle has joined #openstack-infra | 20:01 | |
| *** tiswanso has joined #openstack-infra | 20:02 | |
| *** tiswanso has quit IRC | 20:02 | |
| *** baoli has joined #openstack-infra | 20:02 | |
| *** tiswanso has joined #openstack-infra | 20:03 | |
| *** rlandy has quit IRC | 20:03 | |
| fungi | interestingly, our jenkins account is only the second highest number of concurrent connections. i wonder why neutronryu has 14 connections open, even right after a gerrit restart | 20:03 |
| fungi | that's not insanely high, but it makes me curious about their ci architecture | 20:04 |
| *** rlandy has joined #openstack-infra | 20:04 | |
| *** baoli has quit IRC | 20:04 | |
| fungi | </random_observation> | 20:05 |
| *** baoli has joined #openstack-infra | 20:05 | |
| jeblair | fungi: or if they have 14 ci architectures? | 20:05 |
| mordred | fungi, jeblair: I suggest we leave HP turned off until they've had a chance to run db things to fix the underlying problem | 20:05 |
| anteaya | mestery: correct me if I'm wrong but didn't neutron ryu stop testing a driver? | 20:05 |
| anteaya | mestery: is neutron ryu still a going concern? I can't find a wikipage for them | 20:05 |
| jeblair | mordred: wfm; i'd rather not have the system flapping. have an idea when that might be? | 20:05 |
| jeblair | corvus@fuligin:~/git/openstack-infra/buck$ ./bin/buck --help | 20:05 |
| jeblair | Buck does not appear to have been built -- building Buck! | 20:05 |
| jeblair | also ^ _that_ is insane. | 20:06 |
| fungi | mordred: jeblair: agreed, we were getting nearly no usable nodes from hpcloud anyway | 20:06 |
| fungi | mordred: jeblair: and the serialized alien mop-up i started there hours ago is still churning | 20:06 |
| mordred | jeblair: unsure. I'll try to get an update | 20:06 |
| fungi | jeblair: buck has an executable that's a wrapper which opportunistically builds itself before calling into the real executable? that's pretty crazytown | 20:07 |
| jeblair | fungi: yup | 20:08 |
| anteaya | mestery: it appears they are still testing patches but they don't have an entry on https://wiki.openstack.org/wiki/ThirdPartySystems | 20:08 |
| anteaya | mestery: any idea who is in charge of that system? | 20:08 |
| jeblair | fungi: the continuation of that was that it sat there for a few minutes building, then dutifully output the help i requested | 20:08 |
| fungi | ha | 20:08 |
| fungi | jeblair: did the help include redundant information on how to build buck, in case you hadn't already? | 20:09 |
| mordred | fungi, jeblair: I have been told to check back on Monday for a firm resolution date - contact to check with is Michael Thomas | 20:09 |
| EmilienM | is jeepyb able to run a script/hook when the patch is merged in the repository ? | 20:10 |
| *** annegentle has quit IRC | 20:10 | |
| fungi | perhaps "michael thomas" is the name they've given the wastebasket where they'll be depositing your follow-up correspondence | 20:10 |
| *** annegentle has joined #openstack-infra | 20:11 | |
| *** dboik has quit IRC | 20:11 | |
| anteaya | I've asked in neutron to see if anyone knows anything about ryu ci, so far no reply | 20:11 |
| *** markvoelker has joined #openstack-infra | 20:12 | |
| mordred | fungi: it's a fair strategy | 20:12 |
| fungi | EmilienM: jeepyb is just a collection of scripts and hooks. gerrit definitely has a hook it calls when changes merge, so a hook script could be written to take some desirable action when that happens | 20:12 |
| anteaya | now at our discretion we can disable them for not having a wikipage, which they are required to have, and then we can find out who they are | 20:12 |
| mordred | fungi, jeblair: I've got to run ... | 20:12 |
| anteaya | if anyone wants that option | 20:12 |
| *** dboik has joined #openstack-infra | 20:12 | |
| fungi | mordred: don't forget to bring a towel | 20:13 |
| *** tonytan4ever has quit IRC | 20:13 | |
| *** claudiub has joined #openstack-infra | 20:14 | |
| jeblair | fungi: um. gerrit needs a forked version of buck in order to build. | 20:15 |
| fungi | also pretty uncool | 20:15 |
| *** andreykurilin__ has joined #openstack-infra | 20:16 | |
| EmilienM | fungi: thx | 20:16 |
| *** spzala has joined #openstack-infra | 20:19 | |
| * anteaya heads out for a walk | 20:20 | |
| *** esker has quit IRC | 20:23 | |
| *** Sukhdev has quit IRC | 20:24 | |
| *** emagana has quit IRC | 20:24 | |
| *** dprince has joined #openstack-infra | 20:24 | |
| jeblair | and buck _must_ be in the path | 20:24 |
| jeblair | but i have a gerrit war now! | 20:25 |
| notmyname | what are the jobs gate-tempest-dsvm-neutron-src-python-swiftclient-juno and gate-tempest-dsvm-neutron-src-python-swiftclient? | 20:26 |
| notmyname | seems to be always failing for python-swiftclient patches, but I haven't noticed them before | 20:26 |
| *** ZZelle_ has joined #openstack-infra | 20:29 | |
| *** tonytan4ever has joined #openstack-infra | 20:30 | |
| kevinbenton | can someone help me out and explain why https://review.openstack.org/#/c/174033/ isn't in the gate queue yet? | 20:30 |
| *** mpavone has quit IRC | 20:31 | |
| notmyname | ah, I found a note with "failing because of older (capped) versions of swift client in icehouse/juno. what's going on?" | 20:31 |
| kevinbenton | it passed the check quite a while ago | 20:31 |
| notmyname | kevinbenton: there was a message recently of gerrit being restarted to clear the event queue. might need to be rechecked | 20:31 |
| kevinbenton | how recent? like in the last hour? | 20:32 |
| notmyname | kevinbenton: "NOTICE: gerrit has been restarted to clear a problem with its event stream. any gerrit changes updated or approved between 19:14 and 19:46 utc will need to be rechecked or have their approval reapplied for zuul to pick them up" | 20:32 |
| kevinbenton | notmyname: if i just approve the patch with my account, should that cover it? | 20:33 |
| *** esker has joined #openstack-infra | 20:33 | |
| notmyname | kevinbenton: you can probably just leave a "recheck" comment and it should pick it up | 20:34 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: Add script to update .gitreview files https://review.openstack.org/174594 | 20:34 |
| kevinbenton | notmyname: i don't want to go through the check queue yet again though if i can avoid it | 20:34 |
| kevinbenton | notmyname: this patch and it's dependent need to go in for the kilo RC | 20:34 |
| *** achanda has quit IRC | 20:35 | |
| notmyname | kevinbenton: as one who also deals with those issues, I'm sorry. I think that's just the way it works | 20:35 |
| kevinbenton | notmyname: and the 'reverify' keyword is gone, right? | 20:35 |
| notmyname | I don't know | 20:35 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Temporarily ignore requirements on master libs https://review.openstack.org/174595 | 20:36 |
| fungi | ttx: dhellmann: ^ the job overrides you requested | 20:36 |
| dhellmann | fungi: thanks! | 20:36 |
| fungi | i'll prep a similar wip revert for that too | 20:36 |
| fungi | note i skipped cliff because its reqs check job was already non-voting | 20:37 |
| mestery | anteaya: sorry, had stepped out. | 20:37 |
| dhellmann | fungi: ok | 20:37 |
| mestery | anteaya: Neutron ryu CI is owned by yamamoto | 20:37 |
| dhellmann | fungi: I added that to https://etherpad.openstack.org/p/the-big-thaw | 20:37 |
| fungi | dhellmann: thanks, wasn't quite sure where to wedge it in there ;) | 20:37 |
| *** luqas has joined #openstack-infra | 20:37 | |
| dhellmann | yeah, that's getting a bit messy :-) | 20:37 |
| jeblair | fungi: do you have a thread stack dump from one of the stuck threads? | 20:38 |
| fungi | jeblair: nope, i forgot to grab one before i restarted this time | 20:38 |
| hashar | hey folks :) | 20:39 |
| *** achanda has joined #openstack-infra | 20:40 | |
| hashar | am I correct that the nodepool command is just interacting with the nodepoold database? In the sense it does not query remotes providers? | 20:40 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Temporarily ignore requirements on master libs https://review.openstack.org/174595 | 20:40 |
| dhellmann | fungi: have any idea why https://review.openstack.org/#/c/174488/ didn't merge? jenkins approves and the depends-on patch landed | 20:41 |
| *** mpavone has joined #openstack-infra | 20:41 | |
| fungi | hashar: it considers the nodepoold database as the source of truth anyway. it _does_ also interact with providers and targets, but generally only to take action | 20:41 |
| hashar | fungi: thx :) | 20:42 |
| *** luqas has quit IRC | 20:42 | |
| *** rkukura_ has joined #openstack-infra | 20:42 | |
| hashar | I wanted to figure out whether nodepool can communicate with the openstack provider | 20:42 |
| * hashar starts the daemon and hopes | 20:42 | |
| *** rkukura has quit IRC | 20:42 | |
| *** rkukura_ is now known as rkukura | 20:42 | |
| fungi | dhellmann: yeah, you need to reapply your workflow +1 for that. implementing dependent detection for enqueuing requires looking inside the commit message of all open changes in gerrit currently, so isn't implemented yet (though i think there might be a change in progress to add something like that, i'm not positive) | 20:43 |
| dhellmann | fungi: ok | 20:43 |
| *** markvoelker has quit IRC | 20:44 | |
| jeblair | fungi: i thought it was in production | 20:44 |
| fungi | oh? | 20:44 |
| fungi | maybe that change didn't auto-enqueue for some other reason then | 20:44 |
| sdague | so... we're down to about 100 node quota in RAX now? | 20:45 |
| fungi | the dependency merged at 17:56 so not a victim of our gerrit restart | 20:45 |
| *** emagana has joined #openstack-infra | 20:46 | |
| *** annegentle has quit IRC | 20:46 | |
| sdague | fungi: +2 on https://review.openstack.org/#/c/174595 | 20:46 |
| fungi | sdague: worse, it seems we have a total of 2 nodes in use in rax right now, and 12 in use in hpcloud | 20:47 |
| sdague | fungi: \o/ | 20:47 |
| fungi | i'll see if i can tell what's up in rax | 20:47 |
| *** cdent has quit IRC | 20:47 | |
| fungi | hpcloud is down because we took it down | 20:47 |
| sdague | going to be tons of fun to try to resolve all this requirements unwind | 20:47 |
| sdague | yeh, I vaguely saw | 20:47 |
| *** dkranz has quit IRC | 20:50 | |
| fungi | huh, actually we have almost no nodes at all, period | 20:53 |
| *** dannywilson has joined #openstack-infra | 20:54 | |
| fungi | AttributeError: 'ProviderManager' object has no attribute 'stop' | 20:54 |
| *** dprince has quit IRC | 20:54 | |
| *** mpavone has quit IRC | 20:54 | |
| fungi | mordred: did you verbatim apply your max 0 patch to nodepool.yaml, or do something else? | 20:54 |
| fungi | looks okay and first glance | 20:55 |
| fungi | i wonder if something about one of the recent nodepool patches has broken configuration reloading | 20:55 |
| fungi | jeblair: clarkb: ^ | 20:56 |
| fungi | i'm going to restart nodepoold to get it back onto track | 20:56 |
| fungi | hopefully | 20:56 |
| hashar | for nodepool what you guys think of removing upper version requirements for sqlalchemy and pyzmq ? | 20:56 |
| *** cburgess_ has quit IRC | 20:58 | |
| *** ldnunes has quit IRC | 20:58 | |
| *** cburgess has joined #openstack-infra | 20:58 | |
| *** ddieterly has quit IRC | 20:59 | |
| *** pc_m has quit IRC | 20:59 | |
| *** dannywil_ has joined #openstack-infra | 21:01 | |
| *** patrickeast_ has joined #openstack-infra | 21:02 | |
| *** patrickeast has quit IRC | 21:02 | |
| *** patrickeast_ is now known as patrickeast | 21:02 | |
| *** dannywilson has quit IRC | 21:03 | |
| *** andreykurilin__ has quit IRC | 21:04 | |
| jeblair | hashar: sqlalchemy regularly releases backwards incompatible changes, so we have that for safety. we could increase them if they are compatible | 21:06 |
| *** rfolco has quit IRC | 21:06 | |
| *** baoli has quit IRC | 21:06 | |
| *** otter768 has joined #openstack-infra | 21:06 | |
| jeblair | fungi: yes, that would be an error in yolandas patch that would cause config reloading to break | 21:06 |
| jeblair | fungi: if we're going to rax-only, we can restart without that patch | 21:06 |
| *** yfried__ is now known as yfried|afk | 21:07 | |
| *** baoli has joined #openstack-infra | 21:07 | |
| *** tnovacik has joined #openstack-infra | 21:07 | |
| openstackgerrit | Merged openstack-dev/pbr: Test that MANIFEST.in excludes work https://review.openstack.org/123597 | 21:07 |
| jeblair | fungi: good news: i have found a way to test my stream events patch | 21:08 |
| *** bswartz has quit IRC | 21:08 | |
| hashar | jeblair: at least tests pass with sqlalchemy requirement softened. I have proposed https://review.openstack.org/#/c/173731/ | 21:08 |
| jeblair | d1ba4ca6 14:13:40.153 Stream Events (corvus) | 21:08 |
| jeblair | fungi: ^ | 21:08 |
| fungi | jeblair: well, for the moment it's restarted and seems to be okay, but yes any further changes to nodepoold are going to break it again | 21:08 |
| fungi | er, to nodepool.ymal i mean | 21:08 |
| fungi | yaml | 21:09 |
| fungi | something | 21:09 |
| jeblair | yeah, let's let it settle a bit then restart again with ut the patch | 21:09 |
| fungi | jeblair: did the previous patch need any modifications? | 21:09 |
| jeblair | fungi: yeah, there's an anonymous inner class that is the actual thing in the queue, but since it's anonymous i guess it doesn't show up in the default toString method. so my toString override was on the outer class not the inner one. | 21:11 |
| *** otter768 has quit IRC | 21:11 | |
| jeblair | fungi: patches almost ready | 21:11 |
| *** gyee_ has joined #openstack-infra | 21:11 | |
| *** harlowja is now known as harlowja_away | 21:11 | |
| *** emagana has quit IRC | 21:12 | |
| fungi | okay, cool. having looked at the way the class structures are nested (from when i was trying to figure out how to make the contact info api error messages configurable), it's spaghetti | 21:12 |
| elarson | using jenkins job builder, is there a recommended way to clone a repo from github? | 21:12 |
| *** mwagner_lap has quit IRC | 21:13 | |
| fungi | elarson: we use a builder that runs a script to clone a repo | 21:13 |
| *** EmilienM is now known as EmilienM|afk | 21:13 | |
| *** tonytan4ever has quit IRC | 21:13 | |
| elarson | fungi: rather than an scm macro? | 21:14 |
| fungi | elarson: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/gerrit-git-prep.sh | 21:14 |
| *** Prophidys2 has joined #openstack-infra | 21:14 | |
| fungi | the scm macro is sort of limited in its ability to reuse local caches | 21:14 |
| fungi | and we pass in refnames to check out in envvars (from zuul job parameters) so we can read those directly in the script | 21:15 |
| elarson | fungi: can you show me a job that is configured to use that builder? | 21:15 |
| f1ller | fungi: your refs come from webhook posts? | 21:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/gerrit: Revert "Add username to stream-events queue entries" https://review.openstack.org/174606 | 21:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/gerrit: Add username to stream-events queue entries https://review.openstack.org/174607 | 21:15 |
| * elarson is trying to connect the dots | 21:15 | |
| f1ller | sorry, im on elarson's team and we are talking jjb implementation :) | 21:15 |
| *** f1ller is now known as filler | 21:15 | |
| fungi | f1ller: our refs come from zuul merger workers | 21:15 |
| jeblair | fungi, zaro, clarkb, mordred: ^ that's staged as a revert and re-do so that i can push the same patch to upstream gerrit | 21:16 |
| fungi | filler: elarson: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml the first macro there is the builder running that script | 21:16 |
| *** annegentle has joined #openstack-infra | 21:16 | |
| fungi | filler: elarson: then we just add "gerrit-git-prep" to the list of builders for a particular job definition | 21:16 |
| *** _nadya_ has joined #openstack-infra | 21:17 | |
| *** tiswanso has quit IRC | 21:17 | |
| fungi | filler: elarson: and we make sure when nodepool builds our worker images, it preinstalls that script on them in the path you see referred to in the macro | 21:17 |
| *** Prophidys has quit IRC | 21:17 | |
| jeblair | OMG | 21:19 |
| jeblair | mordred: the google account stuff is SO messed up because of hp using google docs | 21:20 |
| elarson | fungi: ok, I *think* that makes enough sense to me in order to experiment a bit with a builder | 21:21 |
| elarson | we'll see what happens | 21:21 |
| elarson | fungi: Thanks! | 21:21 |
| fungi | elarson: filler: good luck! | 21:21 |
| *** _nadya_ has quit IRC | 21:21 | |
| *** dizquierdo has joined #openstack-infra | 21:22 | |
| fungi | jeblair: tell me about it. i signed into an openstack foundation google doc at one point and then i couldn't use youtube any longer. it about drove me apeshit because google freaks out and wants to merge multiple accounts if you ever logged into more than one of their properties with different e-mail addresses | 21:22 |
| jeblair | fungi, mordred: http://i.imgur.com/35AWLa1.png | 21:22 |
| jeblair | fungi: yeah, i think that's what's happening here | 21:23 |
| jeblair | THAT IS INSANE | 21:23 |
| fungi | yes, exactly that | 21:23 |
| *** dims__ has quit IRC | 21:23 | |
| jeblair | but afaict, i can't use gerrit code review with the organizational account | 21:23 |
| jeblair | it will not let me move that to that account | 21:23 |
| jeblair | my only choice is to move it to a new "personal" account | 21:23 |
| jeblair | however, it is not personal | 21:23 |
| jeblair | so i'm pretty sure that means that i can not submit changes to gerrit with my hp account and email address | 21:24 |
| jeblair | which means i can not submit changes to gerrit | 21:24 |
| fungi | yeah, that's downright stupid | 21:24 |
| filler | oh, i had another jjb q if you have a second. | 21:27 |
| filler | :) | 21:27 |
| openstackgerrit | Merged openstack-infra/system-config: Turn off HP in nodepool https://review.openstack.org/174578 | 21:27 |
| *** yfried|afk is now known as yfried__ | 21:27 | |
| fungi | filler: i can't leave--i'm nailed to the seat | 21:27 |
| filler | haha | 21:27 |
| *** tnovacik has quit IRC | 21:28 | |
| filler | shunting jenkins console logs and related to swift. | 21:28 |
| fungi | ooh, yeah that's a very fun topic | 21:28 |
| *** bswartz has joined #openstack-infra | 21:29 | |
| fungi | i think yesterday clarkb worked out one retry kink in our console log collector script, but otherwise it's pretty stable now | 21:29 |
| *** eharney has quit IRC | 21:30 | |
| *** aysyd has quit IRC | 21:30 | |
| filler | so some python in slave_scripts shunts the logs to the store | 21:30 |
| filler | does that code swizzle jenkins job history to point to said uploaded logs? | 21:31 |
| fungi | dunno if you've looked at it, but basically it injects a unique marker into the console log at the end of the job, then loops reads from the jenkins http interface to retrieve the log until it spots the marker | 21:31 |
| filler | sorry, im not finding slave_scripts in scm yet :) | 21:31 |
| fungi | it doesn't modify the job history, but it does add a link in the jenkins console log to where it uploaded the console log | 21:31 |
| *** mattfarina has quit IRC | 21:31 | |
| fungi | filler: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/scripts/zuul_swift_upload.py | 21:32 |
| fungi | jhesketh wrote the meat of that | 21:32 |
| fungi | the uploads are authorized by zuul creating temporary credentials for write access to a subtree of a swift container, and then passing those credentials along through the gearman request so they end up as job parameters and eventually envvars in the upload script's calling environemnt | 21:34 |
| *** yamahata has quit IRC | 21:36 | |
| *** yamahata has joined #openstack-infra | 21:37 | |
| *** yfried__ is now known as yfried|afk | 21:37 | |
| *** hashar has quit IRC | 21:37 | |
| *** FallenPegasus has joined #openstack-infra | 21:38 | |
| jeblair | fungi: okay, so there's a tiny "not now" link somewhere in that process that let me log into gerrit with my "personal" hp.com account | 21:38 |
| jeblair | fungi: which is actually now <jeblair%hp.com@gtempaccount.com> | 21:38 |
| *** emagana has joined #openstack-infra | 21:38 | |
| fungi | hah | 21:38 |
| jeblair | and after re-registering my hp email with gerrit... | 21:38 |
| jeblair | (and for some reason, when i did that, 3 other really weird email addresses showed up too) | 21:38 |
| jeblair | I was eventually able to do this: https://gerrit-review.googlesource.com/#/c/67126/ | 21:39 |
| *** jamielennox|away is now known as jamielennox | 21:39 | |
| *** stevemar has quit IRC | 21:39 | |
| jeblair | zaro: i submitted a change to gerrit! https://gerrit-review.googlesource.com/#/c/67126/ | 21:39 |
| jeblair | okay. | 21:40 |
| jeblair | i'm looking at the new change screen and trying to figure out how to leave a comment | 21:40 |
| *** erikmwilson has quit IRC | 21:41 | |
| fungi | jeblair: you have to leave a "like" | 21:41 |
| fungi | and then tweet the comment | 21:41 |
| fungi | otherwise it might not get your avatar right | 21:41 |
| *** markvoelker has joined #openstack-infra | 21:42 | |
| zaro | jeblair: comment is now reply | 21:42 |
| jeblair | zaro: ah, yeah, i see the reply button at the top | 21:43 |
| jeblair | that's an interesting choice since the comments are at the bottom | 21:43 |
| *** annegentle has quit IRC | 21:44 | |
| zaro | at least jargon is now more in line with your email like vision | 21:44 |
| *** annegentle has joined #openstack-infra | 21:45 | |
| zaro | i think clarkb raised that had that same comment, but you can also reply from any comment. a little hidden but there's a reply arrow on every comment pane, just need to expand the comment | 21:45 |
| *** dannywil_ has quit IRC | 21:46 | |
| *** annegentle has quit IRC | 21:47 | |
| *** dannywilson has joined #openstack-infra | 21:47 | |
| *** annegentle has joined #openstack-infra | 21:48 | |
| fungi | wow, yeah, hpcloud was seriously in a bad way. my alien mop-up from this morning wrapped up right around when we dropped the max-servers there to 0. _while_ that cleanup was going on, we basically re-filled the entire 600 instance quota with alien nodes again | 21:51 |
| fungi | there are 515 presently | 21:51 |
| fungi | i'll go ahead and slowly delete those | 21:52 |
| *** hdd has quit IRC | 21:52 | |
| *** dizquierdo has quit IRC | 21:53 | |
| *** amitgandhinz has quit IRC | 21:54 | |
| jeblair | zaro: ah, i see the arrow. i did not figure that out. :/ | 21:54 |
| fungi | so we were leaking about 100 aliens an hour in hpcloud, i think | 21:54 |
| jeblair | fungi: in line with our creation rate :/ | 21:55 |
| *** EmilienM|afk is now known as EmilienM | 21:55 | |
| fungi | though i doubt it was anywhere near that linear since there's probably a bit of a feedback loop in performance | 21:55 |
| clarkb | zaro jeblair yes I think its a terrible ui decision | 21:59 |
| *** rlandy has quit IRC | 22:00 | |
| *** yamahata has quit IRC | 22:02 | |
| *** otter768 has joined #openstack-infra | 22:03 | |
| *** mattfarina has joined #openstack-infra | 22:03 | |
| *** ddieterly has joined #openstack-infra | 22:03 | |
| *** esker has quit IRC | 22:05 | |
| *** mattfarina has quit IRC | 22:05 | |
| *** dims__ has joined #openstack-infra | 22:08 | |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Ignore non-release related tags https://review.openstack.org/142144 | 22:09 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Reinstate pure-git-hashes-are-illegal test. https://review.openstack.org/174625 | 22:09 |
| *** annegentle has quit IRC | 22:09 | |
| *** annegentle has joined #openstack-infra | 22:10 | |
| *** gordc has quit IRC | 22:10 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:12 | |
| *** harlowja_away is now known as harlowja | 22:13 | |
| *** markvoelker has quit IRC | 22:13 | |
| *** baoli has quit IRC | 22:13 | |
| *** mrmartin has quit IRC | 22:14 | |
| *** baoli has joined #openstack-infra | 22:14 | |
| *** annegentle has quit IRC | 22:15 | |
| jeblair | fungi, clarkb: i self-approved 174607 because i think it's trivial and i'd like to get it built before the next gerrit restart | 22:16 |
| jeblair | feel free to review and/or object; i'm sure we have a while before zuul gets around to actually doing anything with it | 22:16 |
| fungi | jeblair: lgtm, sorry hadn't looked until just now | 22:17 |
| fungi | and i agree, having it in place before the next inevitable restart would be swell | 22:17 |
| *** tkelsey has joined #openstack-infra | 22:17 | |
| jeblair | fungi: i'm going to restart nodepool without yolanda's change, then enable puppet | 22:18 |
| fungi | thanks jeblair! | 22:19 |
| openstackgerrit | Merged openstack-infra/gerrit: Revert "Add username to stream-events queue entries" https://review.openstack.org/174606 | 22:19 |
| jeblair | huh, that didn't take that long after all | 22:19 |
| fungi | our test nodes graph, while not as tall as usual, looked healthier after the last restart than i've seen it in a long time (as proportion of used to all other node types is concerned) | 22:20 |
| *** mrmartin has joined #openstack-infra | 22:20 | |
| openstackgerrit | Merged openstack-infra/gerrit: Add username to stream-events queue entries https://review.openstack.org/174607 | 22:20 |
| jeblair | fungi: we will also likely find that nodepool performs better without a cloud lying to it all the time | 22:20 |
| openstackgerrit | Merged openstack-infra/nodepool: Do not set up managers on CLI delete https://review.openstack.org/174163 | 22:20 |
| *** erikmwilson has joined #openstack-infra | 22:21 | |
| fungi | for the moment at least, rackspace is booting new instances quite quickly and deleting those we're done with near instantly | 22:21 |
| *** yamahata has joined #openstack-infra | 22:21 | |
| fungi | also you can tell hpcloud was effectively dead because the rate at which we're burning down the waiting jobs in zuul has been linear even through dropping hpcloud from the config | 22:22 |
| *** tkelsey has quit IRC | 22:22 | |
| *** erikmwilson has quit IRC | 22:22 | |
| jeblair | fungi: agreed | 22:22 |
| jeblair | the anomoly was just when nodepool broked | 22:22 |
| fungi | and the running jobs line has been steady | 22:22 |
| *** erikmwilson_ is now known as erikmwilson | 22:23 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/jeepyb: New hook: cherry_pick https://review.openstack.org/174627 | 22:23 |
| lifeless | 6555555555555555555555555555555555~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | 22:23 |
| fungi | lifeless: is that your new postal code? | 22:23 |
| lifeless | ECAT | 22:23 |
| fungi | indeed. i get those too | 22:23 |
| *** weshay has quit IRC | 22:24 | |
| jeblair | fungi, clarkb, mordred: puppet agent re-enabled on nodepool, and nodepool is running master (so should be safe for puppet-driven configuration changes) | 22:24 |
| fungi | error: your keyboard has suddenly grown a cat | 22:24 |
| *** mrmartin has quit IRC | 22:25 | |
| *** mjordan has left #openstack-infra | 22:25 | |
| jeblair | fungi: unicomp should develop a line of cat-resistant keyboards | 22:25 |
| *** dboik_ has joined #openstack-infra | 22:25 | |
| zaro | mordred: is this https://review.openstack.org/#/c/164264 even required after we migrate db and all tables to utf8? | 22:25 |
| fungi | i'm sure it's buried somewhere in the model m patent they acquired | 22:25 |
| jeblair | zaro: it's so that if someone runs the mysql client locally, it does the right thing | 22:26 |
| *** smccully has quit IRC | 22:28 | |
| *** sdake_ has joined #openstack-infra | 22:28 | |
| *** dboik has quit IRC | 22:29 | |
| *** otter768 has quit IRC | 22:29 | |
| *** dboik_ has quit IRC | 22:30 | |
| *** krtaylor has quit IRC | 22:31 | |
| *** mase_x200 has joined #openstack-infra | 22:31 | |
| *** jaypipes has quit IRC | 22:31 | |
| *** sdake has quit IRC | 22:31 | |
| *** annegentle has joined #openstack-infra | 22:31 | |
| lifeless | fungi: hey, so bindep | 22:32 |
| lifeless | fungi: mgedmin is interested in that :) | 22:32 |
| lifeless | fungi: whats the current status, or should I just ignore it ? | 22:32 |
| *** annegentle has quit IRC | 22:33 | |
| *** sdake_ has quit IRC | 22:33 | |
| EmilienM | mordred: I'm really not sure if openstack-infra would like this hook, but I'm trying something that would help us (Puppet) to improve our modules https://review.openstack.org/#/c/174627/ | 22:33 |
| *** annegentle has joined #openstack-infra | 22:33 | |
| EmilienM | any feedback is welcome ^^^^^ | 22:34 |
| *** dimtruck is now known as zz_dimtruck | 22:34 | |
| fungi | lifeless: i think there are still some patches which need review on it. you have retained core reviewer status on it, remember | 22:35 |
| *** otter768 has joined #openstack-infra | 22:35 | |
| fungi | lifeless: it's shaping up. i'm unconvinced the package version parsing bits are of much help for our specific use cases (since it's rare that you're going to be running it on a distro which provides you with a choice of multiple versions of a given package name, some of which will be satisfactory as a requirement while others won't) but otherwise it's all useful | 22:36 |
| lifeless | fungi: cool. Uhm, its less rare than you think, I think. | 22:36 |
| lifeless | fungi: *every* ubuntu install has multiple versions present, for instance. | 22:36 |
| *** annegentle has quit IRC | 22:37 | |
| fungi | lifeless: yes, i get that for a given package there may be backports and stable updates. the chances that you won't have installed the most recent one available to you and that bindep is going to tell you that you need to is slim compared to the complexity involved | 22:37 |
| fungi | also i couldn't figure out how to get that to happen sanely with rpm-based platforms | 22:38 |
| lifeless | fungi: so rpm needs to be treated as yast || yum | 22:38 |
| fungi | they mostly only know about the version of the package you have installed and that's it. you need a higher-level tool (yum, yast, zypper, something) | 22:38 |
| lifeless | since rpm doesn't know about repositories | 22:38 |
| lifeless | (neither does dpkg) | 22:38 |
| fungi | well, the current bindep implementation just uses dpkg-status, not apt-cache or anything | 22:39 |
| lifeless | yeah | 22:39 |
| lifeless | I'm not actually sure if thats correct | 22:39 |
| fungi | i was trying to implement rpm parity to the dpkg implementation present | 22:39 |
| lifeless | e.g. there may be more in apt | 22:39 |
| lifeless | fungi: I wouldn't stress - I think degrading to assuming-its-ok is fine | 22:40 |
| *** EmilienM is now known as EmilienM|afk | 22:40 | |
| lifeless | fungi: I mean, we should still check the needed version is met | 22:40 |
| fungi | it can certainly grow that capability, but i think for our (infra ci job running and helping devs out a little) use case it's fine to not put version constraints in bindep lists for now | 22:40 |
| lifeless | thats an option to | 22:41 |
| lifeless | uhm, I'd be a little sad to rip the code out; is there any need to do that? | 22:41 |
| fungi | instead i've been going with the "if you're on trusty you need this package name, but on precise it's this, and on centos 6 it's this" tactic | 22:41 |
| fungi | i haven't ripped any code out | 22:41 |
| *** mriedem has quit IRC | 22:42 | |
| fungi | i's just that for the moment the patches i have up don't support version constraints on rpm-based platforms | 22:42 |
| lifeless | where did bindep end up? openstack-infra/bindep ? | 22:42 |
| fungi | yes | 22:42 |
| fungi | open reviews are at https://review.openstack.org/#/q/status:open+project:openstack-infra/bindep,n,z | 22:43 |
| fungi | there's fewer than i thought. most have been approved already | 22:43 |
| *** dangers is now known as dangers_away | 22:43 | |
| *** bknudson has quit IRC | 22:43 | |
| *** rkukura has quit IRC | 22:44 | |
| fungi | lifeless: anyway, your input would be awesome. i'd like to finish polishing it enough for a release before we start running it in jobs and nodepool prep-scripts/dib-elements | 22:45 |
| lifeless | k, I'm just paging it all back in | 22:45 |
| lifeless | fungi: oh, can you register it on pypi? | 22:45 |
| lifeless | fungi: easier than me doing it an handing it over | 22:45 |
| fungi | lifeless: oh! i thought i'd already checked that. great catch | 22:46 |
| fungi | doing now | 22:46 |
| *** isviridov_away is now known as isviridov | 22:46 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 | 22:46 |
| lifeless | fungi: are you using mapping or profiles for centos etc? | 22:47 |
| lifeless | fungi: my intent was that we'd have a database of the most common package name translations in the tool, rather than having to have that in each other-requirements.txt file | 22:47 |
| fungi | lifeless: so far i've just expected to have alternatives in the lists themselves, yes | 22:48 |
| *** markvoelker has joined #openstack-infra | 22:48 | |
| lifeless | ok | 22:48 |
| fungi | because it was a quick way to get where we need. it can obviously be improved on | 22:48 |
| lifeless | we can take a lazy approach and see what duplication shows up and fold that into a db | 22:49 |
| fungi | also https://pypi.python.org/pypi/bindep is a thing now | 22:49 |
| lifeless | thanks! | 22:49 |
| *** markvoelker_ has joined #openstack-infra | 22:49 | |
| lifeless | dstufft: how would you feel about an --only-dependencies flag for pip install ? | 22:50 |
| fungi | lifeless: https://review.openstack.org/#/c/172674/1/jenkins/data/bindep-fallback.txt is a good current example of how to turn a devstack-.* node into a bare-.* node for nodepool | 22:50 |
| *** dims__ has quit IRC | 22:50 | |
| *** markvoel_ has joined #openstack-infra | 22:50 | |
| fungi | lifeless: you can get a feel for the sorts of duplication we might expect on a smaller scale in individual repos | 22:50 |
| *** markvoe__ has joined #openstack-infra | 22:51 | |
| fungi | lifeless: that also might make a good start for a map | 22:51 |
| lifeless | yup | 22:51 |
| fungi | lifeless: so far i've got it working with that data source for centos 6, ubuntu precise and ubuntu trusty (the fedora addition was at ianw's request) | 22:52 |
| *** markvoelker has quit IRC | 22:53 | |
| fungi | that is, working once my outstanding patches land | 22:53 |
| *** gyee_ has quit IRC | 22:53 | |
| dstufft | lifeless: I don't hate the idea, what's the use case | 22:53 |
| fungi | also it started out with great testing (thank you btw!) so i've done my best to get ocd about keeping that going with new patches | 22:53 |
| *** MarkAtwood has quit IRC | 22:53 | |
| *** FallenPegasus is now known as MarkAtwood | 22:54 | |
| *** markvoelker_ has quit IRC | 22:54 | |
| *** dboik_ has joined #openstack-infra | 22:54 | |
| jeblair | greghaynes, clarkb: is the dib change to support https://review.openstack.org/126747 released? | 22:54 |
| jeblair | (multiple image types) | 22:54 |
| jeblair | looks like it needs the "-t foo,bar" option | 22:55 |
| *** markvoel_ has quit IRC | 22:55 | |
| openstackgerrit | Merged openstack-infra/bindep: Add codename and release platform profiles https://review.openstack.org/157979 | 22:55 |
| *** otter768 has quit IRC | 22:55 | |
| lifeless | dstufft: getting the deps for something installed without installing it | 22:56 |
| lifeless | dstufft: see e.g. http://coldwa.st/e/blog/2013-08-20-Cabal-sandbox.html for the inspiration | 22:56 |
| lifeless | dstufft: I currently do 'pip install -e .; pip uninstall thing' to achieve the same thing, which is a bit wasteful | 22:56 |
| openstackgerrit | Merged openstack-infra/bindep: Support for older dpkg https://review.openstack.org/170189 | 22:57 |
| jeblair | greghaynes, clarkb: oh, long, long ago. nevermind | 22:57 |
| dstufft | lifeless: well specifically I mean, under what situations is such a flag something you want to do | 22:57 |
| greghaynes | jeblair: yea, dib supports that already | 22:57 |
| *** davideagnello has quit IRC | 22:57 | |
| dstufft | I don't think I've ever really wanted to do it is why I'm asking | 22:57 |
| lifeless | dstufft: oh | 22:58 |
| lifeless | dstufft: I often have multiple working trees for some classes of project | 22:58 |
| *** otter768 has joined #openstack-infra | 22:58 | |
| lifeless | dstufft: and this means I don't need working trees * supported pythons venvs; I can have just one per supported python | 22:58 |
| fungi | lifeless: i like the https://github.com/likema/lsb_release_ex suggestion. i'll add a to-do to look into integrating it | 22:59 |
| lifeless | dstufft: [note, I generally don't use tox venvs because I'm sshing into a brazillion containers to get different pythons] | 22:59 |
| openstackgerrit | Merged openstack-infra/bindep: Add platform support for Fedora https://review.openstack.org/171293 | 22:59 |
| dstufft | lifeless: I'm now imagining you host all your containers exlusively in brazil | 22:59 |
| lifeless | dstufft: I do! | 22:59 |
| * fungi bets brazilian containers are neatly waxed | 22:59 | |
| *** MarkAtwood has quit IRC | 22:59 | |
| dstufft | lifeless: yea ok, seems like a reasonable thing to want | 23:00 |
| lifeless | dstufft: [not]. Anyhow - so if I have the thing, which is generally an 'app' rather than a library, installed, and I forget which copy is installed, it goes badly | 23:00 |
| fungi | also there's a cachaça joke in there somewhere | 23:00 |
| lifeless | so having it not installed, but its deps installed, is safer :) | 23:00 |
| openstackgerrit | Merged openstack-infra/nodepool: Support per provider image build settings https://review.openstack.org/126747 | 23:00 |
| *** chlong has joined #openstack-infra | 23:02 | |
| openstackgerrit | Merged openstack-infra/nodepool: Just set qcow2 version https://review.openstack.org/169383 | 23:02 |
| openstackgerrit | Merged openstack-infra/nodepool: Use correct diskimage when building diskimages https://review.openstack.org/169474 | 23:03 |
| lifeless | mordred: https://review.openstack.org/#/c/150817/ - I'm marking this WIP, AFAICT its still needed though | 23:03 |
| fungi | interestingly, hpcloud is performing much better now (judging by the rate at which i just managed to delete >500 instances) than it was when we were pummeling it with nodepool earlier today | 23:04 |
| *** jgrimm is now known as zz_jgrimm | 23:04 | |
| lifeless | mordred: unless 381b6471b6dddce6bf741ee3795957e25d977b40 is the equivalent ? | 23:04 |
| *** sabeen has quit IRC | 23:04 | |
| jeblair | jesusaurus, cody-somerville: ^ see fungi | 23:04 |
| *** baoli has quit IRC | 23:05 | |
| lifeless | mordred: and there's another patch on top, which may need you to page in the same q | 23:06 |
| *** mase_x200 has quit IRC | 23:06 | |
| fungi | serial delete completed in ~20 minutes where the previous deletion under load of a roughly equivalent number following the same process took ~6 hours | 23:06 |
| jesusaurus | fungi: out of curiosity, how quickly were they deleted? | 23:06 |
| jesusaurus | wow, impressive | 23:06 |
| fungi | i expect most of that was the time it takes to run novaclient 500 times | 23:07 |
| jeblair | also impressive is fungi answering the question before it was asked | 23:07 |
| fungi | i operate on a temporal shift of roughly 60 seconds | 23:07 |
| fungi | it helps a lot | 23:07 |
| jeblair | five million dollors and a hedgehog | 23:08 |
| fungi | sometimes i'm even 20 minutes into the future | 23:08 |
| fungi | (for all you max headroom fans out there) | 23:08 |
| jesusaurus | heh | 23:09 |
| jeblair | (i clearly don't have the same skill) | 23:09 |
| *** yamamoto has joined #openstack-infra | 23:09 | |
| *** gyee_ has joined #openstack-infra | 23:09 | |
| *** EmilienM|afk is now known as EmilienM | 23:10 | |
| fungi | it's disconcerting, but i find that if i drink enough i can mostly ignore it | 23:11 |
| *** stevemar has joined #openstack-infra | 23:12 | |
| openstackgerrit | Merged openstack-infra/project-config: Add stackforge/puppet-nslcd https://review.openstack.org/165922 | 23:14 |
| openstackgerrit | Merged openstack-infra/system-config: Use "$puppetmaster_server" as parameter to the ssh class https://review.openstack.org/167646 | 23:15 |
| jeblair | that's a ^ could-possibly-break-everything change | 23:15 |
| *** stevemar has quit IRC | 23:17 | |
| fungi | i'll keep an eye out, thanks for the heads-up | 23:18 |
| anteaya | fungi: oh course you'll know before the rest of us if things break | 23:19 |
| fungi | touché | 23:19 |
| openstackgerrit | Merged openstack-infra/puppet-zanata: Add SMTP configuration options for Zanata https://review.openstack.org/168167 | 23:19 |
| anteaya | mestery: thanks | 23:20 |
| anteaya | I picked up a rock on the road that I think has garnet chips in it, my accomplishment of the day | 23:21 |
| anteaya | yamamoto: hello, I believe you operate the ryu ci? if yes could you please create a wikipage for it: https://wiki.openstack.org/wiki/ThirdPartySystems let me know if you need help | 23:22 |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 | 23:22 |
| yamamoto | anteaya: ryu ci == ofagent ci https://wiki.openstack.org/wiki/ThirdPartySystems/OFAgent_CI | 23:23 |
| jeblair | nibalizer: oh drat, i merged a change that conflicted with 171359, sorry. | 23:23 |
| *** dguerri is now known as _dguerri | 23:23 | |
| zaro | ls | 23:24 |
| *** _dguerri is now known as dguerri | 23:24 | |
| anteaya | yamamoto: ah okay well in gerrit the account is called ryu ci, so on that page I go looking for an ryu ci page | 23:24 |
| anteaya | yamamoto: can you create a link for an ryu ci page that links to your ofagenct_ci page? | 23:24 |
| anteaya | yamamoto: because I'm never going to remember that, nor can I tell everyone who goes looking for your page | 23:25 |
| nibalizer | jeblair: its all good i'll refresh it | 23:25 |
| yamamoto | anteaya: i will | 23:25 |
| anteaya | yamamoto: thank you | 23:25 |
| anteaya | yamamoto: after you do that, can you help us answer a question? | 23:25 |
| anteaya | yamamoto: we are looking at our open ssh connections since we are having an issue with stuck threads | 23:26 |
| yamamoto | anteaya: i'll do that later today. i have to run right now | 23:26 |
| anteaya | yamamoto: after we restarted gerrit today to unstick the threads, we happened to notice you had 14 open ssh connections | 23:26 |
| yamamoto | anteaya: can you mail me? yamamoto@valinux.co.jp thank you | 23:26 |
| anteaya | yamamoto: when you return perhaps you can take a look and maybe let us know why you might need 14 open ssh connections? | 23:26 |
| fungi | btw it's up to 17 now... i just checked again. not critical at the moment though | 23:26 |
| anteaya | fungi: thanks | 23:27 |
| anteaya | yamamoto: I can wait until you come back to irc | 23:27 |
| anteaya | actually better if we can all discuss it | 23:27 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/shade: create_user should return the user created https://review.openstack.org/174642 | 23:27 |
| anteaya | even if asyncronously | 23:27 |
| yamamoto | anteaya: ok | 23:27 |
| anteaya | yamamoto: thanks | 23:27 |
| fungi | purestorage-cinder is up to 8 and netapp-ci is at 7 now. so it looks like some of these might grow over time | 23:27 |
| *** yamamoto has quit IRC | 23:27 | |
| lifeless | timrc: hi - https://review.openstack.org/#/c/171309/ is failing CI | 23:27 |
| anteaya | fungi: okay, if we figure out what we consider a reasonable number is perhaps we can talk to the outliers? | 23:28 |
| anteaya | maybe they dont' know their system has so many open connections | 23:28 |
| fungi | anteaya: i honestly wouldn't know what to guess at as a reasonable number, though "jenkins" (our ci) had 3 open just now | 23:29 |
| anteaya | okay, so looking at random data points with no idea of what is reasonable, that helps | 23:30 |
| jeblair | no stuck gerrit threads yet | 23:30 |
| anteaya | yay | 23:30 |
| *** davideagnello has joined #openstack-infra | 23:31 | |
| fungi | they do seem to just come on out of nowhere though | 23:32 |
| fungi | i've been checking periodically in past weeks and generally there are no waiting threads at all | 23:32 |
| anteaya | do they arrive in bunches? like 3-5-13 sort of thing | 23:32 |
| fungi | then at some point, boom. dozens or more and gerrit stops streaming | 23:33 |
| jeblair | i've seen us have only one at one point | 23:33 |
| jeblair | i think i noticed that only once though | 23:33 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: New repos for Akanda https://review.openstack.org/174104 | 23:33 |
| fungi | yeah, it doesn't seem like a gradual condition though. there must be some sort of trigger | 23:34 |
| fungi | and then rapid cascade | 23:34 |
| *** sdake has joined #openstack-infra | 23:34 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Refactor o_p::template into o_p::server https://review.openstack.org/171359 | 23:34 |
| fungi | usually when i look before i end up restarting gerrit entirely, most of the waiting stream-events threads all have the same wait time on them | 23:34 |
| *** Sukhdev has joined #openstack-infra | 23:34 | |
| fungi | within fractions of a secnd anyway | 23:34 |
| fungi | er, waiting stream-events tasks | 23:35 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Revert "Support platform-specific requirements files" https://review.openstack.org/174646 | 23:35 |
| openstackgerrit | David Lenwell proposed openstack-infra/project-config: New repos for Akanda https://review.openstack.org/174104 | 23:35 |
| openstackgerrit | lifeless proposed openstack-dev/pbr: Revert "Support platform-specific requirements files" https://review.openstack.org/174646 | 23:39 |
| *** ddieterly has quit IRC | 23:40 | |
| *** thingee has quit IRC | 23:41 | |
| *** otter768 has quit IRC | 23:42 | |
| openstackgerrit | Khai Do proposed openstack-infra/puppet-gerrit: Make mysql clients assume utf8 character encoding https://review.openstack.org/164264 | 23:45 |
| *** isviridov is now known as isviridov_away | 23:45 | |
| *** otter768 has joined #openstack-infra | 23:48 | |
| anteaya | fungi: have we ever timed how long after a re-start the stuck threads occur? | 23:48 |
| *** yamamoto has joined #openstack-infra | 23:49 | |
| anteaya | or what time of day they happen? | 23:49 |
| fungi | my anecdotal recollection puts it as squarely random. sometimes it gets stuck before i come online in the morning, sometimes in the middle of my day, sometimes in my evening | 23:50 |
| *** hichihara has joined #openstack-infra | 23:50 | |
| fungi | the cadence doesn't seem to have been the same number of days between either | 23:50 |
| lifeless | right, *all* pbr patches reviewed | 23:50 |
| *** emagana has quit IRC | 23:50 | |
| lifeless | all bindep patches reviewed | 23:50 |
| lifeless | time for some food, then immolating myself on the fire of pip features | 23:51 |
| anteaya | fungi: yay random | 23:51 |
| fungi | lifeless: thanks! | 23:51 |
| yamamoto | anteaya: hi | 23:53 |
| yamamoto | anteaya: done https://wiki.openstack.org/wiki/ThirdPartySystems/Ryu_CI | 23:53 |
| * yamamoto reading backlog | 23:54 | |
| anteaya | yamamoto: wonderful thank you | 23:54 |
| *** sdake_ has joined #openstack-infra | 23:55 | |
| yamamoto | anteaya: 14 open connections from Ryu CI? i'll take a look when i goes to office. (i'm at home right now) | 23:55 |
| dstufft | lifeless: pip features for the pip features god | 23:55 |
| fungi | yamamoto: at current count, gerrit claims the neutronryu account has 20 open ssh connections into it. i'm curious how those are being used--it's an order of magnitude higher than most other ci systems interacting with our gerrit server | 23:56 |
| fungi | and it seems to steadily increase | 23:56 |
| yamamoto | fungi: interesting. i'll ask our ci expert. | 23:57 |
| yamamoto | fungi: our ci is behind NAT. can it be related? | 23:57 |
| *** rfolco has joined #openstack-infra | 23:58 | |
| anteaya | yamamoto: thank you | 23:58 |
| *** markvoe__ has quit IRC | 23:58 | |
| fungi | yamamoto: i suppose your nat might be causing you some connectivity problems or it could be leaving dead sockets open from our perspective, but i'm not looking at source ip addresses i'm looking at connections per account name | 23:58 |
| *** sdake has quit IRC | 23:59 | |
| fungi | yamamoto: anyway, let us know what you find out when you get a chance, and thanks for looking into it | 23:59 |
| openstackgerrit | Alice Chen proposed openstack-infra/jeepyb: Let pull request message be customized via file https://review.openstack.org/115850 | 23:59 |
| yamamoto | do ssh connections for gerrit have some sort of keepalive mechanism? | 23:59 |
| *** dannywilson has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
