| fungi | mspreitz: that sounds like evil gubments | 00:00 |
|---|---|---|
| *** thuc has quit IRC | 00:00 | |
| fungi | (the kill ssh connections with long key exchanges issue i mean) | 00:01 |
| *** thuc has joined #openstack-infra | 00:01 | |
| mspreitz | Or incompetent security vendors | 00:01 |
| mspreitz | I am going to try it from precise, see if that makes the difference | 00:02 |
| fungi | that would be most network security device vendors, yes | 00:02 |
| ArxCruz | dansmith: hey https://review.openstack.org/#/c/76388/ ;D | 00:02 |
| dansmith | ArxCruz: I just saw that! | 00:02 |
| ArxCruz | you passed on IBM PowerKVM Testing ;D | 00:02 |
| ArxCruz | dansmith: I was talking with krtaylor to configure our ci for always send a "No" when the patch is from you :) | 00:03 |
| krtaylor | hehheh | 00:03 |
| *** moted has quit IRC | 00:03 | |
| dansmith | ArxCruz: +1 to that | 00:03 |
| *** bhuvan_ has joined #openstack-infra | 00:03 | |
| fungi | ArxCruz: krtaylor: what we really need is a gerrit filter to keep dansmith's dependency chains at <=5 | 00:03 |
| cyeoh | ArxCruz: oooh nice to see it reporting! | 00:03 |
| ArxCruz | :) | 00:04 |
| ArxCruz | fungi: agreed :) | 00:04 |
| krtaylor | YES | 00:04 |
| fungi | though then we'd miss those 50-change rebase bombs | 00:04 |
| ArxCruz | lol | 00:05 |
| *** bhuvan has quit IRC | 00:05 | |
| krtaylor | I thought it was fitting that our first patch review was for dansmith | 00:05 |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Remove resolved_at mention in readme https://review.openstack.org/76393 | 00:05 |
| dansmith | oh was I really the first one? | 00:05 |
| *** thuc has quit IRC | 00:05 | |
| fungi | clarkb: i'm enabling puppet on es02 now | 00:06 |
| clarkb | fungi: ok | 00:07 |
| *** bhuvan has joined #openstack-infra | 00:08 | |
| *** bhuvan_ has quit IRC | 00:09 | |
| *** michchap has quit IRC | 00:10 | |
| *** bhuvan_ has joined #openstack-infra | 00:13 | |
| clarkb | fungi: how is it? I don't see es02 in bigdesk yet | 00:13 |
| *** lcostantino has joined #openstack-infra | 00:13 | |
| fungi | clarkb: waiting for the agent to take its natural course, but i'll double-check logs | 00:14 |
| *** bhuvan has quit IRC | 00:14 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Add a remote url override location https://review.openstack.org/76057 | 00:14 |
| *** yamahata has quit IRC | 00:14 | |
| clarkb | jeblair: jhesketh mordred fungi ^ that addresses the existing comments and adds docs for the feature. | 00:14 |
| clarkb | mordred: AaronGr ^ you should probably pull that down and test it | 00:14 |
| clarkb | er I should've run tox locally too /me does that now | 00:15 |
| dansmith | ArxCruz: plans for extending that to be a full run of tempest? | 00:15 |
| fungi | clarkb: my bad :( | 00:15 |
| fungi | Feb 26 00:13:33 elasticsearch02 puppet-agent[24532]: (/Stage[main]/Elasticsearch/File[/etc/elasticsearch/templates]/ensure) change from absent to directory failed: Cannot create /etc/elasticsearch/templates; parent directory /etc/elasticsearch does not exist | 00:15 |
| ArxCruz | krtaylor: can answer that for you :D | 00:15 |
| fungi | the package probably created that before | 00:15 |
| ArxCruz | dansmith: ^ | 00:15 |
| clarkb | fungi: oh right | 00:15 |
| * fungi does a quick scrub for similar problems | 00:15 | |
| sdague | jeblair: cool on logstash queue! | 00:16 |
| clarkb | oh good my change is tox clean | 00:16 |
| *** pmathews has quit IRC | 00:16 | |
| krtaylor | dansmith, it is really a matter of increasing hardware, all in good time | 00:17 |
| dansmith | krtaylor: ah, makes sense | 00:17 |
| sdague | jeblair: any chance we could figure out how far behind we are in logstash instead of just queue depth? | 00:18 |
| *** michchap has joined #openstack-infra | 00:18 | |
| *** lcostantino has quit IRC | 00:19 | |
| jeblair | sdague: translate depth -> time? | 00:19 |
| sdague | jeblair: yeh, basically | 00:20 |
| clarkb | sdague: you can figure that out with a simple es query | 00:20 |
| clarkb | just look for the last console.html timestamp essentially | 00:20 |
| clarkb | which is probably more accurate than depth -> time | 00:20 |
| jeblair | sdague: oh, yeah, i guess there's two things related to time... | 00:20 |
| sdague | clarkb: true | 00:20 |
| jeblair | sdague: clarkb described "what's the most recent entry", and now-then is how far behind we are... | 00:21 |
| sdague | that would be a nice thing to have a running graph for, as that would give us an idea on the impact | 00:21 |
| jeblair | sdague: but there's also "how long do we have to catch up?" | 00:21 |
| jeblair | sdague: which is a function of queue depth and a variable | 00:21 |
| *** michchap has quit IRC | 00:21 | |
| sdague | well, to an approximation, that's first derivative of delay | 00:22 |
| openstackgerrit | Marton Kiss proposed a change to openstack-infra/config: Fix openstackid vhost override https://review.openstack.org/76397 | 00:22 |
| *** michchap has joined #openstack-infra | 00:22 | |
| mrmartin | fungi: I made a patch that fix a symlink file removal in openstackid config, may I get an approval for that? https://review.openstack.org/#/c/76397/ | 00:23 |
| jeblair | sdague: yes, also the 1st derivative of depth | 00:23 |
| jeblair | (btw, graphite does derivates too i think) | 00:24 |
| jeblair | sdague: at any rate, queue depth should at least shed some visibility on it -- i mean, when things are working as planned, the queue shouldn't be more than a few hundred or maybe thousand. | 00:24 |
| sdague | ok, so 90k... bad? | 00:25 |
| jeblair | sdague: yes, very bad | 00:25 |
| *** dolphm is now known as dolphm_503 | 00:25 | |
| jeblair | sdague: if memory serves, that translates to "many hours" for most of those time values | 00:25 |
| fungi | mrmartin: it looks fine, passes checks and only touches openstackid, so i'm fine approving it to speed things along | 00:25 |
| mrmartin | fungi: if it pass through well, may I ask you another rm -rf /srv/openstackid ? I hope it will work now, passed the local vagrant tests | 00:26 |
| sdague | ok, maybe I'll just get intuition on it. The units just don't mean anything to me yet | 00:26 |
| mrmartin | if it won't I'll go to sleep :) | 00:26 |
| fungi | mrmartin: sure, i'll do that once the change is in place on the puppet master in a few minutes, though it's also very late where you are, so sleep is a good idea regardless | 00:27 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix openstackid vhost override https://review.openstack.org/76397 | 00:27 |
| fungi | that ^ should appear on the master in about 2 minutes, just after 00:30 utc | 00:28 |
| mrmartin | fungi: I want to see something that works | 00:28 |
| fungi | mrmartin: i know how you feel | 00:28 |
| mrmartin | :) | 00:28 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Puppet the /etc/elasticsearch directory https://review.openstack.org/76399 | 00:29 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch workers https://review.openstack.org/76051 | 00:29 |
| fungi | clarkb: i think https://review.openstack.org/76399 is all es02 and later need. i'll try that in a dev env | 00:30 |
| *** matsuhashi has joined #openstack-infra | 00:30 | |
| clarkb | k | 00:30 |
| clarkb | looks good from here | 00:31 |
| fungi | mrmartin: deleted it again just now. looks like the next puppet run should be in a minute or two, around 00:34 utc | 00:31 |
| mrmartin | oh great | 00:32 |
| clarkb | jhesketh: https://review.openstack.org/#/c/68828/10/zuul/cmd/client.py has a neat use of babel I will need to remember that rtick | 00:32 |
| *** CaptTofu has joined #openstack-infra | 00:34 | |
| jhesketh | clarkb: :-), that was me googling to not reinvent the wheel | 00:36 |
| fungi | clarkb: elasticsearch service didn't start on es02... if you have a moment, ssh in and gawk at dmesg with me whydontcha | 00:36 |
| clarkb | sure | 00:37 |
| fungi | some serious oom action up in here | 00:37 |
| fungi | java invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0 | 00:37 |
| clarkb | huh | 00:38 |
| fungi | i see one java process already running... did it try to start twice? | 00:38 |
| clarkb | oh maybe | 00:38 |
| clarkb | if it did that it would explain ooming since it mlockalls half the ram each time | 00:38 |
| fungi | yep | 00:38 |
| fungi | oh, and we have no swap on these things | 00:39 |
| clarkb | whcih is desirable | 00:39 |
| fungi | the launch script didn't do anything with our ephemeral disk | 00:39 |
| clarkb | and one reason for taking half the memory up front | 00:39 |
| clarkb | oh does launch script do the swap thing now? | 00:39 |
| fungi | maybe... checking | 00:40 |
| clarkb | there are a lot of no routes to host in the elasticsearch log, I don't think it can talk to the cluster right now | 00:40 |
| fungi | what size swap do we want on these. just mkswap the entire ephemeral disk or want partitions | 00:41 |
| fungi | ? | 00:41 |
| clarkb | I don't think we want swap | 00:41 |
| fungi | no? okay, no worrying about it in that case | 00:42 |
| *** chenxu_ has joined #openstack-infra | 00:42 | |
| clarkb | by mlocking all of the memory upfront we keep es under control and have ltos of memory for everything else | 00:42 |
| clarkb | and swap is pretty fatal to es | 00:42 |
| *** mriedem has joined #openstack-infra | 00:42 | |
| mspreitz | Ah, I was using the command line for cherry-picking via "Anonymous HTTP" rather than SSH or HTTPS | 00:42 |
| *** shashank_ has quit IRC | 00:42 | |
| fungi | clarkb: worth noting the memory utilization on es01 | 00:42 |
| mrmartin | fungi: thanks, it works now, I really going sleep :) | 00:43 |
| mrmartin | have a nice day | 00:43 |
| clarkb | fungi: that looks about right, lots of cached data | 00:43 |
| fungi | mrmartin: thanks again for working on it! | 00:43 |
| clarkb | fungi: on 02 I think we should stop the service and try starting it again cleanly | 00:43 |
| fungi | clarkb: oh, right, half of that is in cache. good | 00:43 |
| clarkb | I don't understand these no route to host errors | 00:43 |
| fungi | doing | 00:43 |
| mspreitz | When I try the command line for SSH, experimenting with .ssh/config shows that git fetch will succeed or fail depending on length of algorithm negotiation packet | 00:43 |
| clarkb | oh I know | 00:44 |
| clarkb | puppet isn't running on logstash worker16 | 00:44 |
| * clarkb fixes that | 00:44 | |
| *** mrmartin has quit IRC | 00:44 | |
| fungi | ahh, firewall update missing there ;) | 00:44 |
| mspreitz | Looks like https://github.com/net-ssh/net-ssh/issues/93 | 00:44 |
| *** bhuvan_ has quit IRC | 00:44 | |
| clarkb | fungi: and 02 has joined the cluster | 00:45 |
| clarkb | fungi: did you restart the service too? | 00:45 |
| fungi | clarkb: i did | 00:45 |
| fungi | mspreitz: where is ruby net-ssh involved in the situation? | 00:46 |
| *** dolphm_503 is now known as dolphm | 00:47 | |
| clarkb | fungi: 02 looks happy to me | 00:48 |
| fungi | clarkb: any ideas on the double-start and oom there? should we merge 76399 and see if es03 hits that on the first go? | 00:49 |
| clarkb | fungi: best guess is it might have been leftovers from the previous run? | 00:49 |
| clarkb | the puppet run that failed | 00:49 |
| fungi | that's all i've got :/ | 00:49 |
| fungi | i mean, the initscript really ought to be smart enough not to let that happen in the first place, right? | 00:50 |
| fungi | so weird | 00:50 |
| clarkb | I would hope so | 00:50 |
| *** UtahDave has quit IRC | 00:51 | |
| fungi | it makes a pidfile and does what looks like some fairly standard handling/checking for it | 00:51 |
| clarkb | is it possibly pushing that into es itself so it had to fork and do the mlock before it could notice there was another process going? | 00:52 |
| *** amcrn has joined #openstack-infra | 00:52 | |
| fungi | unless the initscript returns control before the pidfile exists, and the maintscripts and puppet service ensure raced one another | 00:52 |
| fungi | yeah, similar to what i'm thinking | 00:52 |
| fungi | if it's package install related, at least it's only a concern for first bootstrap | 00:53 |
| *** bhuvan has joined #openstack-infra | 00:53 | |
| *** shashank_ has joined #openstack-infra | 00:53 | |
| fungi | oh, though i guess on puppeted es package upgrades we might also run into it | 00:53 |
| clarkb | I have always done those by hand because you don't really want version mismatches | 00:54 |
| clarkb | so coordinated upgrades all at once is a good thing after having stopped all the nodes | 00:55 |
| fungi | approved 76399 which should merge prior to 01:00, at which point i'll enable puppet on es03 | 00:55 |
| clarkb | though I was thinking it wouldn't be terrible to orchestrate that if we could stop all the gearman workers, then do the massive upgrade | 00:55 |
| clarkb | but salt and all that need to happen first | 00:55 |
| fungi | i guess there's not enough redundancy to upgrade half the cluster offline and pivor | 00:56 |
| fungi | pivot | 00:56 |
| clarkb | right, we could do that by doubling the data we keep | 00:56 |
| openstackgerrit | A change was merged to openstack-infra/config: Puppet the /etc/elasticsearch directory https://review.openstack.org/76399 | 00:56 |
| clarkb | which doesn't seem worth it | 00:56 |
| clarkb | at least not yet | 00:56 |
| fungi | yeah, it's already a metric crapton of resources just to run what we're running | 00:57 |
| jeblair | PIVOT! http://www.youtube.com/watch?v=n67RYI_0sc0 | 00:57 |
| *** thuc has joined #openstack-infra | 00:57 | |
| *** Ryan_Lane has joined #openstack-infra | 00:57 | |
| Ryan_Lane | howdy | 00:57 |
| fungi | it's a wandering Ryan_Lane! | 00:57 |
| jeblair | Ryan_Lane: long time no see! | 00:58 |
| fungi | long time no irc ;) | 00:58 |
| Ryan_Lane | heh | 00:58 |
| *** atiwari has quit IRC | 00:58 | |
| Ryan_Lane | I've been avoiding a lot of channels since at the new job | 00:58 |
| *** lcheng has quit IRC | 00:58 | |
| *** bhuvan_ has joined #openstack-infra | 00:58 | |
| clarkb | jeblair: ha | 00:58 |
| fungi | clearly you forgot you should not avoid *this* channel | 00:58 |
| Ryan_Lane | I like to align communities though, so I'd like to pick some brains :) | 00:58 |
| Ryan_Lane | have you guys considered phabricator over gerrit? | 00:58 |
| fungi | oh boy yes | 00:59 |
| fungi | we did look it over at least | 00:59 |
| Ryan_Lane | it's come a *long* way since last time wikimedia looked at it | 00:59 |
| Ryan_Lane | to a point we're considering it again | 00:59 |
| clarkb | Ryan_Lane: there seemed to be a philosophical difference when we looked at it | 00:59 |
| jeblair | Ryan_Lane: the fact that it didn't actually do anything with git repos was kind of a big minus | 00:59 |
| clarkb | there is no enforcement of when things can merge and so on, its all manual behind the scenes | 00:59 |
| clarkb | that | 00:59 |
| Ryan_Lane | I think it does now | 00:59 |
| jeblair | Ryan_Lane: neato, maybe we should look again. | 01:00 |
| fungi | they've replaced hadn-waving with mechanical hands? | 01:00 |
| Ryan_Lane | that was my major hesitation too | 01:00 |
| Ryan_Lane | and lyft is considering it and I'd really like to use something that has zuul integration :D | 01:00 |
| *** wenlock has quit IRC | 01:01 | |
| Ryan_Lane | wikimedia will obviously need zuul integration too | 01:01 |
| *** bhuvan has quit IRC | 01:01 | |
| reed | Ryan_Lane, good to see you back here | 01:02 |
| clarkb | ooh I would defnitely be willing to look at it again | 01:02 |
| jeblair | Ryan_Lane: we'll definitely accept zuul patches for it (supporting non-gerrit is considered in-scope for zuul) even if we don't switch (ever or for a while) | 01:02 |
| Ryan_Lane | cool | 01:02 |
| fungi | non-gerrit, much like non-jenkins, means non-jave potentially | 01:02 |
| fungi | java | 01:02 |
| Ryan_Lane | reed: good to be back :) | 01:03 |
| Ryan_Lane | I'd really like to be able to share some CI effort at lyft as well, so I'll likely be around | 01:03 |
| *** bhuvan has joined #openstack-infra | 01:03 | |
| jeblair | Ryan_Lane: so are you at the "gap analysis" phase or more like the "let's poke at this" phase? | 01:03 |
| Ryan_Lane | it's installed in our labs environment and people are poking around at it | 01:03 |
| fungi | phabricator is written in php... guess i'll still have that to hold against it | 01:04 |
| Ryan_Lane | at lyft we're at the gap analysis phase | 01:04 |
| mspreitz | fungi: read all the way down, the problem is not the software it is something in the network and many kinds of software can trigger it | 01:04 |
| Ryan_Lane | it's easier to deal with php than java :) | 01:04 |
| jeblair | fungi: yeah, we can still gripe about the language. :) | 01:04 |
| Ryan_Lane | I wish it was python | 01:04 |
| *** bhuvan_ has quit IRC | 01:04 | |
| clarkb | well it comes from facebook, we should be happy ti speaks git | 01:04 |
| fungi | jeblair: works for me. i'd actually get work done if i didn't have language preferences to complain about | 01:04 |
| Ryan_Lane | hahaha. indeed | 01:04 |
| Ryan_Lane | it's not maintained by facebook anymore, though | 01:04 |
| Ryan_Lane | hasn't been for a couple years I think | 01:05 |
| reed | uh, my danger alert went off | 01:05 |
| lifeless | Ryan_Lane: what do they use now? | 01:05 |
| reed | facebook, php, unmaintained ... sirens off | 01:06 |
| clarkb | Ryan_Lane: right the maintainer left to do phabricator full time | 01:06 |
| clarkb | lifeless: they use hg | 01:06 |
| lifeless | clarkb: I meant other than phabrifcator | 01:06 |
| jeblair | Ryan_Lane: i'd love to hear about what you find | 01:07 |
| clarkb | jhesketh: https://review.openstack.org/#/c/68828/10/zuul/cmd/client.py I am doing proper review of that now. lines 165 to 167, does that mean last worker suffix value wins? | 01:07 |
| clarkb | jhesketh: is that intentional? | 01:07 |
| clarkb | jhesketh: oh wait I grok now nevermind | 01:07 |
| clarkb | lifeless: that I do not know | 01:07 |
| fungi | mspreitz: oh, interesting. this takes us back to the misbehaving (or malicious!) middleboxes theory | 01:08 |
| jhesketh | clarkb: it's to access sub-dicts using a period in the column name | 01:08 |
| *** sarob has quit IRC | 01:08 | |
| *** bhuvan_ has joined #openstack-infra | 01:08 | |
| clarkb | Ryan_Lane: there was a thread active today in the gerrit ml about gerrit critics | 01:08 |
| clarkb | Greg kh is one of them | 01:08 |
| clarkb | was interesting to read | 01:08 |
| *** sarob has joined #openstack-infra | 01:08 | |
| Ryan_Lane | yeah, but phabricator felll into the same issues | 01:09 |
| Ryan_Lane | in the article | 01:09 |
| reed | the website has some funny text :) http://phabricator.org/ | 01:09 |
| *** bhuvan has quit IRC | 01:09 | |
| Ryan_Lane | since it uses the same workflow | 01:09 |
| Ryan_Lane | jeblair: I'll let you know. | 01:09 |
| Ryan_Lane | I brought this up because I like to keep toolchains close :) | 01:10 |
| reed | Written in PHP so literally anyone can contribute, even if they have no idea how to program. | 01:10 |
| Ryan_Lane | hahaha | 01:10 |
| reed | their words not mine :) | 01:10 |
| fungi | reed: that's always been how i characterize php amyway | 01:10 |
| fungi | anyway | 01:10 |
| reed | You can make text bold. Bold text wins arguments! | 01:10 |
| *** oubiwann_ has joined #openstack-infra | 01:10 | |
| reed | they're funny :) | 01:10 |
| *** gokrokve_ has quit IRC | 01:10 | |
| anteaya | and all caps gets you escorted to the door | 01:11 |
| fungi | anteaya: and don't get me started on ansi blink escapes | 01:11 |
| anteaya | no | 01:11 |
| clarkb | jhesketh: I am feeling dense. how does worker_name which is in the default list match the worker.name key? | 01:11 |
| anteaya | no don't get started on anything blinky | 01:11 |
| clarkb | jhesketh: wouldn't that fail to match removing worker_name from fields? | 01:12 |
| jhesketh | clarkb: ah, it wouldn't.. that'd be an error from a previous patchset | 01:12 |
| clarkb | jhesketh: so I should leave a legit comment on that? | 01:12 |
| fungi | clarkb: so with the fix in place, es03 tried to double-start and oom'd too | 01:12 |
| jhesketh | yep | 01:12 |
| jhesketh | clarkb: so it was worker_name, and the job had 'worker_name', 'worker_ips' etc in the dict, but now it goes job['worker']['names'] | 01:13 |
| clarkb | fungi: weird I have no idea why that would happen | 01:13 |
| clarkb | fungi: oh you know what | 01:13 |
| *** sarob has quit IRC | 01:13 | |
| clarkb | fungi: is it double forking? | 01:13 |
| clarkb | fungi: we might want to set HEAP_SIZE to 29g if so | 01:13 |
| fungi | it... might be? i dunno, java and all | 01:13 |
| fungi | as long as it stays inside its jvm, it can fork itself all it likes | 01:14 |
| clarkb | jhesketh: I will finish reviewing this and leave comments | 01:14 |
| jhesketh | cheerws | 01:14 |
| jhesketh | *cheers | 01:14 |
| fungi | but yeah, i suppose leaving it a little room for briefly doubling its memory footprint, while somewhat insane from a memory utilization perspective, is certainly doable | 01:15 |
| fungi | clarkb: though if that were really the case, i'd expect to hit it any time we started the service, not just the first time around | 01:15 |
| *** zhiyan_ is now known as zhiyan | 01:16 | |
| clarkb | fungi: right and on the old nodes we set heap size to 16GB on 30GB nodes so should hit it every time | 01:16 |
| clarkb | so I really don't know what is going on | 01:16 |
| * fungi blames puppet some more, fir good measure | 01:16 | |
| clarkb | jesusaurus: ^ have you seen that with es? | 01:16 |
| clarkb | I will be going to an es + logstash thing early march in seattle maybe they will know there if we don't get it sorted before then | 01:16 |
| sdague | Ryan_Lane: so I'm looking at the phabricator site. I wonder how it would deal with openstack review load. They look like they are at 8k reviews total in their system, and I think that's currently about 5 weeks for us. | 01:17 |
| fungi | interestingly, at start it immediately gobbles as much cache in ram as it uses resident | 01:18 |
| fungi | is that just a javaism, or elasticsearch-specific behavior? | 01:18 |
| clarkb | I thin es specific | 01:18 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add storyboard and openstackid-dev ssl certs https://review.openstack.org/76407 | 01:18 |
| Ryan_Lane | sdague: it's used at facebook still | 01:19 |
| Ryan_Lane | and I'd imagine they have a lot more | 01:19 |
| *** reed has quit IRC | 01:19 | |
| sdague | sure, I wonder what their workflows look like | 01:19 |
| Ryan_Lane | yeah, me too | 01:20 |
| fungi | sdague: they post reviews on their wall | 01:20 |
| sdague | hah | 01:20 |
| * jeblair likes this | 01:20 | |
| * fungi has learnt some faceplant lingo | 01:20 | |
| clarkb | fungi: and just like our gerrit everonye can see them. there is no privacy :) | 01:20 |
| Ryan_Lane | I've heard their deployer actually has a dislike button | 01:20 |
| jeblair | actually i don't like it but there's no button for that | 01:20 |
| sdague | jeblair: you really think the internet needs more drive by hate? | 01:20 |
| Ryan_Lane | (and I'm not kidding ;) ) | 01:21 |
| jeblair | sdague: only being able to like things is double plus good | 01:21 |
| *** fifieldt has joined #openstack-infra | 01:21 | |
| fungi | some source code needs more drive-by hate | 01:21 |
| clarkb | jhesketh: and the formatJSON() move was just copy pasta for the most part? | 01:21 |
| *** ryanpetrello has joined #openstack-infra | 01:21 | |
| clarkb | I wish gerrit would let me align diffs manually sometimes | 01:21 |
| * anteaya thinks sdague makes a good point | 01:21 | |
| clarkb | sdague: downvotes everywhere | 01:22 |
| sdague | I save that for gerrit :) | 01:22 |
| fungi | clarkb is cmdrtaco in disguise | 01:22 |
| sdague | Ryan_Lane: it would be really interesting to see if you can make a sane integration here. Because as much as I have issues with gerrit, I also look at what | 01:22 |
| jhesketh | clarkb: it's similar but a lot more information has been added into the jobs list to better reflect the data model | 01:22 |
| sdague | 's in the public phabricator, and my gerrit workflow, and can't imagine getting that interface to handle my workflow | 01:23 |
| Ryan_Lane | I think pre-merge workflow is similar | 01:23 |
| Ryan_Lane | what's different about it? | 01:23 |
| sdague | the review view | 01:23 |
| Ryan_Lane | ah | 01:24 |
| Ryan_Lane | anything specific missing? | 01:24 |
| mspreitz | New question: how do I make git tell me where my FETCH_HEAD is pointing? | 01:25 |
| fungi | mspreitz: git show | 01:25 |
| jeblair | git show FETCH_HEAD | 01:25 |
| fungi | that ^ | 01:25 |
| mspreitz | $ git show FETCH_HEAD fatal: ambiguous argument 'FETCH_HEAD': unknown revision or path not in the working tree. | 01:26 |
| fungi | or you can git checkout FETCH_HEAD and then just git show | 01:26 |
| fungi | mspreitz: is that before or after a fetch? | 01:26 |
| mspreitz | after a fetch | 01:26 |
| fungi | strange. works for me when i try it | 01:27 |
| sdague | Ryan_Lane: so views like this I use a lot - http://i.imgur.com/aRERaXd.png | 01:27 |
| *** zhiwei has joined #openstack-infra | 01:27 | |
| jeblair | cat .git/FETCH_HEAD | 01:27 |
| sdague | it being screen dense, having some idea on current votes, and who voted for stuff | 01:28 |
| clarkb | jhesketh: the worker_name thing was the only thing I found. looks good otherwise | 01:28 |
| *** oubiwann_ has quit IRC | 01:28 | |
| Ryan_Lane | sdague: ahhh, ok | 01:28 |
| jhesketh | thanks clarkb, I'll reroll | 01:28 |
| clarkb | I need to eat dinner shortly but can rereview if you push up a fix | 01:28 |
| mspreitz | I have git version 1.8.3.2 on MacOSX | 01:28 |
| Ryan_Lane | interesting | 01:28 |
| Ryan_Lane | wikimedia doesn't really have that use case as much | 01:29 |
| fungi | clarkb: should we proceed puppeting and restarting services on the other half of the new cluster members, or try more options i can't seem to think of? | 01:29 |
| mspreitz | sorry, I was reading wrong terminal | 01:30 |
| clarkb | fungi: can you try doing a puppet agent --test in order to get verbose output and see if puppet is kicking it more than once? | 01:30 |
| mspreitz | I have git version 1.8.3.2 on Ubuntu saucy | 01:30 |
| clarkb | fungi: I just relized there may be a subscribe somewhere that does something that triggers it twice | 01:30 |
| *** tomhe has quit IRC | 01:30 | |
| *** mrodden has quit IRC | 01:30 | |
| sdague | Ryan_Lane: how many reviews does a person tend to do over some unit time at wikimedia? | 01:30 |
| fungi | clarkb: sure, i'll do that and record the output into a (redacted if necessary) paste | 01:30 |
| clarkb | fungi: and puppet being at fault would explain why it didn't happen before considering puppet order isn't garunteed and determined by the catalog which we just cahnged | 01:30 |
| Ryan_Lane | no clue. they track that poorly | 01:30 |
| *** tjones has quit IRC | 01:30 | |
| mspreitz | git checkout FETCH_HEAD also craps out for me | 01:30 |
| mspreitz | like this: error: pathspec 'FETCH_HEAD' did not match any file(s) known to git | 01:31 |
| *** zul has quit IRC | 01:31 | |
| sdague | so http://stackalytics.com/?release=icehouse&metric=marks&project_type=openstack&module=&company=&user_id= | 01:31 |
| sdague | we'll have a dozen people in the openstack community with > 1000 reviews during icehouse on current course and speed | 01:31 |
| * Ryan_Lane nods | 01:31 | |
| fungi | yarg | 01:32 |
| *** apevec has quit IRC | 01:32 | |
| *** bhuvan has joined #openstack-infra | 01:32 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/zuul: Add support to list running jobs to zuul client https://review.openstack.org/68828 | 01:32 |
| fungi | how am i in the top 10 contributions by engineers? | 01:32 |
| fungi | though kudos to ajaeger! | 01:32 |
| anteaya | is jenkins in there? | 01:32 |
| jhesketh | clarkb: ^ (whenever :-)) | 01:32 |
| *** lnxnut has quit IRC | 01:33 | |
| sdague | yeh, looks like 13 in havana, so realistically I bet we end up with 15 in icehouse (I bet anyone with > 750 now will hit 1000 by end of cycle) | 01:33 |
| sdague | anteaya: no, this is reviews :) | 01:33 |
| anteaya | sdague: ah | 01:33 |
| Ryan_Lane | I wonder if it's a matter of having reasonable queries | 01:33 |
| fungi | i need to find another 75 patches to review asap ;) | 01:34 |
| clarkb | jeblair: if jhesketh's change still lg to you we can rpobably double tab it and get it merged | 01:34 |
| anteaya | fungi: like you have to look | 01:34 |
| sdague | Ryan_Lane: reasonable queries is definitely key, honestly gerrit only partially works that way. Having the queries as get strings so you can bookmark them is goodness as well. | 01:34 |
| fungi | anteaya: good point. my present watched backlog is at least triple that i think | 01:34 |
| Ryan_Lane | looks like phabricator can save queries | 01:35 |
| clarkb | fungi: you jeblair and I are | 01:35 |
| anteaya | no doubt | 01:35 |
| mspreitz | Oh, I see. I was discounting the "git fetch" that failed. If I try again right after a "git fetch" that succeeds then "git show FETCH_HEAD" works | 01:35 |
| sdague | more fun stat | 01:35 |
| fungi | mspreitz: that does make sense | 01:35 |
| *** jergerber has quit IRC | 01:35 | |
| sdague | hit the last page on stackalytics | 01:35 |
| *** Guest63393 has quit IRC | 01:35 | |
| *** Guest63393 has joined #openstack-infra | 01:35 | |
| fungi | clarkb: you're beating me by at least several reviews though | 01:35 |
| *** Guest63393 is now known as persia | 01:35 | |
| sdague | 1011 - the number of individuals that have reviewed code in icehouse | 01:35 |
| *** thuc has quit IRC | 01:35 | |
| *** krotscheck has quit IRC | 01:35 | |
| *** bhuvan_ has quit IRC | 01:35 | |
| jesusaurus | clarkb: fungi: im confused, es03 tried to start two separate nodes on the same host? | 01:36 |
| *** thuc has joined #openstack-infra | 01:36 | |
| Ryan_Lane | sdague: yep. lots of reviewers | 01:36 |
| fungi | jesusaurus: it did indeed. still digging into it | 01:36 |
| fungi | jesusaurus: it's also repeatable | 01:36 |
| *** bhuvan has quit IRC | 01:36 | |
| jesusaurus | fungi: what are the steps to reproduce? | 01:36 |
| clarkb | jesusaurus: ya not sure if init script/es fault or puppet double tapping weird | 01:37 |
| fungi | jesusaurus: use out puppet, and build a new cluster member? | 01:37 |
| fungi | s/out/our/ | 01:37 |
| Ryan_Lane | oh, wow. phabricator's task manager works across projects: https://secure.phabricator.com/T3089 | 01:37 |
| Ryan_Lane | isn't that the reason launchpad is still being used for issues? | 01:37 |
| anteaya | we are looking for the twelfth reviewer | 01:38 |
| * Ryan_Lane hates launchpad | 01:38 | |
| fungi | Ryan_Lane: that was one of our major requirements for storyboard, so good to know we're not insane for wanting it | 01:38 |
| Ryan_Lane | it's a feature of jira too | 01:38 |
| Ryan_Lane | so it's not insane at all | 01:38 |
| jeblair | Ryan_Lane: mordred and ttx can probably provide more details, but there was still some fundamental misalignments between phabricator and what we need | 01:39 |
| jeblair | for bugs | 01:39 |
| jeblair | and blueprints | 01:39 |
| Ryan_Lane | yep, not surprising | 01:39 |
| *** sabari has quit IRC | 01:39 | |
| Ryan_Lane | I'm just going through the features now and seeing some nice things :) | 01:39 |
| jeblair | Ryan_Lane: yeah, we looked hard at it for bugs | 01:39 |
| *** gokrokve has joined #openstack-infra | 01:39 | |
| *** thuc has quit IRC | 01:40 | |
| jesusaurus | clarkb: to answer your question, there was one lone incident where i found a second node running on 9201/9301, but i simply killed the second process and never saw any other strange behaviour | 01:40 |
| fungi | clarkb: jesusaurus: http://paste.openstack.org/show/69633 | 01:41 |
| *** yamahata has joined #openstack-infra | 01:41 | |
| *** yamahata has quit IRC | 01:41 | |
| *** nosnos has joined #openstack-infra | 01:41 | |
| *** yamahata has joined #openstack-infra | 01:42 | |
| fungi | clarkb: jesusaurus: and that puppet agent --test did indeed wind up trying to start a second jvm, which then got oom-killed | 01:42 |
| fungi | and following a service stop/start, it joined the cluster normally and successfully | 01:42 |
| clarkb | so new theory. package install starts it then service double taps | 01:44 |
| jesusaurus | fungi: before that --test run es04 already had elasticsearch installed and running? | 01:44 |
| fungi | clarkb: jesusaurus: staring at that log, i'm back to my original supposition, which is that the initscript returns control to the calling process before the pidfile is created | 01:44 |
| jesusaurus | clarkb: oh, thats a possibility | 01:44 |
| clarkb | fungi: can you try a test change that doesnt manage the service? | 01:44 |
| fungi | jesusaurus: before that the elasticsearch puppet module was not applied at all | 01:44 |
| fungi | clarkb: doing now | 01:45 |
| clarkb | fungi: that seems to be a general direction we are headed in anyways | 01:45 |
| *** esker has joined #openstack-infra | 01:45 | |
| *** apevec has joined #openstack-infra | 01:46 | |
| sdague | I do like the fact that phabricator does all the diffs in one page | 01:46 |
| *** apevec has quit IRC | 01:46 | |
| *** apevec has joined #openstack-infra | 01:46 | |
| Ryan_Lane | sdague: same. I've wanted that in gerrit since forever | 01:47 |
| Ryan_Lane | well, phabricator has stolen a couple hours of my day now. back to the grind for me ;) | 01:48 |
| *** amcrn has quit IRC | 01:49 | |
| clarkb | woot /me has hibiki 12 and code reviews to do now | 01:49 |
| jeblair | sdague: you know someone wrote a patch to gerrit for that, but didn't sign the cla, so it was never merged. | 01:49 |
| jeblair | (CLAs are evil) | 01:49 |
| clarkb | jeblair: did you catch jhesketh's updated patch? the delta between ps 10 and 11 is mall | 01:49 |
| jeblair | clarkb: is it urgent? | 01:50 |
| jhesketh | nope | 01:50 |
| jhesketh | (the real delta is one character, just rebased while I had the chance) | 01:50 |
| clarkb | jhesketh: no | 01:50 |
| clarkb | gah | 01:50 |
| clarkb | jeblair: no not urgent at all | 01:50 |
| *** ryanpetrello has quit IRC | 01:52 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Don't ensure elasticsearch service is running https://review.openstack.org/76412 | 01:53 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch workers https://review.openstack.org/76051 | 01:53 |
| fungi | clarkb: as suspected, https://review.openstack.org/76412 there seems to solve it | 01:53 |
| fungi | so my first theory was probably correct | 01:53 |
| clarkb | ya I think so | 01:53 |
| sdague | jeblair: sigh | 01:53 |
| fungi | clarkb: if we merge that, i'll puppet es06 from master with it, make sure that is sane, and call it a day | 01:54 |
| sdague | I wonder if we are ever going to get rid of our CLA, or if that will be the board fight to end all board fights | 01:54 |
| fungi | sdague: the tc should fire the board | 01:54 |
| fungi | ;) | 01:54 |
| clarkb | fungi: I +2'd it. Looks relatively safe. I did not however check to see if there were subscriptions to the service or notifies to it | 01:54 |
| clarkb | sdague: you could fork openstack and have the fork be CLA free | 01:55 |
| fungi | clarkb: well, i did apply the same change from a puppet dev env on es05 and it didn't error. i think a subscrive/refresh would have counted as a reference to a nonexistent service and errored in that case | 01:55 |
| clarkb | fungi: maybe | 01:56 |
| * clarkb looks closer | 01:56 | |
| *** ryanpetrello has joined #openstack-infra | 01:56 | |
| fungi | clarkb: if there's any mention of a service, it's not in the init.pp anyway (which is the only puppetfile in the module) | 01:57 |
| clarkb | fungi: git grep "'elasticsearch'" doesn't show any incidences either | 01:57 |
| clarkb | so I think your change is fine as is | 01:57 |
| *** khyati has quit IRC | 01:57 | |
| fungi | into the drink with it, i say | 01:57 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't ensure elasticsearch service is running https://review.openstack.org/76412 | 01:59 |
| *** gyee has quit IRC | 02:00 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerritlib: add getVersion and listPlugin commands and update replicate command https://review.openstack.org/69768 | 02:02 |
| *** ryanpetrello has quit IRC | 02:02 | |
| *** gyee has joined #openstack-infra | 02:03 | |
| *** gyee has quit IRC | 02:04 | |
| fungi | tomorrow i'll try to remember to get the new elasticsearch cluster members and the puppetdb server into cacti too | 02:04 |
| clarkb | zaro: ^ failed pep8 | 02:05 |
| *** gyee has joined #openstack-infra | 02:05 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerritlib: add getVersion and listPlugin commands and update replicate command https://review.openstack.org/69768 | 02:05 |
| clarkb | zaro: whats with ascii_names? | 02:06 |
| *** mgagne has quit IRC | 02:08 | |
| *** krotscheck has joined #openstack-infra | 02:08 | |
| *** chenxu_ has quit IRC | 02:09 | |
| fungi | clarkb: with that last patch merged, enabling puppet agent on es06 cause it to install, start and join the cluster successfully | 02:10 |
| fungi | i'll check back in on it in the morning to see how shard replication is coming along | 02:10 |
| clarkb | perfect thanks again | 02:11 |
| clarkb | I am watching a movie now | 02:11 |
| fungi | es01 has a dozen shards at this point, and es02 has a couple as well | 02:11 |
| fungi | clarkb: great--i'm going to do the same. have a good evening | 02:12 |
| clarkb | you too | 02:12 |
| *** arborism has joined #openstack-infra | 02:12 | |
| *** arborism is now known as amcrn | 02:12 | |
| *** Sukhdev has joined #openstack-infra | 02:15 | |
| *** changbl has joined #openstack-infra | 02:19 | |
| *** mspreitz has left #openstack-infra | 02:25 | |
| *** yaguang has joined #openstack-infra | 02:25 | |
| *** esker has quit IRC | 02:26 | |
| *** SumitNaiksatam has quit IRC | 02:28 | |
| *** thuc has joined #openstack-infra | 02:29 | |
| openstackgerrit | A change was merged to openstack/requirements: Bump python-savannaclient to 0.5.0 https://review.openstack.org/76357 | 02:29 |
| *** thuc_ has joined #openstack-infra | 02:29 | |
| *** thomasem has joined #openstack-infra | 02:29 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/gerritlib: add getVersion and listPlugin commands and update replicate command https://review.openstack.org/69768 | 02:31 |
| *** amcrn has quit IRC | 02:32 | |
| *** thuc has quit IRC | 02:33 | |
| *** mgagne has joined #openstack-infra | 02:35 | |
| *** apevec has quit IRC | 02:36 | |
| zaro | clarkb: ^ | 02:36 |
| *** mgagne has quit IRC | 02:38 | |
| zaro | clarkb: caught another 3 episodes of game of thrones on plane. pretty good. | 02:38 |
| *** shashank_ has quit IRC | 02:40 | |
| *** Ryan_Lane has quit IRC | 02:48 | |
| *** thomasem has quit IRC | 02:50 | |
| *** david-lyle has joined #openstack-infra | 02:52 | |
| *** yaguang has quit IRC | 02:52 | |
| *** Ryan_Lane has joined #openstack-infra | 02:55 | |
| *** yaguang has joined #openstack-infra | 02:56 | |
| *** sabari has joined #openstack-infra | 02:57 | |
| *** mspreitz has joined #openstack-infra | 03:00 | |
| mspreitz | Does anybody here test with Python 2.6 on ubuntu? | 03:00 |
| mspreitz | Is anybody here? | 03:01 |
| fifieldt | probably | 03:01 |
| pleia2 | it's tough because none of the versions of Ubuntu that can a) run modern openstack can b) easily run python 2.6, the last release that met this criteria was 11.10/oneiric (now EOL) | 03:04 |
| pleia2 | which is why we use centos to test 2.6 in the infra now | 03:04 |
| mspreitz | Ah, thanks | 03:05 |
| mspreitz | So developers normally use Ubuntu 12.04 and test with only Python 2.7, right? | 03:06 |
| pleia2 | we also have 3.3 packages for 12.04 that we test some things on, but yeah, 2.7 is the status quo | 03:06 |
| mspreitz | thanks | 03:07 |
| pleia2 | sure | 03:07 |
| *** zhiwei has quit IRC | 03:11 | |
| *** Sukhdev has quit IRC | 03:12 | |
| *** amotoki_ has quit IRC | 03:13 | |
| *** morganfainberg is now known as morganfainberg_Z | 03:19 | |
| *** krotscheck has quit IRC | 03:19 | |
| *** Ryan_Lane has quit IRC | 03:23 | |
| *** jcooley_ has quit IRC | 03:23 | |
| ianw | mspreitz: python2.6 is used on RHEL6 | 03:24 |
| mspreitz | But most developers just use Ubuntu 12.04, right? | 03:25 |
| ianw | mspreitz: hard to say, fedora is popular too | 03:26 |
| *** jcooley_ has joined #openstack-infra | 03:27 | |
| *** sabari has quit IRC | 03:27 | |
| mspreitz | Does Fedora have Python 2.6? | 03:27 |
| *** cody-somerville has joined #openstack-infra | 03:29 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 03:29 | |
| ianw | no, RHEL/Centos 6 is the only python2.6 environment | 03:30 |
| *** CaptTofu has quit IRC | 03:35 | |
| *** pcrews has quit IRC | 03:35 | |
| *** nati_ueno has quit IRC | 03:37 | |
| *** matsuhashi has quit IRC | 03:37 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 03:39 | |
| *** rcleere has joined #openstack-infra | 03:40 | |
| *** rfolco has quit IRC | 03:41 | |
| *** thuc_ has quit IRC | 03:41 | |
| *** thuc has joined #openstack-infra | 03:42 | |
| *** khyati has joined #openstack-infra | 03:51 | |
| *** gyee has quit IRC | 03:52 | |
| *** mriedem has quit IRC | 03:55 | |
| *** markwash has quit IRC | 03:56 | |
| *** harlowja is now known as harlowja_away | 03:57 | |
| *** sarob has joined #openstack-infra | 04:00 | |
| *** sarob has quit IRC | 04:08 | |
| *** ArxCruz has quit IRC | 04:12 | |
| *** ArxCruz has joined #openstack-infra | 04:19 | |
| *** ArxCruz has quit IRC | 04:20 | |
| *** sabari has joined #openstack-infra | 04:23 | |
| *** Ryan_Lane has joined #openstack-infra | 04:24 | |
| *** wenlock has joined #openstack-infra | 04:27 | |
| *** jcooley_ has quit IRC | 04:27 | |
| *** ArxCruz has joined #openstack-infra | 04:31 | |
| *** ArxCruz has quit IRC | 04:31 | |
| *** matsuhashi has joined #openstack-infra | 04:32 | |
| jhesketh | jeblair, clarkb: when you guys get a chance, could you please respond to my comments on this: https://review.openstack.org/#/c/73461/ | 04:33 |
| *** ArxCruz has joined #openstack-infra | 04:34 | |
| *** ArxCruz has quit IRC | 04:35 | |
| *** lcheng has joined #openstack-infra | 04:37 | |
| clarkb | done | 04:38 |
| *** sarob has joined #openstack-infra | 04:40 | |
| *** sarob has quit IRC | 04:43 | |
| *** masayukig has joined #openstack-infra | 04:45 | |
| *** ArxCruz has joined #openstack-infra | 04:50 | |
| *** ArxCruz has quit IRC | 04:50 | |
| *** ryanpetrello has joined #openstack-infra | 04:52 | |
| *** jcooley_ has joined #openstack-infra | 04:55 | |
| *** boris-42 has quit IRC | 04:57 | |
| *** boris-42 has joined #openstack-infra | 04:59 | |
| *** jcooley_ has quit IRC | 05:00 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 05:00 | |
| *** masayukig has quit IRC | 05:01 | |
| *** jcooley_ has joined #openstack-infra | 05:01 | |
| *** ArxCruz has joined #openstack-infra | 05:04 | |
| *** ArxCruz has quit IRC | 05:05 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 05:06 | |
| jhesketh | thanks clarkb | 05:06 |
| *** chandan_kumar has joined #openstack-infra | 05:07 | |
| openstackgerrit | Joshua Hesketh proposed a change to openstack-infra/config: Set up opportunistic bare-metal postgres db https://review.openstack.org/73461 | 05:09 |
| *** coolsvap has joined #openstack-infra | 05:11 | |
| *** jcooley_ has quit IRC | 05:12 | |
| *** vogxn has joined #openstack-infra | 05:12 | |
| *** ryanpetrello has quit IRC | 05:18 | |
| *** jcooley_ has joined #openstack-infra | 05:18 | |
| *** sarob has joined #openstack-infra | 05:18 | |
| *** ArxCruz has joined #openstack-infra | 05:19 | |
| *** ArxCruz has quit IRC | 05:20 | |
| *** jcooley_ has quit IRC | 05:21 | |
| *** vogxn has quit IRC | 05:24 | |
| *** khyati has quit IRC | 05:28 | |
| *** gokrokve has quit IRC | 05:31 | |
| *** gokrokve has joined #openstack-infra | 05:31 | |
| *** talluri has joined #openstack-infra | 05:33 | |
| *** ArxCruz has joined #openstack-infra | 05:34 | |
| *** ArxCruz has quit IRC | 05:35 | |
| *** gokrokve has quit IRC | 05:36 | |
| *** CaptTofu has joined #openstack-infra | 05:36 | |
| *** nicedice has quit IRC | 05:36 | |
| *** gokrokve has joined #openstack-infra | 05:38 | |
| *** CaptTofu has quit IRC | 05:41 | |
| *** dolphm is now known as dolphm_503 | 05:41 | |
| *** dkliban has quit IRC | 05:43 | |
| *** sarob has quit IRC | 05:47 | |
| *** sdake_ has quit IRC | 05:47 | |
| *** ArxCruz has joined #openstack-infra | 05:49 | |
| *** ArxCruz has quit IRC | 05:50 | |
| *** nati_ueno has joined #openstack-infra | 05:50 | |
| *** talluri has quit IRC | 05:53 | |
| *** vogxn has joined #openstack-infra | 05:54 | |
| *** zhiyan is now known as zhiyan_ | 05:56 | |
| *** vogxn has quit IRC | 06:00 | |
| *** talluri has joined #openstack-infra | 06:03 | |
| *** yolanda_ has joined #openstack-infra | 06:07 | |
| *** talluri has quit IRC | 06:09 | |
| *** vkozhukalov_ has joined #openstack-infra | 06:10 | |
| *** dolphm_503 is now known as dolphm | 06:11 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Add oslo.vmware https://review.openstack.org/75555 | 06:12 |
| *** vkozhukalov_ has quit IRC | 06:16 | |
| *** jcooley_ has joined #openstack-infra | 06:19 | |
| *** gokrokve has quit IRC | 06:20 | |
| *** gokrokve has joined #openstack-infra | 06:20 | |
| *** wenlock has quit IRC | 06:20 | |
| *** gokrokve has quit IRC | 06:21 | |
| *** dolphm is now known as dolphm_503 | 06:23 | |
| *** talluri has joined #openstack-infra | 06:23 | |
| *** rwsu has quit IRC | 06:27 | |
| *** rlandy has joined #openstack-infra | 06:32 | |
| *** jcooley_ has quit IRC | 06:37 | |
| *** wchrisj has quit IRC | 06:41 | |
| jhesketh | any grenade devs in here able to help me with a problem? | 06:42 |
| *** lcheng has quit IRC | 06:42 | |
| mordred | Ryan_Lane: yeah - it doesn't do bug tasks though (which is why we're still on launchpad and writing storyboard) - and its integrated code review makes it hard for us to use it integrated with gerrit | 06:46 |
| mordred | Ryan_Lane: but yea - I found it very tempting when looking at it and I tried to make it make sense to use it | 06:46 |
| *** pblaho has joined #openstack-infra | 06:47 | |
| *** denis_makogon has joined #openstack-infra | 06:47 | |
| Ryan_Lane | mordred: well, the idea would be to move away from gerrit completely | 06:47 |
| Ryan_Lane | the code review in phabricator looks quite a bit better | 06:47 |
| *** talluri_ has joined #openstack-infra | 06:47 | |
| *** rcarrillocruz has joined #openstack-infra | 06:48 | |
| *** flaper87|afk is now known as flaper87 | 06:48 | |
| *** sarob has joined #openstack-infra | 06:48 | |
| *** rcarrillocruz1 has quit IRC | 06:50 | |
| *** saju_m has joined #openstack-infra | 06:50 | |
| clarkb | phabricator doesn't host repositories, what does that mean? | 06:50 |
| clarkb | er I guess it is beta. Does that mean the merge in phabricator is beta? | 06:50 |
| *** talluri has quit IRC | 06:50 | |
| *** ildikov_ has quit IRC | 06:51 | |
| *** valentinbud has joined #openstack-infra | 06:51 | |
| *** sarob has quit IRC | 06:53 | |
| *** thuc has quit IRC | 06:55 | |
| *** thuc has joined #openstack-infra | 06:55 | |
| *** dolphm_503 is now known as dolphm | 06:56 | |
| *** mspreitz has quit IRC | 06:58 | |
| clarkb | finally figured out where phabricator self hosts, there is a land revision to hosted repository button | 06:59 |
| clarkb | so it does seem to have grown that possibly as a beta feature | 06:59 |
| *** thuc has quit IRC | 07:00 | |
| Ryan_Lane | clarkb: yep | 07:05 |
| Ryan_Lane | I found that out recently | 07:05 |
| Ryan_Lane | which is what made me reconsider it | 07:05 |
| *** dolphm is now known as dolphm_503 | 07:05 | |
| *** rcarrillocruz1 has joined #openstack-infra | 07:06 | |
| *** sdake_ has joined #openstack-infra | 07:06 | |
| *** rcarrillocruz has quit IRC | 07:07 | |
| *** rcarrillocruz has joined #openstack-infra | 07:10 | |
| *** rcarrillocruz1 has quit IRC | 07:12 | |
| *** jcooley_ has joined #openstack-infra | 07:15 | |
| *** sld has joined #openstack-infra | 07:20 | |
| *** gokrokve has joined #openstack-infra | 07:21 | |
| *** jcooley_ has quit IRC | 07:21 | |
| *** nati_ueno has quit IRC | 07:24 | |
| *** jlibosva has joined #openstack-infra | 07:25 | |
| *** yolanda_ has quit IRC | 07:26 | |
| *** gokrokve has quit IRC | 07:26 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 07:27 | |
| *** skraynev_afk is now known as skraynev | 07:32 | |
| *** sabari has quit IRC | 07:35 | |
| *** CaptTofu has joined #openstack-infra | 07:36 | |
| *** jcoufal has joined #openstack-infra | 07:38 | |
| *** CaptTofu has quit IRC | 07:41 | |
| *** sld has left #openstack-infra | 07:42 | |
| *** vkozhukalov_ has joined #openstack-infra | 07:48 | |
| *** sarob has joined #openstack-infra | 07:50 | |
| *** sarob has quit IRC | 07:54 | |
| *** luqas has joined #openstack-infra | 07:56 | |
| *** dolphm_503 is now known as dolphm | 07:56 | |
| *** shashank_ has joined #openstack-infra | 07:57 | |
| *** e0ne has joined #openstack-infra | 07:58 | |
| *** afazekas has joined #openstack-infra | 08:05 | |
| *** luqas has quit IRC | 08:06 | |
| *** dolphm is now known as dolphm_503 | 08:06 | |
| *** luqas has joined #openstack-infra | 08:08 | |
| *** ildikov_ has joined #openstack-infra | 08:16 | |
| mordred | Ryan_Lane: sure. but that would be a bit of a large undertaking ... | 08:18 |
| Ryan_Lane | totally agree :) | 08:19 |
| mordred | Ryan_Lane: we'd need to add an event stream so that zuul could do its thing | 08:19 |
| Ryan_Lane | yep. wikimedia is considering phabricator again | 08:19 |
| Ryan_Lane | so some of this would be needed by them too | 08:19 |
| mordred | ah. they don't want to use storyboard from us? | 08:19 |
| Ryan_Lane | they use bugzilla | 08:19 |
| Ryan_Lane | they don't really need a bug tracker | 08:19 |
| mordred | oh | 08:19 |
| Ryan_Lane | they want to replace gerrit | 08:19 |
| mordred | so why would they want to move to phabricator? | 08:20 |
| mordred | ah | 08:20 |
| mordred | well, that would make me sad | 08:20 |
| *** jgallard has joined #openstack-infra | 08:20 | |
| Ryan_Lane | gerrit has a pretty unresponsive upstream | 08:20 |
| Ryan_Lane | it's UX is crap | 08:20 |
| Ryan_Lane | its* | 08:20 |
| mordred | we have some thoughts on UX for gerrit, fwiw | 08:20 |
| Ryan_Lane | its UX is consistently getting worse | 08:20 |
| mordred | which don't involve upstream patches | 08:20 |
| Ryan_Lane | have you seen the new change page? | 08:20 |
| mordred | yes | 08:20 |
| Ryan_Lane | the only thing gerrit does really well is handle the repos | 08:21 |
| mordred | right. but that's the hard part | 08:21 |
| Ryan_Lane | well, and its gating model | 08:21 |
| mordred | a UI can be added | 08:21 |
| Ryan_Lane | phabricator now handles repos | 08:21 |
| mordred | on top of its APIs | 08:21 |
| Ryan_Lane | gerrits public APIs kind of suck :( | 08:21 |
| mordred | the ssh api is actually quite amazing | 08:22 |
| Ryan_Lane | yeah, the ssh api isn't bad, but I wouldn't build a UI on it | 08:22 |
| mordred | in that it allows the effects of event/callback without the receiving end needingto be pulic | 08:22 |
| *** gokrokve has joined #openstack-infra | 08:22 | |
| mordred | like, from the event stream perspective - this makes the "run a 3rd party testing" thing work really well | 08:22 |
| Ryan_Lane | I honestly think moving away from gerrit is such a major undertaking that wikimedia may not do it | 08:22 |
| Ryan_Lane | but at some point something's got to change | 08:23 |
| mordred | yeah. it's a BIG amount of work | 08:23 |
| mordred | yeah | 08:23 |
| mordred | well, we're with you on that :) | 08:23 |
| Ryan_Lane | gerrit either needs to improve, or it's got to go | 08:23 |
| mordred | I'm basically being me right now because we agree and would love to work together on that | 08:23 |
| Ryan_Lane | heh | 08:23 |
| mordred | from my perspective, I'd like to replace teh UI first, which would mean that gerrit would be down to being the repo engine | 08:24 |
| Ryan_Lane | I think wikimedia is likely to investigate it for a while | 08:24 |
| *** denis_makogon has quit IRC | 08:24 | |
| mordred | then replacing the repo engine would be much easier | 08:24 |
| Ryan_Lane | yeah, but at that point, why not use phabricator? | 08:24 |
| mordred | because we can do it piecemeal | 08:24 |
| mordred | like we've been working towards replacing jenkins | 08:24 |
| mordred | even while adding jenkins features | 08:24 |
| mordred | because we've gota large production deploy | 08:24 |
| Ryan_Lane | ah, so you'd switch out the ui, then you'd switch to phabricator? | 08:25 |
| mordred | we can't just replace a large porition all in one go | 08:25 |
| mordred | no | 08:25 |
| Ryan_Lane | then you'd switch the ui again? | 08:25 |
| mordred | I wouldn't switch to phabricator at that point | 08:25 |
| Ryan_Lane | what would manage the repos? | 08:25 |
| Ryan_Lane | I think phabricator is doing an amazing job with the UI work | 08:25 |
| mordred | that's what I'm saying - at that point, what we'd need is an api service that can manage repos | 08:25 |
| mordred | sure they are - but they're also a big thing adn I don't need most of what they're doing | 08:26 |
| mordred | it's too hard | 08:26 |
| mordred | if we were starting from scratch today, sure | 08:26 |
| * Ryan_Lane nods | 08:26 | |
| mordred | but they're pitching a whole worldview | 08:26 |
| mordred | which is great | 08:26 |
| Ryan_Lane | so basically write something new from scratch? | 08:26 |
| mordred | it's what we're doing too | 08:26 |
| mordred | yeah | 08:26 |
| mordred | all we'd need is an api server | 08:26 |
| Ryan_Lane | that scares me ;) | 08:26 |
| mordred | we're good at writing those around here | 08:26 |
| Ryan_Lane | an api server that would manage repos? | 08:27 |
| mordred | yeah | 08:27 |
| *** gokrokve has quit IRC | 08:27 | |
| Ryan_Lane | well, it would surely be nice to have a python alternative | 08:27 |
| mordred | or, a scale-out system that can manage repos across multiple hosts - gerrit it too monolithic | 08:27 |
| mordred | is | 08:27 |
| Ryan_Lane | I just worry that it would take ages to reach feature parity with gerrit or phabricator | 08:27 |
| mordred | I agree | 08:27 |
| mordred | the question is - which features do we actually need | 08:27 |
| Ryan_Lane | ah. so... I've been thinking about scaling git across systems... | 08:28 |
| mordred | and/or which ones can we do away with if we're not working on a generalized solution that needs a bunch of customizability | 08:28 |
| mordred | yeah | 08:28 |
| mordred | ? | 08:28 |
| Ryan_Lane | but I'll just wait till I have something implemented till I discuss it :) | 08:28 |
| mordred | ok. I'm in favor of thoughts on scaling git | 08:28 |
| mordred | Ryan_Lane: have you been following the recent zuul merger work? | 08:29 |
| Ryan_Lane | nope | 08:29 |
| Ryan_Lane | but I have a deployment system using salt that can easily be extended to scale out git | 08:29 |
| Ryan_Lane | and already have a game plan for making it work for that | 08:29 |
| Ryan_Lane | what does zuul merger do? | 08:29 |
| * Ryan_Lane is very interested in zuul, but can't use it | 08:29 | |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/config: Don't enable oslo-incubator py33 for stable/havana https://review.openstack.org/76464 | 08:33 |
| *** sergmelikyan has quit IRC | 08:33 | |
| mordred | Ryan_Lane: it offloads the merging workload from zuul itself into a set of gearman workers | 08:36 |
| avishay | anyone knows why jenkins has tried checked this patch like 20 times in a row? https://review.openstack.org/#/c/49755/ | 08:36 |
| Ryan_Lane | ah. neat. | 08:37 |
| mordred | Ryan_Lane: we reached the point where the work of performing merges was a scaling bottleneck | 08:37 |
| Ryan_Lane | that's a good bottleneck to have ;) | 08:37 |
| mordred | not aroudn feature freeze time it's not :) | 08:37 |
| *** protux has joined #openstack-infra | 08:37 | |
| Ryan_Lane | :D | 08:37 |
| *** andreaf has joined #openstack-infra | 08:39 | |
| *** talluri_ has quit IRC | 08:47 | |
| *** talluri has joined #openstack-infra | 08:48 | |
| *** ociuhandu has joined #openstack-infra | 08:50 | |
| *** ociuhandu has quit IRC | 08:52 | |
| *** talluri has quit IRC | 08:52 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Fix API launch https://review.openstack.org/76467 | 08:53 |
| *** thuc has joined #openstack-infra | 08:56 | |
| *** dolphm_503 is now known as dolphm | 08:57 | |
| *** protux has quit IRC | 08:58 | |
| openstackgerrit | sahid proposed a change to openstack-infra/config: Adds new project Warm https://review.openstack.org/76247 | 08:59 |
| *** vogxn has joined #openstack-infra | 09:01 | |
| *** jcooley_ has joined #openstack-infra | 09:03 | |
| *** jpich has joined #openstack-infra | 09:04 | |
| *** rossella_s has joined #openstack-infra | 09:07 | |
| *** shashank_ has quit IRC | 09:09 | |
| *** dolphm is now known as dolphm_503 | 09:09 | |
| *** jcooley_ has quit IRC | 09:09 | |
| *** katyafervent_awa is now known as katyafervent | 09:11 | |
| lifeless | hello infra! https://bugs.launchpad.net/tripleo/+bug/1284054 - the ticket was updated but not set to fixcommitted. | 09:12 |
| lifeless | clarkb: ^ food for ? tomorrow morning:) | 09:12 |
| ttx | lifeless: Partial-Bug: does comment but not change status | 09:13 |
| lifeless | oh nvm, 'partial bug' | 09:13 |
| lifeless | ttx: hah, junix - thanks | 09:13 |
| *** yassine has joined #openstack-infra | 09:14 | |
| *** derekh has joined #openstack-infra | 09:14 | |
| *** marun has quit IRC | 09:15 | |
| lifeless | bah, jinx | 09:16 |
| SpamapS | hrm... | 09:16 |
| SpamapS | does tox 1.7.0 require significant changes to tox.ini? | 09:17 |
| SpamapS | tox.ConfigError: ConfigError: substitution key 'posargs' not found | 09:17 |
| SpamapS | appears so | 09:17 |
| *** talluri has joined #openstack-infra | 09:18 | |
| *** thuc has quit IRC | 09:18 | |
| *** jcooley_ has joined #openstack-infra | 09:20 | |
| *** luqas has quit IRC | 09:22 | |
| *** locke105 has quit IRC | 09:22 | |
| *** hashar has joined #openstack-infra | 09:22 | |
| *** hashar has quit IRC | 09:22 | |
| *** gokrokve has joined #openstack-infra | 09:22 | |
| *** talluri has quit IRC | 09:23 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Fix API launch https://review.openstack.org/76467 | 09:24 |
| *** jcooley_ has quit IRC | 09:25 | |
| *** dmakogon_ is now known as denis_makogon | 09:25 | |
| *** rcarrillocruz1 has joined #openstack-infra | 09:25 | |
| *** hashar has joined #openstack-infra | 09:25 | |
| *** fbo_away is now known as fbo | 09:26 | |
| *** gokrokve has quit IRC | 09:27 | |
| *** rcarrillocruz has quit IRC | 09:27 | |
| lifeless | SpamapS: 1.7.0 is broken | 09:30 |
| lifeless | SpamapS: AIUI | 09:30 |
| *** talluri has joined #openstack-infra | 09:31 | |
| SpamapS | of course | 09:32 |
| SpamapS | pycon is coming | 09:32 |
| SpamapS | have to release new features | 09:32 |
| rcarrillocruz1 | heh | 09:32 |
| *** valentinbud has quit IRC | 09:32 | |
| *** talluri has quit IRC | 09:33 | |
| *** talluri has joined #openstack-infra | 09:33 | |
| openstackgerrit | Martin Mágr proposed a change to openstack-infra/config: Watch also havana branch for packstack https://review.openstack.org/76206 | 09:34 |
| *** alexpilotti has joined #openstack-infra | 09:34 | |
| *** luqas has joined #openstack-infra | 09:35 | |
| *** jamielennox has quit IRC | 09:36 | |
| *** CaptTofu has joined #openstack-infra | 09:37 | |
| *** talluri has quit IRC | 09:37 | |
| *** yaguang has quit IRC | 09:38 | |
| *** jamielennox has joined #openstack-infra | 09:39 | |
| *** CaptTofu has quit IRC | 09:42 | |
| *** ianw has quit IRC | 09:43 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 09:45 | |
| *** johnthetubaguy has joined #openstack-infra | 09:47 | |
| *** ianw has joined #openstack-infra | 09:47 | |
| *** bogdando has quit IRC | 09:50 | |
| *** valentinbud has joined #openstack-infra | 09:50 | |
| *** rcarrillocruz has joined #openstack-infra | 09:51 | |
| *** bogdando has joined #openstack-infra | 09:52 | |
| *** rcarrillocruz1 has quit IRC | 09:52 | |
| *** jp_at_hp has joined #openstack-infra | 09:52 | |
| *** jcooley_ has joined #openstack-infra | 09:59 | |
| *** dolphm_503 is now known as dolphm | 10:00 | |
| *** rcarrillocruz1 has joined #openstack-infra | 10:00 | |
| *** rcarrillocruz has quit IRC | 10:03 | |
| *** shardy is now known as shardy_afk | 10:03 | |
| *** SergeyLukjanov has quit IRC | 10:04 | |
| *** SergeyLukjanov has joined #openstack-infra | 10:04 | |
| *** davidhadas has quit IRC | 10:06 | |
| *** ociuhandu has joined #openstack-infra | 10:06 | |
| *** jcooley_ has quit IRC | 10:06 | |
| *** jlibosva has quit IRC | 10:08 | |
| *** jlibosva has joined #openstack-infra | 10:08 | |
| *** dolphm is now known as dolphm_503 | 10:09 | |
| *** che-arne has joined #openstack-infra | 10:12 | |
| *** johnthetubaguy has quit IRC | 10:12 | |
| *** johnthetubaguy has joined #openstack-infra | 10:12 | |
| *** luqas has quit IRC | 10:15 | |
| *** mkoderer has joined #openstack-infra | 10:18 | |
| *** david-lyle has quit IRC | 10:19 | |
| *** mrda is now known as mrda_away | 10:21 | |
| *** dizquierdo has joined #openstack-infra | 10:25 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 10:30 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Add token header to requests https://review.openstack.org/75961 | 10:32 |
| *** rcarrillocruz has joined #openstack-infra | 10:39 | |
| *** davidhadas has joined #openstack-infra | 10:41 | |
| *** rcarrillocruz1 has quit IRC | 10:42 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Auth support https://review.openstack.org/73219 | 10:47 |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Add token header to requests https://review.openstack.org/75961 | 10:47 |
| openstackgerrit | Flavio Percoco proposed a change to openstack-infra/devstack-gate: Archive config files along with logs https://review.openstack.org/69344 | 10:51 |
| *** luqas has joined #openstack-infra | 10:52 | |
| *** thuc has joined #openstack-infra | 10:55 | |
| *** dolphm_503 is now known as dolphm | 11:00 | |
| *** thuc has quit IRC | 11:00 | |
| *** jaypipes has quit IRC | 11:03 | |
| *** ianw has quit IRC | 11:03 | |
| *** jamielennox has quit IRC | 11:04 | |
| *** jamielennox has joined #openstack-infra | 11:05 | |
| *** ianw has joined #openstack-infra | 11:05 | |
| *** rossella_s has quit IRC | 11:06 | |
| *** dolphm is now known as dolphm_503 | 11:10 | |
| *** Xurong has quit IRC | 11:12 | |
| *** mfer has joined #openstack-infra | 11:12 | |
| *** ociuhandu has quit IRC | 11:19 | |
| *** jhesketh_ has quit IRC | 11:22 | |
| *** gokrokve has joined #openstack-infra | 11:22 | |
| *** jerryz has quit IRC | 11:22 | |
| *** rcarrillocruz1 has joined #openstack-infra | 11:22 | |
| katyafervent | Hi everybody! | 11:25 |
| *** rcarrillocruz has quit IRC | 11:25 | |
| *** gokrokve has quit IRC | 11:27 | |
| katyafervent | Does anyone have experience in renaming repository? Will history be saved? | 11:27 |
| SergeyLukjanov | katyafervent, hey | 11:30 |
| SergeyLukjanov | katyafervent, are you talking about your murano repos at stackforge? | 11:30 |
| katyafervent | SergeyLukjanov, yes) | 11:30 |
| katyafervent | we want to start repository reorganization but want history to be kept | 11:31 |
| SergeyLukjanov | katyafervent, review history will be saved, but renaming requires gerrit shutdown, so, it'll be postponed and batched with other renaming requests | 11:31 |
| SergeyLukjanov | I think the next renaming batch will be for renaming savanna | 11:32 |
| SergeyLukjanov | katyafervent, if you'd like to merge some repos and keep their review history than I think that it's impossible | 11:32 |
| *** jhesketh has quit IRC | 11:32 | |
| ruhe | katyafervent: another option would be to create new repo named "murano". merge all the other repos to the new repo, then remove/deprecate old repos | 11:33 |
| SergeyLukjanov | ruhe, yup | 11:33 |
| *** rcarrillocruz has joined #openstack-infra | 11:34 | |
| SergeyLukjanov | katyafervent, if you'd like to keep git history the only way is to manually merge git trees with help of git-filter-branch | 11:34 |
| katyafervent | and what about commit history? | 11:35 |
| katyafervent | Oh, ok | 11:35 |
| *** rcarrillocruz1 has quit IRC | 11:36 | |
| *** BobBallAway is now known as BobBall | 11:38 | |
| *** CaptTofu has joined #openstack-infra | 11:38 | |
| openstackgerrit | sahid proposed a change to openstack-infra/config: new-project: warm https://review.openstack.org/76247 | 11:40 |
| *** sergmelikyan has joined #openstack-infra | 11:40 | |
| *** CaptTofu has quit IRC | 11:43 | |
| *** matsuhashi has quit IRC | 11:46 | |
| *** jcooley_ has joined #openstack-infra | 11:48 | |
| *** yamahata has quit IRC | 11:50 | |
| *** salv-orlando has quit IRC | 11:52 | |
| *** salv-orlando has joined #openstack-infra | 11:52 | |
| *** jgallard has quit IRC | 11:53 | |
| *** sergmelikyan has quit IRC | 11:54 | |
| *** jcooley_ has quit IRC | 11:54 | |
| *** sergmelikyan has joined #openstack-infra | 11:57 | |
| *** mfer has quit IRC | 11:57 | |
| openstackgerrit | Ilya Sviridov proposed a change to openstack-infra/config: Added new MagnetoDB project to Stackforge https://review.openstack.org/71308 | 11:59 |
| *** rfolco has joined #openstack-infra | 12:01 | |
| *** dolphm_503 is now known as dolphm | 12:01 | |
| *** CaptTofu has joined #openstack-infra | 12:02 | |
| sdague | SergeyLukjanov: so the rename thing, why do we need to shut down gerrit for that again? Why don't we just import it as a new repo? | 12:02 |
| SergeyLukjanov | sdague, in case of renaming we're update gerrit's db to keep all reviews | 12:02 |
| SergeyLukjanov | sdague, morning | 12:02 |
| *** coolsvap has quit IRC | 12:02 | |
| sdague | SergeyLukjanov: ah, right. | 12:03 |
| *** luqas has quit IRC | 12:05 | |
| *** luqas has joined #openstack-infra | 12:07 | |
| *** sergmelikyan has quit IRC | 12:08 | |
| *** dolphm is now known as dolphm_503 | 12:11 | |
| *** rcarrillocruz1 has joined #openstack-infra | 12:12 | |
| *** rcarrillocruz has quit IRC | 12:14 | |
| dims | sdague, Good morning. I have 2 +2's on https://review.openstack.org/#/c/75539/ but had to rebase, can you please approve when you get a chance? | 12:14 |
| sdague | done | 12:15 |
| dims | thanks a ton | 12:17 |
| lifeless | erm | 12:17 |
| lifeless | I'm clearly up way to late | 12:17 |
| dims | lifeless, good night :) | 12:18 |
| *** gokrokve has joined #openstack-infra | 12:22 | |
| *** cody-somerville has quit IRC | 12:24 | |
| *** hashar has quit IRC | 12:24 | |
| *** gokrokve has quit IRC | 12:27 | |
| *** tsufiev___ has joined #openstack-infra | 12:30 | |
| *** zhiyan_ is now known as zhiyan | 12:32 | |
| *** zul has joined #openstack-infra | 12:32 | |
| *** rossella_s has joined #openstack-infra | 12:36 | |
| *** luqas has quit IRC | 12:36 | |
| *** valentinbud has quit IRC | 12:42 | |
| *** CaptTofu has quit IRC | 12:43 | |
| *** CaptTofu has joined #openstack-infra | 12:43 | |
| *** mbacchi has joined #openstack-infra | 12:54 | |
| *** dolphm_503 is now known as dolphm | 12:57 | |
| *** lcostantino has joined #openstack-infra | 12:58 | |
| *** beagles has quit IRC | 12:58 | |
| *** dstanek has joined #openstack-infra | 13:00 | |
| *** smarcet has joined #openstack-infra | 13:02 | |
| *** vogxn has quit IRC | 13:03 | |
| *** johnthetubaguy has quit IRC | 13:05 | |
| *** johnthetubaguy1 has joined #openstack-infra | 13:05 | |
| *** marun has joined #openstack-infra | 13:08 | |
| *** yamahata has joined #openstack-infra | 13:09 | |
| *** rfolco has quit IRC | 13:13 | |
| *** valentinbud has joined #openstack-infra | 13:19 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 13:20 | |
| anteaya | avishay: thank you for asking about https://review.openstack.org/#/c/49755/ I am also curious as to why Jenkins was caught in a perpetual testing loop there. | 13:21 |
| *** e0ne has quit IRC | 13:21 | |
| avishay | anteaya: i hope it's over :) | 13:21 |
| *** e0ne has joined #openstack-infra | 13:21 | |
| *** gokrokve has joined #openstack-infra | 13:22 | |
| anteaya | I was just checking, I don't see it in the check queue now. | 13:22 |
| anteaya | Abandoning it might have stopped the cycle. | 13:22 |
| anteaya | The -2 after 2 months of inactivity appears to have been the trigger. | 13:24 |
| *** johnthetubaguy1 is now known as johnthetubaguy | 13:25 | |
| *** mrmartin has joined #openstack-infra | 13:26 | |
| openstackgerrit | Julien Vey proposed a change to openstack/requirements: Add Docker requirement https://review.openstack.org/76535 | 13:26 |
| *** gokrokve has quit IRC | 13:27 | |
| *** sdake has quit IRC | 13:28 | |
| *** sdake has joined #openstack-infra | 13:28 | |
| *** rfolco has joined #openstack-infra | 13:29 | |
| *** ociuhandu has joined #openstack-infra | 13:29 | |
| *** hashar has joined #openstack-infra | 13:31 | |
| *** rfolco has quit IRC | 13:34 | |
| *** ArxCruz has joined #openstack-infra | 13:35 | |
| *** yamahata has quit IRC | 13:36 | |
| *** jcooley_ has joined #openstack-infra | 13:36 | |
| *** yamahata has joined #openstack-infra | 13:37 | |
| *** dprince has joined #openstack-infra | 13:37 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 13:38 | |
| *** jgallard has joined #openstack-infra | 13:40 | |
| *** sandywalsh has quit IRC | 13:40 | |
| *** thuc has joined #openstack-infra | 13:40 | |
| *** nosnos has quit IRC | 13:41 | |
| *** zul has quit IRC | 13:42 | |
| *** lcheng has joined #openstack-infra | 13:43 | |
| *** zul has joined #openstack-infra | 13:44 | |
| *** dkranz has joined #openstack-infra | 13:44 | |
| avishay | anteaya: strange, but ok :) | 13:45 |
| *** thuc has quit IRC | 13:45 | |
| *** ArxCruz has quit IRC | 13:46 | |
| anteaya | avishay: when fungi is around he may have more | 13:47 |
| anteaya | at least I hope so | 13:47 |
| *** dhellman_ has joined #openstack-infra | 13:49 | |
| *** dhellman_ has quit IRC | 13:49 | |
| avishay | anteaya: ok cool, just thought i'd let you guys know | 13:49 |
| *** rfolco has joined #openstack-infra | 13:49 | |
| anteaya | glad you did thank you | 13:50 |
| avishay | sure | 13:51 |
| *** shardy_afk is now known as shardy | 13:52 | |
| *** sandywalsh has joined #openstack-infra | 13:53 | |
| *** mfer has joined #openstack-infra | 13:53 | |
| *** rfolco has quit IRC | 13:54 | |
| *** mestery has quit IRC | 13:54 | |
| *** thuc_ has joined #openstack-infra | 13:55 | |
| fungi | something critical? | 13:56 |
| *** weshay has joined #openstack-infra | 13:59 | |
| *** ArxCruz has joined #openstack-infra | 13:59 | |
| anteaya | no | 14:00 |
| anteaya | nothing critical | 14:00 |
| anteaya | take your time, it can wait | 14:00 |
| *** bknudson has quit IRC | 14:00 | |
| *** zehicle_at_dell has quit IRC | 14:03 | |
| fungi | SpamapS: lifeless: https://bitbucket.org/hpk42/tox/pull-request/85/fix-command-expansion-and-parsing/activity | 14:03 |
| *** ryanpetrello has joined #openstack-infra | 14:08 | |
| ArxCruz | fungi: around ? | 14:10 |
| fungi | ArxCruz: yeah, just checking some logs and coming up to speed for the morning... what's up? | 14:11 |
| ArxCruz | fungi: http://paste.openstack.org/show/69859/ I have this in my layout.yaml, I though it would report only success jobs, but it's reporting failures too | 14:11 |
| ArxCruz | Did I something wrong ? | 14:11 |
| fungi | ArxCruz: do you have an example of a change it commented on? | 14:12 |
| *** jeckersb_gone is now known as jeckersb | 14:12 | |
| ArxCruz | fungi: https://review.openstack.org/#/c/76411 | 14:12 |
| *** lcheng has quit IRC | 14:14 | |
| fungi | ArxCruz: i think the " | 14:14 |
| *** mestery has joined #openstack-infra | 14:14 | |
| fungi | er | 14:14 |
| fungi | the "Build succeeded." there implies that zuul reported it as a success | 14:15 |
| fungi | because all the jobs were non-voting\ | 14:15 |
| *** dolphm is now known as dolphm_503 | 14:15 | |
| ArxCruz | fungi: so, there's no way to don't report? | 14:15 |
| fungi | ArxCruz: i think if you changed one or more of those jobs to be a voting job, the way your layout.yaml is configured zuul will add a comment (but not a vote) on success and will do nothing on failure | 14:16 |
| *** jcooley_ has quit IRC | 14:16 | |
| ArxCruz | fungi: let me save this conversation :P | 14:16 |
| ArxCruz | I will do that :) | 14:17 |
| fungi | right now it's commenting because you've configured it to leave a comment on success, and with all jobs non-voting, any run will be interpreted by zuul as a success even if the jobs themselves failed (because they're non-voting jobs) | 14:17 |
| ArxCruz | got it | 14:17 |
| fungi | "non-voting" for a job merely indicates whether it contributes to the completion result, but the actual verified: 1 or -1 is what causes zuul itself to leave a vote | 14:18 |
| ArxCruz | ok, so, if I mark one job as voting, it will report, only if both jobs success, or it will report if the voting one succeed ? | 14:18 |
| *** julim has joined #openstack-infra | 14:19 | |
| fungi | the way you have it configured right now, any one voting job which fails will cause your zuul not to comment | 14:19 |
| *** vponomaryov has joined #openstack-infra | 14:19 | |
| fungi | even if one or more other voting jobs succeed | 14:19 |
| fungi | basically as long as at least one voting job fails, zuul considers the overall result to be a failure | 14:20 |
| fungi | if all voting jobs succeed, zuul considers that to be a success | 14:20 |
| *** bknudson has joined #openstack-infra | 14:20 | |
| fungi | if no jobs are voting, zuul interprets that as a success as well | 14:20 |
| *** mriedem has joined #openstack-infra | 14:20 | |
| fungi | hopefully that makes sense | 14:21 |
| fungi | ArxCruz: obligatory documentation link... http://ci.openstack.org/zuul/zuul.html#pipelines | 14:22 |
| *** gokrokve has joined #openstack-infra | 14:22 | |
| *** dims has quit IRC | 14:22 | |
| ArxCruz | fungi: okay, so I need to turn voting for both jobs :) | 14:22 |
| ArxCruz | fungi: thanks, I read that before, didn't pay attention for the votting/non-votting | 14:23 |
| fungi | that sounds like what you want, yes | 14:23 |
| ArxCruz | fungi: thanks! | 14:24 |
| *** dolphm_503 is now known as dolphm | 14:25 | |
| fungi | ArxCruz: reviewing the documentation there, the "voting" parameter for jobs could definitely stand to have a more detailed description. feel free to contribute a patch to the documentation in the openstack-infra/zuul project or open a low-hanging-fruit bug against the zuul project in launchpad | 14:25 |
| ArxCruz | fungi: ok, cool | 14:25 |
| anteaya | fungi: as an fyi I need to go to town sometime this week for a few hours and since jim will be away Thurs and Fri, I thought I would go today | 14:25 |
| anteaya | I'll probably head out around noon | 14:26 |
| fungi | anteaya: okay, sounds good | 14:26 |
| anteaya | thanks | 14:26 |
| fungi | ArxCruz: we always love documentation improvements, and the people best positioned to point out where documentation is lacking are those who are encountering these systems for the first time | 14:26 |
| *** gokrokve has quit IRC | 14:27 | |
| ArxCruz | fungi: ;) okay I will work on that | 14:27 |
| anteaya | and my brain is starting to feel like my brain again, I'm so happy | 14:27 |
| openstackgerrit | Ziad Sawalha proposed a change to openstack-infra/config: Add satori as a new project https://review.openstack.org/73667 | 14:29 |
| fungi | ArxCruz: so yeah, if you want to propose a patch for that, we can easily help correct any misstatements or improve the wording | 14:30 |
| fungi | once it's in review | 14:30 |
| *** dolphm is now known as dolphm_503 | 14:30 | |
| *** wchrisj has joined #openstack-infra | 14:30 | |
| *** rfolco has joined #openstack-infra | 14:31 | |
| *** e0ne_ has joined #openstack-infra | 14:32 | |
| *** thuc_ has quit IRC | 14:32 | |
| *** thuc has joined #openstack-infra | 14:33 | |
| *** dims has joined #openstack-infra | 14:33 | |
| openstackgerrit | Sahdev Zala proposed a change to openstack-infra/config: New StackForge project heat-translator https://review.openstack.org/75988 | 14:34 |
| *** esker has joined #openstack-infra | 14:34 | |
| vponomaryov | jeblair, fungi, mordred: hello, I have little bugfix - https://review.openstack.org/#/c/76115/ , could you review it please? | 14:36 |
| *** e0ne has quit IRC | 14:36 | |
| *** thuc has quit IRC | 14:37 | |
| *** dkliban has joined #openstack-infra | 14:37 | |
| *** prad_ has joined #openstack-infra | 14:37 | |
| anteaya | fungi: so the item I had was Jenkins seemed to be caught in a check testing loop on this patch: https://review.openstack.org/#/c/49755/ | 14:37 |
| anteaya | was wondering what you thought would be the fix for this situation | 14:37 |
| fungi | anteaya: yeah, i saw--i've been looking at the zuul logs trying to figure out why it thought it should recheck that | 14:38 |
| anteaya | k thanks | 14:38 |
| *** jpeeler has quit IRC | 14:40 | |
| fungi | i think we need the logs to start including more of the parameters zuul considers when deciding whether to enqueue a change | 14:41 |
| anteaya | hmmmm | 14:43 |
| *** alexpilotti_ has joined #openstack-infra | 14:44 | |
| anteaya | so nothing useful | 14:44 |
| anteaya | should i file a bug? | 14:45 |
| *** yamahata has quit IRC | 14:46 | |
| *** yamahata has joined #openstack-infra | 14:46 | |
| *** alexpilotti has quit IRC | 14:47 | |
| *** alexpilotti_ is now known as alexpilotti | 14:47 | |
| *** freyes has joined #openstack-infra | 14:47 | |
| fungi | i'm still digging | 14:47 |
| anteaya | k | 14:47 |
| *** alexpilotti has quit IRC | 14:51 | |
| *** krotscheck has joined #openstack-infra | 14:52 | |
| *** jcoufal has quit IRC | 14:52 | |
| fungi | it definitely seems to have been responding to its own comment events... http://paste.openstack.org/show/69865/ | 14:55 |
| fungi | when it gets gerrit information on the change, we should probably have debug log entries mentioning some of the things it looks for like age of its own most recent previous comment, existing votes on the change and so on | 14:57 |
| *** wenlock has joined #openstack-infra | 14:59 | |
| anteaya | hmmmm | 14:59 |
| anteaya | I wonder if the fact that the check tests returned VRIF:1 sent it back to check | 15:00 |
| anteaya | but why would it send it back to check if that is the case? | 15:00 |
| anteaya | yes, more logs would help | 15:01 |
| *** e0ne has joined #openstack-infra | 15:01 | |
| *** saju_m has quit IRC | 15:02 | |
| *** gokrokve has joined #openstack-infra | 15:02 | |
| fungi | because while this could be caused by gerrit misreporting or zuul misinterpreting the re | 15:02 |
| fungi | sults | 15:02 |
| fungi | we don't really know what they were | 15:02 |
| fungi | anteaya: if you want to open a bug against zuul on launchpad and paste that bit of logs in along with the observed behavior and link to the change, that would be a wonderful help | 15:03 |
| anteaya | I can do that | 15:04 |
| *** e0ne_ has quit IRC | 15:05 | |
| *** jeckersb is now known as jeckersb_gone | 15:05 | |
| *** wenlock has quit IRC | 15:06 | |
| *** rossella_s has quit IRC | 15:09 | |
| *** jgrimm has joined #openstack-infra | 15:10 | |
| *** jeckersb_gone is now known as jeckersb | 15:11 | |
| SergeyLukjanov | fungi, anteaya, could you, please, take a look on https://review.openstack.org/#/c/76312 when have some time | 15:12 |
| *** pmathews has joined #openstack-infra | 15:13 | |
| *** luqas has joined #openstack-infra | 15:15 | |
| *** pblaho` has joined #openstack-infra | 15:15 | |
| *** pblaho has quit IRC | 15:16 | |
| anteaya | fungi: https://bugs.launchpad.net/zuul/+bug/1285210 | 15:16 |
| anteaya | hi SergeyLukjanov | 15:16 |
| *** Sukhdev has joined #openstack-infra | 15:16 | |
| fungi | anteaya: thanks! | 15:16 |
| * anteaya clicks on 76312 | 15:16 | |
| anteaya | np | 15:16 |
| SergeyLukjanov | anteaya, hey, morning | 15:16 |
| SergeyLukjanov | anteaya, thx | 15:17 |
| anteaya | fungi: thanks for teeing it up | 15:17 |
| fungi | jeblair: you also may be interested in zuul bug https://launchpad.net/bugs/1285210 when you're up and about | 15:17 |
| SergeyLukjanov | I've shared my idea one (several?) days ago in the channel, so, that's the first patch | 15:17 |
| *** rossella_s has joined #openstack-infra | 15:17 | |
| SergeyLukjanov | and there three more on the road | 15:17 |
| *** rwsu has joined #openstack-infra | 15:18 | |
| *** dkliban has quit IRC | 15:18 | |
| *** dkliban has joined #openstack-infra | 15:18 | |
| anteaya | SergeyLukjanov: silly question time | 15:19 |
| *** jpeeler has joined #openstack-infra | 15:19 | |
| *** jpeeler has quit IRC | 15:19 | |
| *** jpeeler has joined #openstack-infra | 15:19 | |
| anteaya | this job is in the check queue and the post queue | 15:20 |
| anteaya | what happens if it fails in post? | 15:20 |
| SergeyLukjanov | anteaya, in check and post? | 15:20 |
| SergeyLukjanov | anteaya, sounds strange | 15:20 |
| anteaya | next silly question: it is in check and post and called gate-project-requirements | 15:20 |
| mordred | anteaya: nice catch | 15:20 |
| anteaya | https://review.openstack.org/#/c/76312/2/modules/openstack_project/files/zuul/layout.yaml | 15:20 |
| anteaya | line 256 | 15:21 |
| SergeyLukjanov | shame on me, it should gate | 15:21 |
| SergeyLukjanov | :( | 15:21 |
| anteaya | thank you | 15:21 |
| anteaya | I was wondering what the purpose was if we skipped the gate | 15:21 |
| anteaya | mordred: thanks, taking my new brain for a test drive | 15:22 |
| anteaya | so far it seems to be working | 15:22 |
| SergeyLukjanov | anteaya, thanks for catching this | 15:23 |
| *** mrodden has joined #openstack-infra | 15:23 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Extract check/gate req jobs to check-requirements https://review.openstack.org/76312 | 15:23 |
| anteaya | SergeyLukjanov: np | 15:24 |
| *** chuck__ has joined #openstack-infra | 15:24 | |
| SergeyLukjanov | btw the next step is to extract tarballs + pypi-release and then to extract common integrated jobs | 15:25 |
| *** johnthetubaguy has quit IRC | 15:26 | |
| openstackgerrit | David Ripton proposed a change to openstack-dev/hacking: Add H308 rule. https://review.openstack.org/76559 | 15:26 |
| *** pblaho` is now known as pblaho | 15:27 | |
| *** dcramer__ has joined #openstack-infra | 15:30 | |
| openstackgerrit | Nikita Konovalov proposed a change to openstack-infra/storyboard-webclient: Add token header to requests https://review.openstack.org/75961 | 15:30 |
| *** changbl has quit IRC | 15:30 | |
| *** dolphm_503 is now known as dolphm | 15:31 | |
| *** VINOD has joined #openstack-infra | 15:32 | |
| *** pdmars has joined #openstack-infra | 15:33 | |
| *** rossella_s has quit IRC | 15:34 | |
| openstackgerrit | David Ripton proposed a change to openstack-dev/hacking: Add H308 rule https://review.openstack.org/76559 | 15:34 |
| mordred | SergeyLukjanov: +2 | 15:34 |
| SergeyLukjanov | mordred, thx | 15:35 |
| fungi | wow--yeah, i totally missed that said "post" and not "gate" (my brain simply corrected what my eyeballs read i guess) | 15:35 |
| fungi | great catch, anteaya | 15:35 |
| *** mgagne has joined #openstack-infra | 15:36 | |
| anteaya | fungi: thanks | 15:36 |
| anteaya | SergeyLukjanov: no requirements for oslo.vmware? | 15:36 |
| SergeyLukjanov | anteaya, they have only non-voting job atm | 15:37 |
| SergeyLukjanov | and we prefere to not add non-voting jobs to the gate pipeline - so | 15:38 |
| openstackgerrit | A change was merged to openstack-infra/config: Extract check/gate req jobs to check-requirements https://review.openstack.org/76312 | 15:38 |
| anteaya | okay | 15:38 |
| anteaya | SergeyLukjanov: what about pbr? | 15:41 |
| *** dcramer__ has quit IRC | 15:41 | |
| anteaya | it is a different requirements job | 15:41 |
| anteaya | requirements integration | 15:42 |
| *** zhiyan is now known as zhiyan_ | 15:42 | |
| *** mgagne has quit IRC | 15:43 | |
| SergeyLukjanov | anteaya, it's not synced now with requirements | 15:43 |
| SergeyLukjanov | AFAIR there was an intention to sync it and setup requirements jobs | 15:43 |
| anteaya | ah okay | 15:43 |
| SergeyLukjanov | I've already fixed sync script in os/req repo to not brake pbr | 15:43 |
| anteaya | yeah the job is named differently | 15:43 |
| anteaya | you'd be the only thing that doesn't | 15:44 |
| SergeyLukjanov | because previously it was breaking pbr by overriding seyup.py | 15:44 |
| anteaya | ah | 15:44 |
| VINOD | Hi anteaya... | 15:44 |
| *** lnxnut has joined #openstack-infra | 15:44 | |
| SergeyLukjanov | anteaya, I mean https://review.openstack.org/#/c/73431/ | 15:45 |
| VINOD | Could you able to check the logs of https://review.openstack.org/#/c/75967/ | 15:45 |
| *** chuck__ has quit IRC | 15:45 | |
| anteaya | SergeyLukjanov: climate is the only stackforge project with a requirements job? | 15:45 |
| *** zhiyan_ is now known as zhiyan | 15:45 | |
| anteaya | VINOD: hi, are you also in #openstack-nova channel? | 15:46 |
| VINOD | yes | 15:46 |
| SergeyLukjanov | anteaya, savanna was too, when it was on stackforge ;) | 15:46 |
| anteaya | VINOD: if not can you join please | 15:46 |
| anteaya | VINOD: great, stand by I will be with you in a minute | 15:46 |
| anteaya | SergeyLukjanov: okay | 15:46 |
| fungi | VINOD: that's an extremely large change (over 3000 lines). it's unlikely you'll find anyone actually reviewing it unless you break it up into smaller, more manageable patches | 15:46 |
| VINOD | thanks | 15:46 |
| VINOD | fungi: ok | 15:46 |
| *** wenlock has joined #openstack-infra | 15:47 | |
| fungi | generally once a change gets past about 200-300 lines, people tend not to review it because it's difficult to reason about its implications | 15:47 |
| VINOD | ok | 15:47 |
| anteaya | oh and rally and solum and openstacksdk | 15:47 |
| *** prad_ has quit IRC | 15:48 | |
| anteaya | SergeyLukjanov: so much for my review, by the time I was done it was merged | 15:48 |
| anteaya | but it was fun to look at it | 15:48 |
| *** zhiyan is now known as zhiyan_ | 15:48 | |
| fungi | anteaya: if you discover something's wrong with a change after it's merged, it can always be solved with a new change, so no review is really a waste | 15:49 |
| SergeyLukjanov | yup, agreed | 15:49 |
| SergeyLukjanov | config-compare-xml Jenkins XML output is unchanged. in 1m 35s (non-voting) | 15:49 |
| SergeyLukjanov | fungi, mordred, does it mean that such review is correct? | 15:49 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix manila-tempest-job to use exported env var https://review.openstack.org/76115 | 15:49 |
| *** e0ne has quit IRC | 15:50 | |
| SergeyLukjanov | that nothing changed | 15:50 |
| *** e0ne has joined #openstack-infra | 15:50 | |
| jeblair | SergeyLukjanov: sometimes we want the xml to change, and sometimes we don't... | 15:50 |
| *** mrmartin has quit IRC | 15:50 | |
| fungi | SergeyLukjanov: if the change didn't modify any jjb configuration for existing job definitions, then there should be no difference reported by that test | 15:50 |
| SergeyLukjanov | I mean for the change like mine | 15:51 |
| *** yolanda_ has joined #openstack-infra | 15:51 | |
| fungi | SergeyLukjanov: you only modified zuul configuration | 15:51 |
| SergeyLukjanov | when expect only jobs to templates extraction w/o adding new ones | 15:51 |
| SergeyLukjanov | oh, yup | 15:51 |
| SergeyLukjanov | it's a zuul layout change | 15:51 |
| SergeyLukjanov | need to have some coffee | 15:51 |
| anteaya | fungi: true | 15:51 |
| jeblair | SergeyLukjanov: oh, for the zuul change -- here's something you might want to do.... | 15:51 |
| fungi | SergeyLukjanov: the zuul layout test will, however, do some sanity checks on changes like yours | 15:51 |
| jeblair | SergeyLukjanov: if you look at the output of the layout test, it prints zuul's final configuration | 15:52 |
| *** chandan_kumar has quit IRC | 15:52 | |
| jeblair | SergeyLukjanov: you might want to compare old and new output to make sure that it's the same | 15:52 |
| SergeyLukjanov | jeblair, awesome tip | 15:52 |
| SergeyLukjanov | I'll investigate how to make the same job for checking layout unchanged | 15:52 |
| SergeyLukjanov | to be sure that my job2tmpl extractions will not miss jobs | 15:52 |
| jeblair | SergeyLukjanov: probably not a bad idea | 15:52 |
| *** dizquierdo has quit IRC | 15:53 | |
| *** dprince has quit IRC | 15:53 | |
| fungi | SergeyLukjanov: that's also an easy one to run locally with tox on master and your topic branch | 15:53 |
| SergeyLukjanov | fungi, yup | 15:53 |
| *** dcramer__ has joined #openstack-infra | 15:53 | |
| *** dizquierdo has joined #openstack-infra | 15:53 | |
| *** dprince has joined #openstack-infra | 15:53 | |
| SergeyLukjanov | sorry, gtg, time to go home ;) | 15:54 |
| fungi | SergeyLukjanov: have a great evening! | 15:54 |
| SergeyLukjanov | thx! | 15:54 |
| anteaya | thanks SergeyLukjanov | 15:56 |
| *** chandan_kumar has joined #openstack-infra | 15:56 | |
| *** mgagne has joined #openstack-infra | 15:57 | |
| *** david-lyle has joined #openstack-infra | 15:57 | |
| *** rlandy has quit IRC | 15:58 | |
| *** rcleere has quit IRC | 15:58 | |
| *** thedodd has joined #openstack-infra | 15:59 | |
| dansmith | ArxCruz: https://review.openstack.org/#/c/70533/ | 15:59 |
| dansmith | ArxCruz: build failed (others succeeded) and the link to the test report is 404 | 15:59 |
| ArxCruz | dansmith: we are working on that :) | 16:00 |
| *** jlibosva has quit IRC | 16:00 | |
| dansmith | ArxCruz: okay, just noticed, wanted to make sure | 16:00 |
| ArxCruz | dansmith: yeah, if possible, just ignore :) | 16:00 |
| dansmith | ArxCruz: I cannot ignore. I will be unable to do any more work until you fix it. | 16:01 |
| *** amotoki_ has joined #openstack-infra | 16:01 | |
| ArxCruz | dansmith: seriously ? O.o | 16:02 |
| *** david_lyle_ has joined #openstack-infra | 16:02 | |
| dansmith | ArxCruz: no, of course not :) | 16:02 |
| * ArxCruz didn't knew dansmith have sense of humor | 16:03 | |
| dansmith | pfft | 16:03 |
| ArxCruz | maurosr: did you saw this? ^ | 16:03 |
| maurosr | hahah | 16:03 |
| krtaylor | hehheh | 16:04 |
| *** julim_ has joined #openstack-infra | 16:05 | |
| *** julim has quit IRC | 16:05 | |
| *** dcramer__ has quit IRC | 16:05 | |
| anteaya | ArxCruz: well actually asking someone to ignore your output is not the best approach | 16:05 |
| *** david-lyle has quit IRC | 16:06 | |
| ArxCruz | anteaya: I mean, the logs were 404 was an error in our swift script | 16:06 |
| anteaya | ArxCruz: can you make an internal change at least identifying you are aware of this issue while you fix it | 16:06 |
| ArxCruz | we already fix it, but I'm not sure if we will be able to upload the logs, since we don't have it | 16:06 |
| anteaya | ArxCruz: can you add that to your comment message? | 16:06 |
| ArxCruz | anteaya: for the next? | 16:07 |
| dansmith | anteaya: it was their first day of reporting, nobody expects it to be perfect yet | 16:07 |
| ArxCruz | anteaya: it's already fixed :) | 16:07 |
| dansmith | anteaya: I was just making sure they had noticed | 16:07 |
| anteaya | dansmith: okay | 16:07 |
| anteaya | ArxCruz: okay | 16:07 |
| anteaya | we are just asking CI accounts to address issues for devs rather than continue to send out incorrect info while they are fixing things | 16:08 |
| *** prad_ has joined #openstack-infra | 16:08 | |
| *** david-lyle has joined #openstack-infra | 16:08 | |
| dansmith | given that a simple error could report incorrectly on a hundred patches before it goes noticed, I don't think making them comment on each one is very reasonable | 16:09 |
| dansmith | especially on day 1 | 16:09 |
| *** oubiwann_ has joined #openstack-infra | 16:09 | |
| anteaya | dansmith: glad to have your input on this | 16:10 |
| anteaya | thanks | 16:10 |
| ArxCruz | dansmith: anteaya we are monitoring everything to ensure we will fill all the gaps, don't worry :) | 16:11 |
| openstackgerrit | Travis Plummer proposed a change to openstack-infra/config: new-project https://review.openstack.org/71956 | 16:11 |
| *** david_lyle_ has quit IRC | 16:12 | |
| anteaya | ArxCruz: I know you have been great at it | 16:12 |
| ArxCruz | Also, I will stay here pratically 24x7 these days, please, feel free to contact-me if something goes wrong and our team didn't catch up | 16:12 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 16:12 | |
| anteaya | but I gave salv-orlando a hard time recently for something very similar, so I have to be consistent | 16:12 |
| *** e0ne_ has joined #openstack-infra | 16:12 | |
| *** pcrews has joined #openstack-infra | 16:12 | |
| *** david_lyle_ has joined #openstack-infra | 16:13 | |
| anteaya | and to salv-orlando's credit, he responded promptly | 16:13 |
| anteaya | it isn't your system that concerns me, it is the 40 other accounts who look to you as a role model | 16:13 |
| anteaya | and who will do everything you do, except be as vigilant | 16:13 |
| *** KurtMartin has joined #openstack-infra | 16:14 | |
| anteaya | ArxCruz: thanks | 16:14 |
| openstackgerrit | Andreas Rehn proposed a change to openstack-infra/jenkins-job-builder: Added support for Delivery Pipeline Plugin https://review.openstack.org/71658 | 16:15 |
| *** dcramer__ has joined #openstack-infra | 16:16 | |
| *** david-lyle has quit IRC | 16:16 | |
| *** e0ne has quit IRC | 16:16 | |
| krtaylor | anteaya, point taken, we are trying to make sure everything is working very well, as ArxCruz said, we are all watching it very closely right now | 16:17 |
| anteaya | krtaylor: thanks, yes you two are doing a wonderful job | 16:17 |
| *** kmartin has quit IRC | 16:17 | |
| *** changbl has joined #openstack-infra | 16:17 | |
| anteaya | and as I told salvatore, if it was just you, I wouldn't even worry about it | 16:17 |
| anteaya | since the work you are doing is outstanding | 16:18 |
| *** kmartin has joined #openstack-infra | 16:18 | |
| *** stevebaker has quit IRC | 16:18 | |
| anteaya | it is unfortunate that the people that are doing the best work that are limited by the people that do not | 16:19 |
| krtaylor | anteaya, this will all get smooth as more and more 3rd party comes online, more things get documented and fixed | 16:19 |
| krtaylor | agreed | 16:19 |
| *** david-lyle has joined #openstack-infra | 16:19 | |
| anteaya | but after not doing that we realized the mess we were creating | 16:19 |
| anteaya | yes, fortunately there is more documentaion happening and a wider knowledge base | 16:19 |
| *** andre__ has quit IRC | 16:20 | |
| krtaylor | which, reminds me, I want to help organize a self-help group for 3rd party CI testing, maybe a summit BOF | 16:20 |
| *** rcleere has joined #openstack-infra | 16:20 | |
| anteaya | everyone I have talked to wants to do a good job they just lack knowledge in how to do that | 16:20 |
| anteaya | krtaylor: I think that would be well attended | 16:20 |
| krtaylor | we can help each other, share best practices, maybe even weekly irc meetings | 16:21 |
| anteaya | krtaylor: did you know that jaypipes is offering a meeting/workshop for 3rd party ci on Monday? | 16:21 |
| krtaylor | yes, I'll be there for sure | 16:21 |
| krtaylor | although I was looking forward to the hangout :) | 16:21 |
| *** VINOD has quit IRC | 16:21 | |
| *** david_lyle_ has quit IRC | 16:21 | |
| *** KurtMartin has quit IRC | 16:22 | |
| anteaya | krtaylor: http://lists.openstack.org/pipermail/openstack-dev/2014-February/028124.html | 16:22 |
| anteaya | krtaylor: easier to log in irc | 16:22 |
| *** david_lyle_ has joined #openstack-infra | 16:23 | |
| anteaya | one of the really big problems is that a large part of this group has no experience with opensource | 16:23 |
| *** andre__ has joined #openstack-infra | 16:23 | |
| anteaya | and is unfamiliar with the tool set | 16:23 |
| krtaylor | yeah, I saw that, and it is true, scales better, translates better too | 16:23 |
| fungi | and helps get them into the flow of how the rest of the project contributors participate in discussion | 16:24 |
| anteaya | since opensource is a philosphy as much as code, consistent action helps to inform the philosphy | 16:24 |
| *** cody-somerville has joined #openstack-infra | 16:24 | |
| *** cody-somerville has quit IRC | 16:24 | |
| *** cody-somerville has joined #openstack-infra | 16:24 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add new elasticsearch cluster members to cacti https://review.openstack.org/76573 | 16:24 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update logstash doc for an elasticsearch cluster https://review.openstack.org/76574 | 16:24 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Move primary elasticsearch discover node https://review.openstack.org/76575 | 16:24 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch cluster members https://review.openstack.org/76051 | 16:24 |
| * anteaya ducks | 16:24 | |
| anteaya | krtaylor: and I am so glad you would like to organize something at the summit | 16:25 |
| anteaya | I must admit I am all organized out currently | 16:25 |
| anteaya | but I can certainly inform people | 16:25 |
| krtaylor | sure, I'll be happy to help, I was thinking about starting an etherpad to help get organized | 16:25 |
| anteaya | krtaylor: great idea | 16:26 |
| krtaylor | then maillist, etc | 16:26 |
| anteaya | do post the link to the etherpad once you have it | 16:26 |
| anteaya | krtaylor: on the -dev ml right? | 16:26 |
| krtaylor | ok, I'll get that moving | 16:26 |
| krtaylor | yes | 16:26 |
| anteaya | great | 16:26 |
| anteaya | thank you | 16:26 |
| krtaylor | although, my mail filtering is horrible | 16:26 |
| anteaya | mine too | 16:27 |
| krtaylor | I'd like to see us standardize a tag | 16:27 |
| *** david-lyle has quit IRC | 16:27 | |
| anteaya | I have such respect for those who can figure it out | 16:27 |
| *** dcramer___ has joined #openstack-infra | 16:27 | |
| krtaylor | like 3rd party or third-party | 16:27 |
| *** prad__ has joined #openstack-infra | 16:27 | |
| anteaya | krtaylor: yes, I like 3rd party | 16:27 |
| anteaya | fewer characters | 16:27 |
| *** david_lyle_ is now known as david-lyle | 16:27 | |
| krtaylor | true, anyway, good etherpad fodder | 16:27 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Remove resolved_at mention in readme https://review.openstack.org/76393 | 16:27 |
| anteaya | better chance of reading teh subject line | 16:27 |
| anteaya | yes | 16:28 |
| *** david_lyle_ has joined #openstack-infra | 16:28 | |
| *** prad_ has quit IRC | 16:28 | |
| *** dizquierdo has quit IRC | 16:28 | |
| *** cody-somerville has quit IRC | 16:29 | |
| *** johnthetubaguy has joined #openstack-infra | 16:29 | |
| *** alexpilotti has joined #openstack-infra | 16:30 | |
| jeblair | krtaylor: i expect we'll devote some of the infra track to 3rd party testing | 16:30 |
| dansmith | can someone kick the meeting bot in openstack-meeting-3 ? | 16:30 |
| *** dcramer__ has quit IRC | 16:30 | |
| jeblair | krtaylor: and i believe there is going to be a collaboration space with "project pods" | 16:30 |
| krtaylor | jeblair, project pods? | 16:31 |
| jeblair | krtaylor: so that might be a natural place to organize something; if there's an infra "pod", 3rd party ci would be a welcome topic there | 16:31 |
| *** jcooley_ has joined #openstack-infra | 16:31 | |
| jeblair | krtaylor: i'm not entirely sure what it will look like, but i'm imagining a big area with discrete locations (tables? groups of chairs?) designated with different projects (nova, etc) | 16:31 |
| *** david-lyle has quit IRC | 16:31 | |
| krtaylor | ah, like a developers lounge area by project? cool | 16:32 |
| jeblair | krtaylor: so we could potentially organize a formal session to cover some things and then use the collaboration area for the kind of self-help get different groups together activity you were suggestiong | 16:32 |
| jeblair | dansmith: ack | 16:32 |
| dansmith | jeblair: did you do it? seems to be working now | 16:33 |
| jeblair | dansmith: nope. that was easy. :) | 16:33 |
| dansmith | jeblair: okay, weird.. anyway, thanks :) | 16:33 |
| *** amotoki_ has quit IRC | 16:33 | |
| *** thuc has joined #openstack-infra | 16:34 | |
| russellb | fyi, just pushed a novaclient release, ping me if it breaks the world | 16:34 |
| krtaylor | jeblair, sounds good, I was just thinking that there is a lot of experience growing here that we can share between ourselves and offload some of the load from the cores | 16:34 |
| jeblair | dansmith: if it was just the previous meeting running over, we added a feature to meetbot that lets anyone end the meeting after 1 hour | 16:34 |
| dansmith | jeblair: right, we were trying that and it wasn't working, hence my concern | 16:34 |
| jeblair | krtaylor: yay! | 16:34 |
| *** thuc has quit IRC | 16:35 | |
| jeblair | dansmith: maybe it wasn't quite one hour to the second. :) because of course, being programmers, i'm pretty sure that's what we did. :) | 16:35 |
| dansmith | jeblair: heh | 16:35 |
| *** thuc has joined #openstack-infra | 16:35 | |
| *** yolanda_ has quit IRC | 16:35 | |
| *** thuc has quit IRC | 16:36 | |
| anteaya | russellb: thanks for the heads up | 16:36 |
| *** thuc has joined #openstack-infra | 16:36 | |
| *** david_lyle_ has quit IRC | 16:36 | |
| *** thuc has quit IRC | 16:37 | |
| *** thuc_ has joined #openstack-infra | 16:38 | |
| * mordred wants a pod of ice cream | 16:38 | |
| jeblair | ttx: ^ excellent suggestion for design summit pods | 16:38 |
| krtaylor | mordred, +1 | 16:39 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch cluster members https://review.openstack.org/76051 | 16:39 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update logstash doc for an elasticsearch cluster https://review.openstack.org/76574 | 16:39 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Move primary elasticsearch discover node https://review.openstack.org/76575 | 16:39 |
| *** geekinutah1 is now known as geekinutah | 16:40 | |
| *** atiwari has joined #openstack-infra | 16:40 | |
| russellb | mordred: +1 | 16:41 |
| russellb | brilliant | 16:41 |
| *** rossella_s has joined #openstack-infra | 16:41 | |
| *** chuck__ has joined #openstack-infra | 16:43 | |
| *** shardy is now known as shardy_afk | 16:43 | |
| *** chuck__ has quit IRC | 16:44 | |
| *** CaptTofu has quit IRC | 16:45 | |
| *** david-lyle has joined #openstack-infra | 16:46 | |
| *** CaptTofu has joined #openstack-infra | 16:51 | |
| jeblair | fungi: you filed a zuul bug yesterday didn't you? i'm having trouble finding it | 16:52 |
| jeblair | fungi: (i see the one from anteaya you pointed me to this morning) | 16:52 |
| fungi | jeblair: i think someone else filed it against openstack-ci, then i triaged, added log details and moved it to zuul... finding | 16:53 |
| fungi | jeblair: https://launchpad.net/bugs/1284842 | 16:54 |
| jeblair | fungi: thanks | 16:54 |
| jeblair | fungi: cool, i think they are related. | 16:55 |
| *** hashar has quit IRC | 16:56 | |
| *** cadenzajon has joined #openstack-infra | 16:56 | |
| *** coolsvap has joined #openstack-infra | 16:56 | |
| fungi | jeblair: all's the better! | 16:57 |
| *** andre__ has quit IRC | 16:57 | |
| fungi | in completely unrelated news, any bets on how long before someone proposes a new stackforge project for an openstack haxe sdk? http://haxe.org/ | 16:58 |
| fungi | it looks surprisingly organized for a language i've never heard of | 16:59 |
| jeblair | fungi: it's a trap | 17:00 |
| davidlenwell | So here is a question .. how do I change the topic on an already commited review without changing something? to satisfy this? http://lists.openstack.org/pipermail/openstack-dev/2014-February/028130.html | 17:00 |
| davidlenwell | trying to avoid the annoyance that i submited that patch 5 days before monty wrote that.. and just get things the way you need them so we can move on. | 17:01 |
| fungi | davidlenwell: i think we can probably grandfather in any reviews to add new projects which are already in good shape this week, but if it ends up needing a new patchset for any reason then please try to remember to update the topic when you do that | 17:02 |
| *** gokrokve has quit IRC | 17:02 | |
| davidlenwell | fungi: will do | 17:03 |
| *** gokrokve has joined #openstack-infra | 17:03 | |
| davidlenwell | SergeyLukjanov -1'd it already | 17:03 |
| davidlenwell | what do I need to do from this point? | 17:03 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 17:04 | |
| davidlenwell | I fully respect the backlog and the load you guys are under.. that is why I am working to hire you another hand.. but I do need to tell my team something | 17:04 |
| *** gyee has joined #openstack-infra | 17:04 | |
| *** e0ne_ has quit IRC | 17:05 | |
| jeblair | davidlenwell: what's the review? | 17:06 |
| *** saju_m has joined #openstack-infra | 17:06 | |
| davidlenwell | https://review.openstack.org/#/c/75226/ | 17:06 |
| davidlenwell | jeblair: this is the one with the irc problems last week | 17:07 |
| anteaya | okay time for me to run away for several hours | 17:07 |
| *** gokrokve has quit IRC | 17:08 | |
| anteaya | two failures in the gate, both on grenade both for different reasons, not sure exactly what is taking down the cinder patch but horizon has a cliff dependency issue | 17:08 |
| anteaya | so for my eyes, no explosions from novaclient | 17:08 |
| *** freyes has quit IRC | 17:08 | |
| *** sabari_ has joined #openstack-infra | 17:09 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add puppetdb server to cacti https://review.openstack.org/76583 | 17:09 |
| jeblair | davidlenwell: +2 from me | 17:09 |
| davidlenwell | thanks jeblair.. | 17:09 |
| davidlenwell | who can +a it? or do I have to wait till new project friday ? | 17:10 |
| fungi | davidlenwell: waits until friday | 17:10 |
| davidlenwell | okay .. putting my patience hat on | 17:10 |
| *** markmcclain has joined #openstack-infra | 17:10 | |
| *** changbl has quit IRC | 17:10 | |
| fungi | davidlenwell: i suspect SergeyLukjanov will rescind his -1 from 75226 the next time he's around, so i wouldn't worry about that | 17:11 |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add Eavesdrop bot to #openstack-oslo https://review.openstack.org/76584 | 17:11 |
| davidlenwell | we should really put something on the wiki .. explaining the irc stuff and now this stuff.. | 17:12 |
| *** oubiwann_ has quit IRC | 17:12 | |
| davidlenwell | would save review cycles | 17:12 |
| fungi | we might also want to start batching up changes like https://review.openstack.org/76584 there so as to minimize impact of meetbot restarts during heavy meeting periods | 17:12 |
| *** sabari_ has quit IRC | 17:13 | |
| fungi | at least until someone manages to tackle the open bug about implementing graceful restarts for it | 17:13 |
| jeblair | davidlenwell: it should go in the stackforge documentation | 17:14 |
| jeblair | the stackforge HOWTO in the ci docs | 17:14 |
| fungi | davidlenwell: there is already a mention in openstack-infra/config:docs/source/irc.rst but i agree copying that bit of detail to stackforge.rst would be helpful | 17:15 |
| *** Sukhdev has quit IRC | 17:16 | |
| *** moted has joined #openstack-infra | 17:19 | |
| jeblair | jgriffith: https://review.openstack.org/#/c/49755/ exposed a zuul bug; do you mind if i unabandon it for about 10 minutes for some testing? | 17:20 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/gear: Close poll loop pipes on cleanup https://review.openstack.org/76588 | 17:22 |
| openstackgerrit | Derek Higgins proposed a change to openstack-infra/gear: Throttle geard connect loop https://review.openstack.org/76589 | 17:22 |
| *** afazekas has quit IRC | 17:24 | |
| sdague | what's up with the whole queue hanging on an unallocated pep8 node? | 17:24 |
| *** coolsvap has quit IRC | 17:25 | |
| jeblair | sdague: my guess would be it's a re-enqueued job | 17:25 |
| sdague | jeblair: so we have 18 changes that are ready to merge | 17:25 |
| jeblair | sdague: i see that | 17:25 |
| *** coolsvap has joined #openstack-infra | 17:25 | |
| sdague | ok | 17:25 |
| *** alexpilotti has quit IRC | 17:25 | |
| jeblair | i will look further into it. just throwing out a guess because that's the usual answer in these situations | 17:26 |
| sdague | yeh, seems odd that other nodes are getting pep8 allocations | 17:26 |
| sdague | but not that job | 17:26 |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 17:27 | |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 17:29 | |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/config: Add NPM mirror https://review.openstack.org/68818 | 17:30 |
| sdague | and it's a grizzly change at that :) | 17:31 |
| *** cody-somerville has joined #openstack-infra | 17:31 | |
| *** dcramer___ has quit IRC | 17:31 | |
| *** vkozhukalov_ has quit IRC | 17:32 | |
| *** SumitNaiksatam has joined #openstack-infra | 17:33 | |
| wendar | anyone: I found a pattern yesterday in gate issues, and wondered if it had already been found? A specific recurring case of failure to connect to nova's git repo. | 17:34 |
| *** coolsvap has quit IRC | 17:34 | |
| *** coolsvap has joined #openstack-infra | 17:35 | |
| jeblair | sdague: i think gearman lost track of that job, and unfortunately i don't think the log level is low enough for us to why | 17:37 |
| jeblair | wendar: do you have a link to a specific failure? | 17:37 |
| sdague | jeblair: is there anyway to manually inject the job back in? | 17:37 |
| sdague | or do we just have to reset the queue manually to get anything moving | 17:37 |
| *** mgagne has quit IRC | 17:38 | |
| jeblair | sdague: not in a way that will cause zuul to recognize it, no. i think we have to dequeue that patch. | 17:38 |
| openstackgerrit | Travis Plummer proposed a change to openstack-infra/config: New project request: OpenStack Powershell CLI and SDK https://review.openstack.org/71956 | 17:38 |
| wendar | jeblair: http://logs.openstack.org/62/74662/1/gate/gate-devstack-dsvm-cells/b7878e9/logs/devstack-gate-setup-workspace-new.txt.gz | 17:38 |
| sdague | jeblair: ok, so before we do that, what do we need to change on gearman to catch this the next time? | 17:38 |
| jeblair | sdague: working on that | 17:39 |
| sdague | because I feel like we've had some much more inconsistent allocations recently, so it would be good to nail that down | 17:39 |
| jeblair | what do you mean? | 17:39 |
| sdague | like head of gate queue not getting resources until most of the rest of it did | 17:40 |
| *** krotscheck has quit IRC | 17:40 | |
| sdague | basically I'd assume allocation should be top down in the gate, and I don't always see that happening | 17:40 |
| *** pmathews has quit IRC | 17:41 | |
| jeblair | sdague: so remember when i threw out my incorrect guess about why this job was stuck? i said i thought it was a re-enqueued job | 17:41 |
| *** pmathews has joined #openstack-infra | 17:41 | |
| sdague | yeh | 17:41 |
| openstackgerrit | Chris Johnson proposed a change to openstack-infra/config: add sdks irc room to eavesdrop bot config https://review.openstack.org/76594 | 17:42 |
| sdague | I'm sniping out the top change now | 17:42 |
| jeblair | sdague: as you know, jenkins fails to complete builds for reasons of its own quite a lot | 17:42 |
| sdague | yeh | 17:42 |
| jeblair | sdague: some of those are detectable by zuul. in those cases, zuul will re-enqueue the job to give jenkins a chance at another go. | 17:43 |
| jeblair | sdague: unfortunatly, that means the jobs go to the back of the queue. we can probably fix that, but it's non-trivial | 17:43 |
| jeblair | sdague: (whereas just having it retry at the back of the queue was trivial, so we did it) | 17:43 |
| *** luqas has quit IRC | 17:43 | |
| jeblair | sdague: at any rate, that has happened 224 times today (since 00:00 UTC) | 17:44 |
| jeblair | sdague: so it's not at all suprising that you would actually see it in action | 17:44 |
| jeblair | sdague: but look at the bright side -- that's 224 false failures that we didn't have to deal with. :) | 17:44 |
| sdague | sure | 17:45 |
| *** luqas has joined #openstack-infra | 17:45 | |
| sdague | ok, that at least explains that :) | 17:45 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Set Zuul gear server logs to debug https://review.openstack.org/76596 | 17:45 |
| jeblair | and that ^ will hopefully explain the other if it happens again | 17:45 |
| sdague | any chance we could got the other direction and pop then on top of queue? | 17:45 |
| sdague | or does the datastructure not allow us to manip it that way | 17:46 |
| *** amcrn has joined #openstack-infra | 17:46 | |
| jeblair | sdague: i think the easiest think is to submit it with a high priority, but that's still a nontrivial zuul change. it's not too hard, it's just more than a one-liner. | 17:47 |
| *** ildikov_ has quit IRC | 17:48 | |
| *** dcramer___ has joined #openstack-infra | 17:48 | |
| *** oubiwann_ has joined #openstack-infra | 17:49 | |
| sdague | jeblair: ok. Well that's beyond my plate at the moment | 17:49 |
| jeblair | sdague: me too. i'll get to it, but i think we're in a pretty good position (we're way ahead of where we would be) so it's not at the top of my lis | 17:50 |
| jeblair | t | 17:50 |
| sdague | yeh, though it will be interesting to see how it will impact us during the i3 rush | 17:51 |
| *** bhuvan has joined #openstack-infra | 17:51 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 17:51 | |
| jeblair | fungi, anteaya: the loop was caused because jenkins had previously submitted the patchset. the submit counts as an approval that sticks around (it isn't reset by the 0 vote) | 17:51 |
| *** jcooley_ has quit IRC | 17:52 | |
| fungi | oh... | 17:52 |
| jeblair | so it always matches "any jenkins approval older than 72 hours" | 17:52 |
| sdague | wendar: so one of the problems with that issue you found, is it's in a log that we weren't indexing at the time | 17:53 |
| *** apevec has joined #openstack-infra | 17:53 | |
| wendar | sdague: ah-ha | 17:53 |
| sdague | so we can't build a pattern out of it to see it's likeliness | 17:53 |
| sdague | it should be in the index now, though elastic search is still in a world of hurt | 17:53 |
| *** NikitaKonovalov_ is now known as NikitaKonovalov | 17:53 | |
| wendar | sdague: yeah, the console.html was exceedingly un-unique on those failures | 17:53 |
| apevec | sdague, https://review.openstack.org/#/c/76020/2..3//COMMIT_MSG,unified - "gearman lost the job here" what does that mean? | 17:53 |
| sdague | apevec: read backscroll | 17:54 |
| sdague | wendar: yeh | 17:54 |
| sdague | so new jobs will both timestamp and index that log file | 17:54 |
| sdague | so we can use it in elastic recheck | 17:54 |
| wendar | cool | 17:54 |
| * apevec scrolls to eavesdrop | 17:54 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add notifications to #openstack-oslo channel https://review.openstack.org/76598 | 17:55 |
| fungi | apevec: in summary, for some reason as of yet unidentified (due to insufficient logging) zuul's gearman server lost track of the fact that it needed to request for a worker on the pep8 test for that change, so it was sitting at the top of the gate not clearing, and this preventing anything else from merging | 17:56 |
| *** jcooley_ has joined #openstack-infra | 17:56 | |
| apevec | fungi, thanks, so yet another gate mystery :( | 17:57 |
| *** oubiwann_ has quit IRC | 17:58 | |
| fungi | apevec: one which we'll have sufficient logging in place to diagnose when it reoccurs, in all probability | 17:58 |
| fungi | each time we add new ways to scale this system, we add complexity and expose (or sometimes introduce) new and exciting bugs | 17:59 |
| *** jpich has quit IRC | 17:59 | |
| fungi | as with any software | 17:59 |
| *** oubiwann_ has joined #openstack-infra | 18:00 | |
| *** jcooley_ has quit IRC | 18:01 | |
| *** oubiwann_ has quit IRC | 18:01 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Add notifications to #openstack-oslo channel https://review.openstack.org/76598 | 18:01 |
| *** khyati has joined #openstack-infra | 18:02 | |
| clarkb | morning | 18:02 |
| fungi | this month we've averaged 50 changes merged per day, including weekends (for projects reflected in the openstack/openstack meta-repo) | 18:03 |
| *** rossella_s has quit IRC | 18:03 | |
| clarkb | fungi: wow is it that high? | 18:03 |
| fungi | git log --pretty=fuller --date=iso | grep ^CommitDate: | cut -d' ' -f2 | cut -d- -f-2 | uniq -c | 18:04 |
| sdague | I still think our i3 capacity target is 200 merges / day. | 18:04 |
| *** mgagne has joined #openstack-infra | 18:04 | |
| fungi | sdague: agreed, though the highest day this month was "only" 145 changes merged | 18:05 |
| sdague | yeh, but we backlogged during i2 | 18:05 |
| fungi | though i2 was before some of our new scaling improvements, and in the face of additional bugs, so hopefully we can jump the hurdle this time | 18:06 |
| *** shashank__ has joined #openstack-infra | 18:06 | |
| fungi | clarkb: we have no purple shards now, and every cluster member has 11 or 12 assigned | 18:06 |
| *** tkorochka has joined #openstack-infra | 18:06 | |
| clarkb | fungi: great. I am pulling that stuff up now but I am going to guess we can go ahead and stop es on es6 | 18:07 |
| fungi | clarkb: current pending patch series is https://review.openstack.org/#/q/status:open+project:openstack-infra/config+branch:master+topic:elasticsearch,n,z | 18:07 |
| clarkb | looking now | 18:07 |
| fungi | clarkb: i'm pretty confident in the cacti one, less so in the discover node and doc update patches | 18:07 |
| *** johnthetubaguy has quit IRC | 18:08 | |
| fungi | and i think the delete change is comprehensive enough now, but i'm obviously leaving my -2 on it until we're all clear | 18:08 |
| *** andreaf has quit IRC | 18:08 | |
| *** andreaf has joined #openstack-infra | 18:10 | |
| fungi | also, xkcd.com/now is my new favorite tool for planning meetings ;) | 18:12 |
| clarkb | fungi: commented on the doc change responding to inline comments | 18:13 |
| fungi | clarkb: perfect. i'll update that asap | 18:13 |
| *** lcheng_ has joined #openstack-infra | 18:14 | |
| *** alexpilotti has joined #openstack-infra | 18:14 | |
| fungi | the docs had fallen a bit behind reality, but i was still a little unsure of my understanding there | 18:14 |
| *** dcramer___ has quit IRC | 18:15 | |
| *** luqas has quit IRC | 18:16 | |
| *** e0ne has joined #openstack-infra | 18:17 | |
| *** skraynev is now known as skraynev_afk | 18:18 | |
| clarkb | and commented on the change primary node change. Whcih needs a small tweak | 18:18 |
| *** gokrokve has joined #openstack-infra | 18:19 | |
| *** alexpilotti_ has joined #openstack-infra | 18:19 | |
| *** alexpilotti has quit IRC | 18:20 | |
| *** alexpilotti_ is now known as alexpilotti | 18:20 | |
| fungi | k | 18:20 |
| *** Ryan_Lane1 has joined #openstack-infra | 18:20 | |
| *** nati_ueno has joined #openstack-infra | 18:20 | |
| *** cody-somerville has quit IRC | 18:20 | |
| *** jcooley_ has joined #openstack-infra | 18:20 | |
| *** pdmars has quit IRC | 18:20 | |
| *** changbl has joined #openstack-infra | 18:20 | |
| *** derekh has quit IRC | 18:21 | |
| *** Ryan_Lane1 has quit IRC | 18:21 | |
| *** reed has joined #openstack-infra | 18:22 | |
| *** andreaf has quit IRC | 18:22 | |
| *** e0ne has quit IRC | 18:22 | |
| *** e0ne has joined #openstack-infra | 18:22 | |
| *** reed has quit IRC | 18:23 | |
| *** prad__ has quit IRC | 18:24 | |
| *** khyati has quit IRC | 18:24 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Ignore approvals without descriptions https://review.openstack.org/76610 | 18:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Handle builds without gearman jobs https://review.openstack.org/76611 | 18:25 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Handle builds without gearman jobs https://review.openstack.org/76611 | 18:26 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Ignore approvals without descriptions https://review.openstack.org/76610 | 18:26 |
| *** jgallard has quit IRC | 18:26 | |
| fungi | clarkb: so while i'm tweaking those, should i go ahead and stop elasticsearch on elasticsearch6 and wait for replication to settle again? | 18:27 |
| jeblair | fungi, clarkb: ^ can you speedy-review those? I'd like to go aheand and put into production since they (hopefully) fix two production problems we're seeing | 18:27 |
| clarkb | sure | 18:27 |
| fungi | jeblair: was already in the middle of doing so ;) | 18:27 |
| clarkb | fungi: I think that sounds like a great idea | 18:27 |
| *** oubiwann_ has joined #openstack-infra | 18:27 | |
| clarkb | just stopping the service should be fine, but if you are really lazy you can do it via elasticsearch-head | 18:28 |
| clarkb | I typically am on the host anyways so don't use the web gui thing | 18:28 |
| jeblair | stupid flake8 | 18:28 |
| *** oubiwann_ has quit IRC | 18:28 | |
| *** Ryan_Lane1 has joined #openstack-infra | 18:29 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Handle builds without gearman jobs https://review.openstack.org/76611 | 18:29 |
| *** pdmars has joined #openstack-infra | 18:29 | |
| fungi | clarkb: the "yellow" shards are ones which currently lack redundancy now that their counterparts on es6 are unavailable? | 18:31 |
| *** prad_ has joined #openstack-infra | 18:31 | |
| clarkb | fungi: correct, once they have been replicated from the other copy the cluster will go back to green | 18:31 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Handle builds without gearman jobs https://review.openstack.org/76611 | 18:32 |
| clarkb | fungi: those yellow ones are the ones that are recovering | 18:32 |
| jeblair | fungi: so sorry ^ :( | 18:32 |
| fungi | jeblair: was flake8 complaining about directly accessing a private method from another module? | 18:32 |
| jeblair | fungi: no it was complaining that job was never used | 18:32 |
| fungi | oh! | 18:32 |
| jeblair | fungi: but __ is extra magic and you can't test it with hasattr | 18:33 |
| fungi | right | 18:33 |
| jeblair | OH! | 18:33 |
| *** harlowja_away is now known as harlowja | 18:34 | |
| jeblair | jhesketh added worker information to the data model though, and i think that would probably be a good place to put that information so we can stop using __ | 18:34 |
| jeblair | but that's for another day | 18:34 |
| clarkb | 76610 has been approved onto 11 no1 | 18:34 |
| *** saju_m has quit IRC | 18:35 | |
| *** reed has joined #openstack-infra | 18:35 | |
| *** morganfainberg_Z is now known as morganfainberg | 18:36 | |
| *** Ryan_Lane1 has quit IRC | 18:36 | |
| *** Ryan_Lane1 has joined #openstack-infra | 18:36 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Ignore approvals without descriptions https://review.openstack.org/76610 | 18:37 |
| *** nicedice has joined #openstack-infra | 18:38 | |
| jesusaurus | clarkb: i just had a thought: if i push a change to gerrit, then quickly make a small change and push a second changeset to the change, will zuul abort the tests on the first changeset? | 18:39 |
| jesusaurus | that probably doesnt happen /too/ often... | 18:39 |
| clarkb | and 76611 approved | 18:40 |
| clarkb | jesusaurus: it will | 18:40 |
| jesusaurus | ah, cool | 18:40 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add the bugdaystats to openstack-infra https://review.openstack.org/69489 | 18:40 |
| *** jpeeler has quit IRC | 18:40 | |
| *** jpeeler has joined #openstack-infra | 18:40 | |
| * SergeyLukjanov is to slow to review Jim's patches to zuul :) | 18:41 | |
| clarkb | SergeyLukjanov: oh sorry | 18:42 |
| clarkb | jeblair put it on the asap list so I didn't wait to approve | 18:42 |
| SergeyLukjanov | clarkb, nope, everything is ok ;) | 18:42 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Handle builds without gearman jobs https://review.openstack.org/76611 | 18:43 |
| Ryan_Lane1 | so it seems you're slowly replacing jenkins with zuul | 18:43 |
| Ryan_Lane1 | any idea on when that'll be completely replaced? I have basically no love for jenkins :) | 18:44 |
| jeblair | Ryan_Lane1: yep. the end is in sight. | 18:44 |
| clarkb | jeblair: log gearman client has a cpu pegged onlogstash.o.o is that possibly related to the stats reporting? | 18:44 |
| Ryan_Lane1 | I'd love to have a python alternative | 18:44 |
| *** davidhadas has quit IRC | 18:44 | |
| clarkb | Ryan_Lane1: if turbohipster meets your needs you can do it today | 18:44 |
| jeblair | Ryan_Lane1: jhesketh wrote turbo-hipster which is python-based job runner for zuul | 18:45 |
| clarkb | Ryan_Lane1: mikal and jhesketh and mattoliverau are doing it with zuul in rax land for third part db tests | 18:45 |
| Ryan_Lane1 | ah. is this what openstack will go with in the future? | 18:45 |
| jeblair | Ryan_Lane1: close but not exactly as it is now... | 18:45 |
| jeblair | Ryan_Lane1: we still want to use jenkins-job-builder | 18:45 |
| openstackgerrit | A change was merged to openstack-infra/config: Set Zuul gear server logs to debug https://review.openstack.org/76596 | 18:45 |
| jeblair | Ryan_Lane1: (because of the templating, not because we love xml) | 18:45 |
| *** chandan_kumar has quit IRC | 18:46 | |
| Ryan_Lane1 | heh. docs for turbohipster are basically non-existent | 18:46 |
| * Ryan_Lane1 nods | 18:46 | |
| *** valentinbud has quit IRC | 18:46 | |
| jeblair | Ryan_Lane1: so we'll have something like turbo-hipster (maybe turbo-hipster if it wants to grow these features) that will read jjb-style yaml and do simple shell-based builds | 18:46 |
| *** esker has quit IRC | 18:47 | |
| jeblair | Ryan_Lane1: but before we get there, we're working on swift-based artifact archiving | 18:47 |
| Ryan_Lane1 | I think I'll wait a while and take a look again. I'd like to replace jenkins with something better, but I'm also not using gerrit | 18:47 |
| jeblair | Ryan_Lane1: so that our logs and tarballs can be uploaded to swift, replacing the jenkins scp stuff. | 18:47 |
| Ryan_Lane1 | so it'll be quite a bit of work for me | 18:47 |
| Ryan_Lane1 | ah. nice | 18:47 |
| *** esker has joined #openstack-infra | 18:48 | |
| *** krotscheck has joined #openstack-infra | 18:48 | |
| fungi | Ryan_Lane1: the solum devs are also eyeballing zuul as a basic work engine, potentially completely detached from gerrit scenarios, so there's a good chance they're be incented to start hanging new sorts of triggers in there | 18:48 |
| *** khyati has joined #openstack-infra | 18:49 | |
| jeblair | Ryan_Lane1: also some python devs are considering attaching it to rietveld | 18:49 |
| jeblair | Ryan_Lane1: http://legacy.python.org/dev/peps/pep-0462/ | 18:50 |
| *** Mithrandir has quit IRC | 18:50 | |
| *** Mithrandir has joined #openstack-infra | 18:50 | |
| *** mpanetta has joined #openstack-infra | 18:51 | |
| Ryan_Lane1 | I'd need github, but hopefully in the future phabricator | 18:52 |
| *** dcramer___ has joined #openstack-infra | 18:52 | |
| Ryan_Lane1 | there's no chance I'll introduce gerrit to another organization | 18:52 |
| *** esker has quit IRC | 18:52 | |
| Ryan_Lane1 | that was painful enough at wikimedia | 18:52 |
| mpanetta | Hey guys, anyone that can help me? I am having issues signing the contributor agreement. It is giving me this error: | 18:52 |
| *** julim_ has quit IRC | 18:53 | |
| clarkb | Ryan_Lane1: :P adding a github callback trigger to zuul shouldn't be terrible | 18:53 |
| mpanetta | The request could not be completed <blah blah> Make sure you have joined the foundation... | 18:53 |
| mpanetta | But I am logged in, so I must have joined no? | 18:53 |
| clarkb | mpanetta: no, login is via launchpad openid | 18:53 |
| mpanetta | Ah ok | 18:53 |
| fungi | mpanetta: https://wiki.openstack.org/wiki/CLA-FAQ#When_trying_to_sign_the_new_ICLA_and_include_contact_information.2C_why_am_I.27m_getting_an_error_message_saying_that_my_E-mail_address_doesn.27t_correspond_to_a_Foundation_membership.3F | 18:53 |
| mpanetta | fungi: Thanks! | 18:54 |
| clarkb | mpanetta: https://www.openstack.org/join/ is where you can join the foundation | 18:54 |
| mpanetta | Thank you clarkb :) | 18:54 |
| SergeyLukjanov | folks, could you, please, take a look at https://review.openstack.org/#/c/74309/? it's direct-release hard-coded list removal from jeepyb | 18:54 |
| *** pmathews has quit IRC | 18:55 | |
| sdague | what do we need to get an elastic-recheck-core made? | 18:55 |
| sdague | in gerrit | 18:55 |
| *** shashank__ has quit IRC | 18:55 | |
| fungi | sdague: is it mentioned in a gerrit acl? | 18:55 |
| SergeyLukjanov | sdague, fix acls in infra/config and ask infra-root to add you to the new group | 18:56 |
| *** pblaho has quit IRC | 18:56 | |
| sdague | fungi: I don't think so | 18:56 |
| fungi | sdague: groups get automagically created in gerrit if they're part of an acl | 18:56 |
| fungi | sdague: if they're not used by any gerrit acl, what is the use case? | 18:56 |
| jeblair | sdague: it already exists | 18:56 |
| jeblair | sdague: https://review.openstack.org/#/admin/groups/218,members | 18:56 |
| SergeyLukjanov | sdague, and acls are correct | 18:56 |
| SergeyLukjanov | label-Approved = +0..+1 group elastic-recheck-core | 18:57 |
| SergeyLukjanov | label-Code-Review = -2..+2 group elastic-recheck-core | 18:57 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Remove hardcoded direct-release project list https://review.openstack.org/74309 | 18:57 |
| SergeyLukjanov | + refs/tags for elastic-recheck-ptl | 18:57 |
| sdague | oh, right, so we just need to add folks | 18:57 |
| sdague | my bad | 18:57 |
| *** mpanetta has left #openstack-infra | 18:57 | |
| *** CaptTofu has quit IRC | 18:57 | |
| jeblair | sdague: i expect that you should have a field on that page to add people; let me know if that's not the case. | 18:58 |
| *** thomasem has joined #openstack-infra | 18:58 | |
| *** CaptTofu has joined #openstack-infra | 18:58 | |
| *** melwitt has joined #openstack-infra | 18:58 | |
| sdague | yep, I do | 18:58 |
| sdague | it was just all group oriented before | 18:58 |
| *** shashank__ has joined #openstack-infra | 19:00 | |
| *** bhuvan_ has joined #openstack-infra | 19:01 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch cluster members https://review.openstack.org/76051 | 19:01 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update logstash doc for an elasticsearch cluster https://review.openstack.org/76574 | 19:01 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Move primary elasticsearch discover node https://review.openstack.org/76575 | 19:01 |
| jeblair | i plan on restarting zuul when the current gate job finishes | 19:02 |
| Ryan_Lane1 | clarkb: yeah, but I don't necessarily want to put the work in for github if we're going to move away from it | 19:02 |
| *** CaptTofu has quit IRC | 19:02 | |
| clarkb | ah | 19:02 |
| jeblair | Ryan_Lane1: if a github trigger existed, we would use it so that we could CI the repos of our dependencies | 19:03 |
| Ryan_Lane1 | oh, interesting. | 19:03 |
| jeblair | Ryan_Lane1: i don't know if that interests you, but ther's certainly use for a github trigger even for orgs that don't use it as their primary | 19:03 |
| fungi | Ryan_Lane1: the solum devs were actually discussing that as possibly one of their first patches to zuul anyway | 19:03 |
| Ryan_Lane1 | that would be awesome | 19:03 |
| *** fbo is now known as fbo_away | 19:03 | |
| * Ryan_Lane1 wants | 19:04 | |
| clarkb | fungi: they actually wrote an amqp trigger but haven't upstremaed | 19:04 |
| Ryan_Lane1 | :( | 19:04 |
| openstackgerrit | A change was merged to openstack/requirements: Allow projects to use oslo.vmware https://review.openstack.org/75539 | 19:04 |
| Ryan_Lane1 | heh | 19:04 |
| jeblair | the queue is clearing, i'll be restarting zuul in a second | 19:04 |
| * jeblair watches in #openstack-merges | 19:05 | |
| Ryan_Lane1 | well, that does interest me, but I don't want to lick that cookie | 19:05 |
| Ryan_Lane1 | since I may not have time to actually do it | 19:05 |
| *** bhuvan has quit IRC | 19:05 | |
| Ryan_Lane1 | man, there's someone I really need to hire, because he'd be able to do this in a matter of hours | 19:05 |
| jeblair | Ryan_Lane1: maybe an openstack company can hire him to do this. :) | 19:06 |
| Ryan_Lane1 | notice I didn't mention who it was ;) | 19:06 |
| *** tkorochka has quit IRC | 19:06 | |
| jeblair | Ryan_Lane1: as long as they are hired by someone to work on zuul, i don't care who. :) | 19:06 |
| *** ociuhandu has quit IRC | 19:07 | |
| Ryan_Lane1 | :D | 19:07 |
| *** dcramer___ has quit IRC | 19:07 | |
| dhellmann | jeblair, SergeyLukjanov: I posted a couple of questions on https://review.openstack.org/#/c/76381/ | 19:07 |
| dkranz | The new log links in jenkins to tempest.conf.gz are not setting the uncompress headers. Should I file a bug? | 19:07 |
| jeblair | Zuul is restarted | 19:07 |
| jeblair | dkranz: in devstack-gate they should be moved to 'tempest.conf.txt' and then gzipped | 19:08 |
| dkranz | jeblair: ok, I'll take a look. | 19:09 |
| *** mfer has quit IRC | 19:09 | |
| fungi | jeblair: you said mergers still need restarting after the main zuul process, right? | 19:12 |
| jeblair | fungi: so i did | 19:12 |
| fungi | if so, they don't seem to have been yet | 19:12 |
| jeblair | that should really be next on my list | 19:12 |
| fungi | or i can restart them if you like | 19:12 |
| clarkb | jeblair: fungi: why is that the case? because the mergers don't rejoin the gaerd? | 19:13 |
| jeblair | fungi: i'm on it | 19:13 |
| jeblair | clarkb: they only re-register one of their functions | 19:13 |
| openstackgerrit | David Kranz proposed a change to openstack-infra/devstack-gate: Rename tempest.conf so it is gz'ed properly https://review.openstack.org/76622 | 19:14 |
| jeblair | also, the pidfile is wrong so the init script doesn't work. | 19:14 |
| clarkb | gotcha | 19:14 |
| jeblair | they are both running now | 19:14 |
| *** freyes has joined #openstack-infra | 19:14 | |
| *** esker has joined #openstack-infra | 19:15 | |
| fungi | we have changes with completion estimates now, so looks good | 19:15 |
| *** esker has quit IRC | 19:15 | |
| apevec | fungi, your blessing was asked for the evil patch https://review.openstack.org/76058 (workaround for grenade job in stable/havana) | 19:16 |
| *** prad_ has quit IRC | 19:16 | |
| *** andreaf has joined #openstack-infra | 19:16 | |
| fungi | apevec: i proposed https://review.openstack.org/76280 instead | 19:16 |
| apevec | ah thanks, I missed that | 19:17 |
| fungi | apevec: we've got a week left on grizzly support, so it didn't seem worthwhile for people to waste any more time fixing upgrade testing on it | 19:17 |
| wendar | jeblair/clarkb: who has access to update the "official tags" list for the Nova project? Only the general OpenStack admins? | 19:17 |
| apevec | I'll add link and abandon evil patch then | 19:17 |
| fungi | apevec: well, mine hasn't exactly been +2'd either | 19:17 |
| fungi | it's an alternate approach, admittedly distasteful, but pragmatic | 19:17 |
| *** sarob has joined #openstack-infra | 19:18 | |
| fungi | apevec: i'm fine with 76058 if it works | 19:18 |
| clarkb | wendar: git tags? usually project ptls and possibly a small group of folks delegated with that responsibility | 19:18 |
| sdague | wendar: I think nova-drivers | 19:18 |
| *** sdake_ has quit IRC | 19:18 | |
| jeblair | dhellmann: responded. i think we have two choices: we can proceed with the devstack-gate plan but that only works with py27 for now (until we make some infra improvements on test nodes) | 19:18 |
| wendar | clarckb: Launchpad bug tags. | 19:18 |
| *** sdake_ has joined #openstack-infra | 19:18 | |
| *** prad has joined #openstack-infra | 19:18 | |
| dhellmann | jeblair: that's acceptable | 19:18 |
| wendar | I mean clarkb ^ :) | 19:19 |
| clarkb | wendar: I think sdague is correct | 19:19 |
| jeblair | dhellmann: or if py26-33 testing is more important, then we can go with an asymmetric gate (or even just non-voting tests) on the regular unit test nodes and forego the devstack-gate git repo setup | 19:19 |
| wendar | sdague: In Launchpad, the only "Adminstrator" in the Nova Bugs team is "OpenStack Administrators". | 19:19 |
| apevec | fungi, that's a good question, not sure that after 76058 is fixed, it won't break in some other place | 19:20 |
| dhellmann | jeblair: the idea is to prevent breaking the other projects, which really means API changes. We do have unit tests for the lib by itself to verify we don't break py26 and py33, I was just being overly cautious | 19:20 |
| wendar | But then, maybe any team member has access to update that list. | 19:20 |
| apevec | e.g. this popped after skipping swift devstack exercises | 19:20 |
| *** davidhadas has joined #openstack-infra | 19:21 | |
| dhellmann | jeblair: and if we'll have the git repo checkout stuff working on unit test nodes at some point, we can add more test jobs then | 19:21 |
| fungi | apevec: yep, i put that previous attempt through, which was what then prompted me to propose just dropping that job a week ahead of schedule | 19:21 |
| *** dcramer___ has joined #openstack-infra | 19:21 | |
| jeblair | dhellmann: okay. i'll redo that change appropriately. i'll just hard-code py27 for now. | 19:22 |
| fungi | apevec: i say we merge 76058 first, since it's already written, and then if it still doesn't help we go forward with 76280 and not waste more developer time on it | 19:22 |
| dhellmann | jeblair: passing it to run_cross_tests.sh? | 19:22 |
| jeblair | dhellmann: yeah, so the arg will already be there, and we can plumb it into jjb when we're actually ready to use it. | 19:22 |
| dhellmann | jeblair: makes sense | 19:22 |
| krotscheck | clarkb: Would you mind pinging me once that permissions patch lands? | 19:22 |
| *** bhuvan has joined #openstack-infra | 19:23 | |
| dhellmann | jeblair: I'll rework run_ross_tests.sh to take those args | 19:23 |
| krotscheck | clarkb: I'm about to go eyeballs deep into OpenID, so... | 19:23 |
| apevec | fungi, ok | 19:23 |
| jeblair | dhellmann: oh, do you need the name of the project or just the path? | 19:23 |
| dhellmann | jeblair: oh, can you pass the root for the repos, too? that way we can use run_cross_tests.sh locally | 19:23 |
| dhellmann | heh | 19:23 |
| apevec | sdague, ^^^ that was fungi re. 76058 | 19:23 |
| dhellmann | paths are probably better | 19:23 |
| jeblair | cool | 19:23 |
| clarkb | krotscheck: sure, it hasn't even been proposed yet, but I suppose I should do that really quick since everyone wants it | 19:23 |
| *** thuc_ has quit IRC | 19:24 | |
| fungi | apevec: sdague: i +1'd and updated the change accordingly | 19:24 |
| dhellmann | jeblair: does my hacky two-calls-to-tox approach still make sense? | 19:24 |
| openstackgerrit | A change was merged to openstack-infra/config: Add new elasticsearch cluster members to cacti https://review.openstack.org/76573 | 19:24 |
| fungi | clarkb: did you do something earlier to put logstash-worker05 and 09 back into service? (i noticed last night they were missing in the status monitors) | 19:25 |
| jeblair | dhellmann: yes. one downside to that is we're missing some of the sanity checks that the run-unittests script gives us, but since these are essentially duplicate tests as far as that goes, we're probably okay. | 19:25 |
| jeblair | dhellmann: (sanity checks like did it oom, or sudo, or run zero tests) | 19:25 |
| jeblair | dhellmann: actually, that last one might be a good one to have in run_cross_tests | 19:25 |
| *** thuc has joined #openstack-infra | 19:26 | |
| clarkb | fungi: I did `sudo restart logstash-indexer`. https://review.openstack.org/#/c/75966/ should automate that for us | 19:26 |
| *** CaptTofu has joined #openstack-infra | 19:26 | |
| *** vkozhukalov_ has joined #openstack-infra | 19:26 | |
| *** bhuvan_ has quit IRC | 19:27 | |
| clarkb | fungi: that does a simple curl query against the cluster and runs jq over it to determine if the local node is a member of the cluster. If it isn't the service is restarted | 19:27 |
| fungi | clarkb: is there an upstream bug which 75966 is working around? | 19:27 |
| clarkb | fungi: I haven't gone digging in logstash and elasticsearch's jiras but either logstash or the es client lib is at fault. It should try reconnecting imo | 19:28 |
| fungi | just curious whether we've been good citizens and reported it upstream, at least | 19:28 |
| clarkb | I haven't had time yet, but should do that | 19:28 |
| clarkb | but jira :P | 19:28 |
| fungi | yuck, agreed | 19:28 |
| fungi | clarkb: you should ask olaph to help | 19:28 |
| dhellmann | jeblair: I would hate to complicate that script by making it know how to do this dance | 19:28 |
| clarkb | ha | 19:28 |
| clarkb | I might be able to con someon in their irc channels to do it for me too | 19:29 |
| dhellmann | jeblair: I could put a tox target in the remote project that knows what to do, but that also feels gross | 19:29 |
| jeblair | dhellmann: yeah. maybe we can refactor run-unittests in the future to have phases we can call externally | 19:29 |
| dhellmann | jeblair: what if we refactor run-unittestes.sh into run-unittests.sh and check-unittest-issues.sh or something | 19:29 |
| dhellmann | haha | 19:30 |
| dhellmann | jeblair: great minds... | 19:30 |
| jeblair | we're on a roll today | 19:30 |
| dhellmann | indeed | 19:30 |
| *** oubiwann_ has joined #openstack-infra | 19:30 | |
| jeblair | i think we may need more git servers | 19:31 |
| dhellmann | so should I go ahead and refactor now, or copy the relevant bits into my new script for the time being? | 19:31 |
| fungi | jeblair: we can make more | 19:31 |
| fungi | jeblair: or do we just need bigger git servers? | 19:31 |
| jeblair | dhellmann: i'd recommend copying for now. that one is is a fairly easy code block that you can just run after the real tox run. | 19:31 |
| dhellmann | jeblair: ack | 19:31 |
| clarkb | jeblair: more git servers for the mirror? | 19:32 |
| jeblair | fungi: i think we found the ram/cpu sweet-spot last time with benchmarking. of course, they are probably standard, not performance nodes. | 19:32 |
| jeblair | clarkb: yes. | 19:32 |
| clarkb | they are standard | 19:32 |
| fungi | yeah we're spiking up to 50mbps on them, looks like | 19:32 |
| jeblair | we should probably add one then take each of the other 4 out and replace them. | 19:33 |
| fungi | oh, and we are tapping out cpu on them occasionally as well | 19:33 |
| fungi | with load averages spiking up to ~40 on zuul restart | 19:33 |
| *** dcramer___ has quit IRC | 19:34 | |
| clarkb | are there pvhvm cenots images? | 19:34 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add oslo.test integration test https://review.openstack.org/76381 | 19:34 |
| clarkb | if so we can go that route too | 19:34 |
| openstackgerrit | Michael Krotscheck proposed a change to openstack-infra/storyboard-webclient: MVP Storyboard Client https://review.openstack.org/70897 | 19:34 |
| fungi | clarkb: ckecking now | 19:34 |
| dhellmann | jeblair: looks good | 19:35 |
| fungi | clarkb: yep... CentOS 6.5 (PVHVM) | 19:36 |
| fungi | clarkb: i'll whip up a change to add a new pvhvm performance node, and then a second change to put it into the haproxy pool | 19:36 |
| fungi | jeblair: ^ | 19:36 |
| jeblair | fungi: awesome, thx | 19:36 |
| clarkb | sounds good | 19:36 |
| *** jp_at_hp has quit IRC | 19:37 | |
| *** bhuvan has quit IRC | 19:37 | |
| jeblair | so the new graph tells us that zuul would be running 1000 simultaneous jobs now if it could, but is currently only able to run a bit more than 500 of them. | 19:37 |
| fungi | clarkb: on the es front, we're down to 2 remaining shards being replicated i think | 19:38 |
| fungi | clarkb: once those finish, we stop elasticsearch on es5? | 19:38 |
| *** coolsvap1 has joined #openstack-infra | 19:38 | |
| *** dstanek_afk has joined #openstack-infra | 19:38 | |
| *** unicell1 has joined #openstack-infra | 19:38 | |
| clarkb | fungi: once those finish and the cluste rshows green | 19:39 |
| clarkb | bigdesk will update its color | 19:39 |
| fungi | yeah | 19:40 |
| fungi | i assumed as much | 19:40 |
| fungi | fancy, fancy colors | 19:40 |
| *** wchrisj_ has joined #openstack-infra | 19:40 | |
| *** stevebaker has joined #openstack-infra | 19:40 | |
| *** dprince_ has joined #openstack-infra | 19:41 | |
| *** vkozhukalov_ has quit IRC | 19:41 | |
| *** moted_ has joined #openstack-infra | 19:41 | |
| *** julim has joined #openstack-infra | 19:41 | |
| *** mbacchi_ has joined #openstack-infra | 19:42 | |
| *** julienvey1 has joined #openstack-infra | 19:42 | |
| *** miqui_ has joined #openstack-infra | 19:42 | |
| *** smarcet1 has joined #openstack-infra | 19:42 | |
| apevec | fungi, mordred - why do have trove in stable/havana requirements-integration? http://logs.openstack.org/09/62209/1/check/check-requirements-integration-dsvm/aceddc0/console.html.gz#_2014-02-18_17_30_41_294 | 19:43 |
| *** dprince_ has quit IRC | 19:43 | |
| apevec | trove stable/havana is not maintained https://github.com/openstack/trove/commits/stable/havana | 19:44 |
| *** dstanek has quit IRC | 19:44 | |
| *** dstanek_afk is now known as dstanek | 19:44 | |
| fungi | apevec: presumably they requested it as they were working toward incubation | 19:44 |
| apevec | it's not part of stable-maint in Havana | 19:44 |
| apevec | fungi, right, but they got integrate at Havana GA i.e. will be part of stable release in stable/icehouse cycle | 19:45 |
| fungi | apevec: if the goal is to only include projects which are under stable-maint in requirements integration, then that probably merits wider discussion | 19:45 |
| *** dhellmann is now known as dhellmann_ | 19:45 | |
| apevec | like Ceilo and Heat Grizzly were not | 19:45 |
| clarkb | fungi: you can also `curl -XGET http://localhost:9200/_cluster/status?pretty=true` if you want to query it directly | 19:45 |
| fungi | clarkb: oh, cool! | 19:45 |
| openstackgerrit | Paul Michali proposed a change to openstack/requirements: Update requests to 2.1.0 and add httmock to tests https://review.openstack.org/75296 | 19:45 |
| apevec | fungi, somebody should actively work on stable/* branch and that's not happening for trove stable/havana | 19:46 |
| apevec | e.g. ceilo and heat team did work on stable/grizzly | 19:46 |
| fungi | apevec: i believe currently the list of projects which are in requirements integration uses teh same list as the projects which wish to receive requirements sync updates, so there are a number of projects which are not official or even curently incubated in the list | 19:46 |
| *** markmcclain has quit IRC | 19:47 | |
| *** krotscheck has quit IRC | 19:47 | |
| *** reed has quit IRC | 19:47 | |
| *** changbl has quit IRC | 19:47 | |
| *** coolsvap has quit IRC | 19:47 | |
| *** moted has quit IRC | 19:47 | |
| *** gyee has quit IRC | 19:47 | |
| *** dprince has quit IRC | 19:47 | |
| *** rfolco has quit IRC | 19:47 | |
| *** wchrisj has quit IRC | 19:47 | |
| *** ryanpetrello has quit IRC | 19:47 | |
| *** sdake has quit IRC | 19:47 | |
| *** smarcet has quit IRC | 19:47 | |
| *** mbacchi has quit IRC | 19:47 | |
| *** che-arne has quit IRC | 19:47 | |
| *** bogdando has quit IRC | 19:47 | |
| *** yassine has quit IRC | 19:47 | |
| *** miqui has quit IRC | 19:47 | |
| *** vishy has quit IRC | 19:47 | |
| *** unicell has quit IRC | 19:47 | |
| *** vponomaryov has quit IRC | 19:47 | |
| *** Hunner has quit IRC | 19:47 | |
| *** bradm has quit IRC | 19:47 | |
| *** akscram has quit IRC | 19:47 | |
| *** moted_ is now known as moted | 19:47 | |
| fungi | apevec: but i recommend following up with hub_cap or SlickNik or one of the other trove heavies on this topic | 19:47 |
| fungi | apevec: if they don't want to support their stable/havana branch, then it's a one-line patch to remove them from the stable/havana branch of openstack/requirements:projects.txt | 19:48 |
| *** vkozhukalov has joined #openstack-infra | 19:48 | |
| apevec | ok, I'll check w/ trove team | 19:49 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Optionally give mysql user all global privs. https://review.openstack.org/76634 | 19:49 |
| sdague | jeblair: is it possible for you to update the er graph with background colors at warning and oh crap thresholds | 19:49 |
| apevec | this is blocking requirements stable/havana so should be fixed one way or the other | 19:49 |
| *** Ajaeger has joined #openstack-infra | 19:49 | |
| fungi | apevec: if they're unresponsive, we can certainly make an in absentia call to just remove them anyway | 19:49 |
| Ajaeger | Infra team, do we have some jenkins that has not been updated? | 19:50 |
| fungi | Ajaeger: jobs which run on workers with names including "hpcloud-az2" tend to often be days behing on image updates because of a current problem in that provider | 19:51 |
| fungi | Ajaeger: you can look at the top of the console log to see the name of the worker where a job ran | 19:51 |
| apevec | fungi, there was an attempt to fix their havana branch https://review.openstack.org/#/c/75386 | 19:51 |
| Ajaeger | fungi, so bare-precise-hpcloud-az2-1677190 is old? | 19:52 |
| apevec | but ihrachys just gave up when issues started piling up | 19:52 |
| fungi | Ajaeger: well, the vm is very new, but it was built from a nightly image which hasn't updated in a while. i'll find out how long | 19:52 |
| *** cadenzajon_ has joined #openstack-infra | 19:52 | |
| *** gyee has joined #openstack-infra | 19:53 | |
| Ajaeger | The change I'm missing is infra change Ia8f60c8a4b9d1b18583366d83ddb82dc61bff9f5 which was merged on Monday | 19:53 |
| *** andreaf has quit IRC | 19:53 | |
| fungi | Ajaeger: just over 6 days oldf | 19:53 |
| fungi | old | 19:53 |
| *** krotscheck has joined #openstack-infra | 19:53 | |
| *** sdake has joined #openstack-infra | 19:54 | |
| *** rfolco has joined #openstack-infra | 19:54 | |
| Ajaeger | It changed the checkbuild link from logs to docs-draft | 19:54 |
| *** reed has joined #openstack-infra | 19:54 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1285323 https://review.openstack.org/76635 | 19:54 |
| *** yassine has joined #openstack-infra | 19:54 | |
| clarkb | Ajaeger: I don't think that depends on the slave | 19:54 |
| clarkb | ya thats a zuul config. Do we have puppet running on zuul.o.o? | 19:54 |
| Ajaeger | clarkb: Ah. So, why does https://review.openstack.org/#/c/76096/ show as link for the checkbuild logs... | 19:55 |
| fungi | clarkb: yes, but also Last reconfigured: Wed Feb 26 2014 19:06:46 GMT+0000 (UTC) | 19:55 |
| Ajaeger | Ooops, I see that for most recent builds | 19:55 |
| *** cadenzajon has quit IRC | 19:56 | |
| Ajaeger | It's fine here: https://review.openstack.org/#/c/75978/ | 19:56 |
| fungi | clarkb: and the success-pattern from that change appears in /etc/zuul/layout.yaml on zuul.o.o | 19:56 |
| *** changbl has joined #openstack-infra | 19:57 | |
| *** jraim has quit IRC | 19:57 | |
| *** thuc has quit IRC | 19:57 | |
| *** mestery_ has joined #openstack-infra | 19:57 | |
| *** enikanorov__ has joined #openstack-infra | 19:57 | |
| Ajaeger | So, working on operatoins-guide but failing in openstack-manuals | 19:57 |
| * Ajaeger searches for a success for openstack-manuals... | 19:57 | |
| clarkb | fungi: ya I think Ajaeger has indicated it is working on newer builds | 19:57 |
| *** thuc has joined #openstack-infra | 19:57 | |
| clarkb | anything run before zuul reloaded its config would have old links | 19:57 |
| *** rossella_s has joined #openstack-infra | 19:58 | |
| *** unicell has joined #openstack-infra | 19:58 | |
| *** hogepodge is now known as 20WABDE58 | 19:58 | |
| *** hogepodge has joined #openstack-infra | 19:58 | |
| *** pmathews has joined #openstack-infra | 19:58 | |
| *** che-arne has joined #openstack-infra | 19:58 | |
| *** vponomaryov has joined #openstack-infra | 19:58 | |
| *** bogdando has joined #openstack-infra | 19:58 | |
| *** vishy has joined #openstack-infra | 19:58 | |
| *** Hunner has joined #openstack-infra | 19:58 | |
| *** bradm has joined #openstack-infra | 19:58 | |
| *** akscram has joined #openstack-infra | 19:58 | |
| *** dkehn__ has joined #openstack-infra | 19:58 | |
| *** ArxCruz_ has joined #openstack-infra | 19:58 | |
| *** jraim has joined #openstack-infra | 19:58 | |
| Ajaeger | clarkb: It seems to fail on gate-openstack-manuals-tox-doc-publish-checkbuild but works on gate-operations-guide-tox-doc-publish-checkbuild | 19:58 |
| *** jraim has quit IRC | 19:59 | |
| *** hogepodge has quit IRC | 19:59 | |
| *** pmathews has quit IRC | 19:59 | |
| *** che-arne has quit IRC | 19:59 | |
| *** bogdando has quit IRC | 19:59 | |
| *** vishy has quit IRC | 19:59 | |
| *** vponomaryov has quit IRC | 19:59 | |
| *** Hunner has quit IRC | 19:59 | |
| *** bradm has quit IRC | 19:59 | |
| *** akscram has quit IRC | 19:59 | |
| *** dripton_ has joined #openstack-infra | 19:59 | |
| openstackgerrit | Matthew Treinish proposed a change to openstack-infra/elastic-recheck: Add query for bug 1285323 https://review.openstack.org/76635 | 19:59 |
| clarkb | Ajaeger: are you sure it isn't just a timing thin? | 19:59 |
| fungi | perhaps we didn't reload the zuul config successfully between when that change merged and a few minutes ago when zuul was restarted? | 19:59 |
| *** alexpilotti_ has joined #openstack-infra | 19:59 | |
| Ajaeger | clarkb: I'm right now confused ;( | 19:59 |
| *** thuc has quit IRC | 20:00 | |
| clarkb | oh hrm, the test that doesn't have the correct link reported after the one that does | 20:00 |
| *** ruhe- has joined #openstack-infra | 20:00 | |
| *** thuc has joined #openstack-infra | 20:01 | |
| clarkb | Ajaeger: I bet it has something to do with order of matching | 20:01 |
| Ajaeger | Is my regex somehow wrong? | 20:01 |
| *** unicell1 has quit IRC | 20:01 | |
| *** mestery has quit IRC | 20:01 | |
| *** yassine has quit IRC | 20:01 | |
| *** alexpilotti has quit IRC | 20:01 | |
| *** ArxCruz has quit IRC | 20:01 | |
| *** 20WABDE58 has quit IRC | 20:01 | |
| *** ruhe has quit IRC | 20:01 | |
| *** branen has quit IRC | 20:01 | |
| *** ilyashakhat has quit IRC | 20:01 | |
| *** enikanorov_ has quit IRC | 20:01 | |
| *** dripton has quit IRC | 20:01 | |
| *** yassine has joined #openstack-infra | 20:01 | |
| *** miqui_ has quit IRC | 20:01 | |
| *** dims has quit IRC | 20:01 | |
| *** dkehn_ has quit IRC | 20:01 | |
| *** alexpilotti_ is now known as alexpilotti | 20:01 | |
| *** ruhe- is now known as ruhe | 20:01 | |
| *** mestery_ has quit IRC | 20:01 | |
| *** mestery_ has joined #openstack-infra | 20:01 | |
| *** thuc has quit IRC | 20:01 | |
| clarkb | https://review.openstack.org/#/c/73185/2/modules/openstack_project/files/zuul/layout.yaml just above where you added the new lines is an entry for the manuals | 20:01 |
| clarkb | which may be winning | 20:01 |
| *** ilyashakhat has joined #openstack-infra | 20:01 | |
| * clarkb reads zuul code | 20:01 | |
| *** dims has joined #openstack-infra | 20:01 | |
| *** thuc has joined #openstack-infra | 20:02 | |
| Ajaeger | clarkb: That would explain it! | 20:02 |
| *** SumitNaiksatam has quit IRC | 20:02 | |
| Ajaeger | I might have not noticed that it worked everywhere except in openstack-manuals... | 20:02 |
| *** miqui has joined #openstack-infra | 20:02 | |
| Ajaeger | clarkb: So, fix would be to copy the success-pattern to the entry above? | 20:03 |
| *** sabari_ has joined #openstack-infra | 20:03 | |
| clarkb | Ajaeger: maybe, I am trying to udnerstand how that is coalesced in zuul now | 20:03 |
| *** amcrn has quit IRC | 20:03 | |
| *** sabari_ is now known as sabari | 20:03 | |
| fungi | yeah, because we definitely have a variety of parameters combined from multiple matches onto a given job | 20:03 |
| *** kiall has quit IRC | 20:04 | |
| fungi | unless it's because one is a regex and the other is a literal job name? | 20:04 |
| fungi | does the literal cause it to short-circuit subsequent matches? | 20:04 |
| *** sabari is now known as 92AAAAI7R | 20:05 | |
| *** branen has joined #openstack-infra | 20:05 | |
| *** jraim has joined #openstack-infra | 20:05 | |
| *** hogepodge has joined #openstack-infra | 20:05 | |
| *** pmathews has joined #openstack-infra | 20:05 | |
| *** che-arne has joined #openstack-infra | 20:05 | |
| *** vponomaryov has joined #openstack-infra | 20:05 | |
| *** bogdando has joined #openstack-infra | 20:05 | |
| *** vishy has joined #openstack-infra | 20:05 | |
| *** Hunner has joined #openstack-infra | 20:05 | |
| *** bradm has joined #openstack-infra | 20:05 | |
| *** akscram has joined #openstack-infra | 20:05 | |
| Ajaeger | fungi, clarkb: Thanks a lot for looking into this. | 20:05 |
| *** kiall has joined #openstack-infra | 20:06 | |
| fungi | Ajaeger: thanks for bringing it to our attention... it does look like it's probably an unintentional corner-case | 20:06 |
| *** hashar has joined #openstack-infra | 20:07 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Use success-pattern for openstack-manuals https://review.openstack.org/76639 | 20:07 |
| Ajaeger | fungi: ;) | 20:07 |
| *** freyes has quit IRC | 20:07 | |
| *** cadenzajon has joined #openstack-infra | 20:07 | |
| clarkb | so the way it works is that regexes are 'metajobs' these are applied to non regex names when they are declared | 20:07 |
| clarkb | so you can just put the regex lines above the non regex lines and that should work | 20:07 |
| Ajaeger | clarkb: ok, will update the change. | 20:08 |
| openstackgerrit | afazekas proposed a change to openstack-infra/devstack-gate: Allow tempest to dump the ovs db https://review.openstack.org/76640 | 20:08 |
| *** dstufft_ has joined #openstack-infra | 20:09 | |
| *** dkehn__ is now known as dkehn_ | 20:09 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:09 | |
| *** nati_uen_ has joined #openstack-infra | 20:10 | |
| openstackgerrit | Andreas Jaeger proposed a change to openstack-infra/config: Fix success-pattern usage for openstack-manuals https://review.openstack.org/76639 | 20:10 |
| clarkb | fungi: so it doesn't short circuit but when normal job is evaluated ti can't apply the meta job stuff | 20:10 |
| Ajaeger | clarkb: here's the updated patch - hope it does the right think. | 20:10 |
| *** dripton_ is now known as dripton | 20:10 | |
| Ajaeger | s/think/thing/ | 20:11 |
| *** doddstack has joined #openstack-infra | 20:11 | |
| *** oubiwann_ has quit IRC | 20:11 | |
| *** salv-orlando_ has joined #openstack-infra | 20:11 | |
| *** Ryan_Lane2 has joined #openstack-infra | 20:12 | |
| *** rlandy has joined #openstack-infra | 20:12 | |
| *** wenlock has quit IRC | 20:12 | |
| *** CaptTofu_ has joined #openstack-infra | 20:12 | |
| *** Sukhdev has joined #openstack-infra | 20:13 | |
| *** SumitNaiksatam has quit IRC | 20:13 | |
| *** smarcet has joined #openstack-infra | 20:13 | |
| *** dkehn__ has joined #openstack-infra | 20:13 | |
| clarkb | Ajaeger: yup lgtm | 20:13 |
| clarkb | fungi: cluster looks happy now | 20:14 |
| Ajaeger | clarkb and fungi: thanks a lot! | 20:15 |
| *** ildikov_ has joined #openstack-infra | 20:15 | |
| *** dstufft has quit IRC | 20:15 | |
| *** dstufft_ is now known as dstufft | 20:15 | |
| *** amrith_ has joined #openstack-infra | 20:15 | |
| *** oubiwann_ has joined #openstack-infra | 20:17 | |
| fungi | clarkb: yep, all clear to stop services on es5 then? | 20:18 |
| *** dims has quit IRC | 20:18 | |
| *** yassine has quit IRC | 20:18 | |
| *** cadenzajon_ has quit IRC | 20:18 | |
| *** smarcet1 has quit IRC | 20:18 | |
| *** CaptTofu has quit IRC | 20:18 | |
| *** nati_ueno has quit IRC | 20:18 | |
| *** thedodd has quit IRC | 20:18 | |
| *** dkliban has quit IRC | 20:18 | |
| *** sandywalsh has quit IRC | 20:18 | |
| *** salv-orlando has quit IRC | 20:18 | |
| *** Ryan_Lane has quit IRC | 20:18 | |
| *** dkehn has quit IRC | 20:18 | |
| *** amrith has quit IRC | 20:18 | |
| *** salv-orlando_ is now known as salv-orlando | 20:18 | |
| *** amrith_ is now known as amrith | 20:18 | |
| clarkb | fungi: yup | 20:19 |
| fungi | done | 20:19 |
| *** dims has joined #openstack-infra | 20:19 | |
| *** sandywalsh has joined #openstack-infra | 20:19 | |
| *** flaper87 has quit IRC | 20:20 | |
| *** flaper87 has joined #openstack-infra | 20:20 | |
| *** afazekas has joined #openstack-infra | 20:20 | |
| fungi | clarkb: later when es4 get to the chopping block, is there anything we need to do to force a graceful master election first, or just let it sort that out on its own? | 20:20 |
| *** yassine has joined #openstack-infra | 20:20 | |
| *** cadenzajon has quit IRC | 20:21 | |
| *** cadenzajon has joined #openstack-infra | 20:21 | |
| *** ildikov_ has quit IRC | 20:21 | |
| *** ildikov_ has joined #openstack-infra | 20:21 | |
| *** dkliban has joined #openstack-infra | 20:21 | |
| *** dkliban has quit IRC | 20:21 | |
| *** dkliban has joined #openstack-infra | 20:21 | |
| clarkb | it will sort that out on its own, but let me read es docs to see if we can force a different master to be elected (I have never worried about it in the past and it has been fine as long as the network doesn't partition) | 20:21 |
| *** shashank__ has quit IRC | 20:21 | |
| clarkb | ya doesn't look like we can force a different master, its fine | 20:21 |
| *** yolanda_ has joined #openstack-infra | 20:21 | |
| *** lcheng_ has quit IRC | 20:22 | |
| *** SumitNaiksatam has joined #openstack-infra | 20:22 | |
| *** wenlock has joined #openstack-infra | 20:23 | |
| *** oubiwann_ has quit IRC | 20:23 | |
| * clarkb is going to grab lunch now that cluster seems to be happily recovering | 20:24 | |
| *** oubiwann_ has joined #openstack-infra | 20:24 | |
| mtreinish | clarkb, fungi, jeblair: shouldn't that be pointed at the infra pypi mirror?: http://logs.openstack.org/35/76635/2/check/gate-elastic-recheck-pep8/b5a7e8c/console.html#_2014-02-26_20_17_58_346 | 20:26 |
| jeblair | mtreinish: it doesn't track openstack/requirements so it doesn't use the mirror | 20:27 |
| mtreinish | ahh ok that makes sense | 20:27 |
| *** mspreitz has joined #openstack-infra | 20:27 | |
| mspreitz | Help, I made a procedural mistake. I submitting a patch for review without making a branch for it. How do I recover? | 20:28 |
| fungi | mspreitz: how do you want to recover? (what end state do you seek?) | 20:29 |
| mspreitz | First and foremost, no harm to anything else. Second, eventually a patch submitted correctly. The edits are few and easily reproduced. | 20:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Fix success-pattern usage for openstack-manuals https://review.openstack.org/76639 | 20:30 |
| fungi | mspreitz: so, as long as the state of your work is captured in the review, you can always pull it back to your local system from gerrit | 20:31 |
| jeblair | fungi, clarkb, sdague: i don't think we have the space to hold zuul gearman server debug logs. :( i think 3 days will take like 97G. | 20:31 |
| jeblair | so i think we're going to have to revert the logging config patch for now, then rework gear's logging so we can get useful information like that with less verbosity, then try again. | 20:31 |
| jeblair | either that or attach a cinder volume for logs | 20:31 |
| mspreitz | Let me start with the first part. Having submitted something on master, does that harm anything besides my submission? | 20:32 |
| fungi | mspreitz: nope, other than it probably has a blank topic in review.openstack.org, but that's not at all uncommon | 20:32 |
| fungi | mspreitz: in which case you can reset your local master branch to the same commit as the remote master with 'git reset --hard origin/master' and then you can retrieve your change into a local topic branch with 'git review -d NNNNN' (where NNNNN is the change number in gerrit) | 20:32 |
| openstackgerrit | A change was merged to openstack-infra/os-loganalyze: Remove tox locale overrides https://review.openstack.org/72218 | 20:33 |
| *** jamielennox is now known as jamielennox|away | 20:34 | |
| mspreitz | fungi: thanks. I'll try that when I get a chance. | 20:34 |
| fungi | mspreitz: git review -l will also list open reviews for the current project, in case you're looking for an easy way to spot the review number for yours | 20:35 |
| mspreitz | BTW, I noticed one other harm to my submission, which is a bug fix: it is not listed in the bug. | 20:36 |
| fungi | mspreitz: right, you'll want to make sure at the end of your commit message you include a separate line like Closes-Bug: #XXXXXX | 20:37 |
| *** Ryan_Lane1 has quit IRC | 20:38 | |
| fungi | mspreitz: however, if that wasn't present on the initial patchset for your change, the bug doesn't get automatically updated until your change gets merged, so you may want to set the bug to in-progress, assign it to yourself, and leave a comment with the url to your proposed fix | 20:38 |
| *** mestery_ is now known as mestery | 20:38 | |
| mspreitz | I had the Closes-Bug line in my original commit message. | 20:38 |
| *** lcostantino has quit IRC | 20:39 | |
| *** dhellmann_ is now known as dhellmann | 20:39 | |
| fungi | mspreitz: what is the url to your review and your bug? | 20:39 |
| clarkb | jeblair wow | 20:39 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add git05 to cacti and gerrit replication https://review.openstack.org/76649 | 20:42 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Add git05 to the git.openstack.org haproxy farm https://review.openstack.org/76650 | 20:42 |
| fungi | clarkb: jeblair: git05 is up (centos 6.5 8g performance pvhvm) and those ^ do what we need next | 20:42 |
| *** yolanda_ has quit IRC | 20:43 | |
| *** alexpilotti has quit IRC | 20:43 | |
| *** mattymo has quit IRC | 20:43 | |
| *** mattymo has joined #openstack-infra | 20:44 | |
| jeblair | fungi: neato. aprvd | 20:44 |
| jeblair | (the first) | 20:44 |
| jeblair | fungi: has create_cgit_repos run on 05? | 20:44 |
| *** mspreitz has quit IRC | 20:45 | |
| *** jhesketh has joined #openstack-infra | 20:45 | |
| fungi | jeblair: yep, looks like it | 20:45 |
| fungi | jeblair: thanks. i'll keep tabs on it and then we can add it with the second patch and start cycling the other 4 out and replacing them | 20:46 |
| jhesketh | Morning | 20:46 |
| clarkb | you can disable nodes with the admin socket to nicely remove nodes | 20:46 |
| fungi | right now /var/lib/git has empty repos and the interface is up at http://git05.openstack.org:8080/cgit | 20:47 |
| fungi | clarkb: in haproxy? | 20:47 |
| *** rhsu has joined #openstack-infra | 20:47 | |
| fungi | clarkb: yeah, will do when the time comes | 20:47 |
| clarkb | yup | 20:47 |
| *** 92AAAAI7R is now known as sabari | 20:48 | |
| *** lcheng_ has joined #openstack-infra | 20:50 | |
| openstackgerrit | Davanum Srinivas (dims) proposed a change to openstack-infra/config: Support filtering by review id(s) https://review.openstack.org/72446 | 20:53 |
| *** mfer has joined #openstack-infra | 20:54 | |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/config: log analyzer for openstack IRC logs https://review.openstack.org/72445 | 20:55 |
| *** bhuvan has joined #openstack-infra | 20:55 | |
| *** markmcclain has joined #openstack-infra | 20:56 | |
| *** markmcclain1 has joined #openstack-infra | 20:59 | |
| *** ryanpetrello has joined #openstack-infra | 20:59 | |
| *** mrodden1 has joined #openstack-infra | 20:59 | |
| dhellmann | jeblair: updated https://review.openstack.org/#/c/74408 | 21:00 |
| *** markmcclain has quit IRC | 21:00 | |
| *** mrodden has quit IRC | 21:01 | |
| *** oubiwann_ has quit IRC | 21:02 | |
| *** denis_makogon_ has joined #openstack-infra | 21:03 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add git05 to cacti and gerrit replication https://review.openstack.org/76649 | 21:04 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-infra/config: Add oslotest jobs for oslo.config https://review.openstack.org/76654 | 21:06 |
| openstackgerrit | Chad Lung proposed a change to openstack-infra/config: Add DevStack job for Barbican https://review.openstack.org/74530 | 21:06 |
| Ajaeger | fungi, clarkb: The docs-draft change works as expected - just tested with https://review.openstack.org/76647 ! Thanks again! | 21:07 |
| *** yassine has quit IRC | 21:09 | |
| *** Ajaeger has quit IRC | 21:12 | |
| *** CaptTofu_ has quit IRC | 21:14 | |
| *** shashank_ has joined #openstack-infra | 21:14 | |
| fungi | clarkb: jeblair: do you recall whether we need a gerrit restart to reread the replication config? i want to say so... | 21:14 |
| *** CaptTofu has joined #openstack-infra | 21:14 | |
| *** CaptTofu has quit IRC | 21:14 | |
| *** CaptTofu has joined #openstack-infra | 21:14 | |
| jeblair | fungi: hrm, i don't recall. you could try replicating one project and see... | 21:15 |
| fungi | yeah, that's where i was headed next. thanks | 21:15 |
| fungi | assuming it doesn't just magically start working once puppet updates the config here in a bit | 21:16 |
| *** bhuvan has quit IRC | 21:16 | |
| *** sandywalsh has quit IRC | 21:16 | |
| jeblair | i'm self-approving https://review.openstack.org/#/c/76657/ and will restart zuul so we don't run out of disk | 21:16 |
| *** dhellmann_ has joined #openstack-infra | 21:17 | |
| *** NikitaKonovalov is now known as NikitaKonovalov_ | 21:17 | |
| *** jd__` has joined #openstack-infra | 21:17 | |
| *** dims_ has joined #openstack-infra | 21:17 | |
| fungi | did we lose openstackgerrit? or is it just slow to the party on that change? | 21:18 |
| *** bookwar1 has joined #openstack-infra | 21:18 | |
| jeblair | fungi: i'm thinking it's lost. i'll kick it. | 21:20 |
| *** openstackgerrit has quit IRC | 21:20 | |
| *** dhellmann has quit IRC | 21:20 | |
| *** dims has quit IRC | 21:20 | |
| *** jd__ has quit IRC | 21:20 | |
| *** bookwar has quit IRC | 21:20 | |
| *** sbadia has quit IRC | 21:20 | |
| *** samalba has quit IRC | 21:20 | |
| *** shortstop has quit IRC | 21:20 | |
| *** sirushti has joined #openstack-infra | 21:20 | |
| *** jd__` is now known as jd__ | 21:20 | |
| *** Ryan_Lane has joined #openstack-infra | 21:20 | |
| jeblair | sender = self.ssl.write if self.ssl else self.socket.send | 21:20 |
| jeblair | AttributeError: 'NoneType' object has no attribute 'send' | 21:20 |
| *** bhuvan_ has joined #openstack-infra | 21:20 | |
| jeblair | yeah ^ that again | 21:20 |
| fungi | yep, ping timeout finally registered | 21:20 |
| *** sbadia has joined #openstack-infra | 21:21 | |
| jog0 | sdague: so I think I introduced instability into the voting grenade job yesterday | 21:21 |
| *** dhellmann_ is now known as dhellmann | 21:21 | |
| sdague | jog0: ok, how? | 21:21 |
| jog0 | sdague jeblair: also https://review.openstack.org/#/c/76658/ | 21:21 |
| *** openstackgerrit has joined #openstack-infra | 21:21 | |
| jog0 | sdague: so my grenade patch resulted in n-cpu not being killed when we expect it to | 21:21 |
| *** samalba has joined #openstack-infra | 21:22 | |
| jog0 | sdague:I think we will see this failurein regular grenade: http://logs.openstack.org/82/76582/2/check/check-grenade-dsvm-partial-ncpu/53ed49c | 21:22 |
| fungi | yeah, it looks like there's definitely the potential for other settings to get wiped without the append operator | 21:22 |
| sdague | jog0: so https://review.openstack.org/#/c/76658/ doesn't change anything | 21:23 |
| *** david_lyle_ has joined #openstack-infra | 21:23 | |
| *** coolsvap_ has joined #openstack-infra | 21:23 | |
| *** mestery_ has joined #openstack-infra | 21:23 | |
| sdague | because there is no other place that localrc for grenade should be written | 21:23 |
| jog0 | look up | 21:23 |
| *** gokrokve_ has joined #openstack-infra | 21:24 | |
| jog0 | line 53 | 21:24 |
| *** davidhadas_ has joined #openstack-infra | 21:24 | |
| *** e0ne has quit IRC | 21:24 | |
| jog0 | I am getting 500 errors from logstash | 21:24 |
| fungi | jog0: on every query? | 21:25 |
| jog0 | fungi: just most | 21:25 |
| *** Ryan_Lane has quit IRC | 21:25 | |
| *** bhuvan has joined #openstack-infra | 21:25 | |
| jeblair | jog0: line 53 is writing the devstack localrc | 21:25 |
| sdague | yeh, what jeblair said | 21:25 |
| jeblair | jog0: line 251 is writing the grenade localrc | 21:25 |
| jog0 | jeblair: ohh | 21:25 |
| fungi | $BASE/new/grenade/localrc specifically | 21:25 |
| *** jeckersb is now known as jeckersb_gone | 21:25 | |
| jog0 | thanks | 21:25 |
| jeblair | i think gerritbot is happy gain | 21:26 |
| *** rcleere_ has joined #openstack-infra | 21:26 | |
| sdague | but, if there is a different grenade race, we should revert the patch yuo think is an issue | 21:26 |
| fungi | seems like a safe enough change in case we decide to start writing to $BASE/new/grenade/localrc prior to that line eventually | 21:26 |
| sdague | then come back to it | 21:26 |
| jeblair | fungi: hrm. i don't think we should do that unless we really mean to. | 21:26 |
| * anteaya is back, after nearly getting hit by a car and getting off the phone with the police | 21:27 | |
| fungi | jeblair: fair enough | 21:27 |
| jeblair | fungi: because it means a change in grenade could surprise us | 21:27 |
| jeblair | anteaya: oh no! are you okay? | 21:27 |
| clarkb | ok food has been consumed | 21:27 |
| clarkb | anteaya: :/ | 21:27 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Write DO_NOT_UPGRADE_SERVICES to grenade's localrc not devstacks https://review.openstack.org/76658 | 21:27 |
| jog0 | sdague: that should be the fixI was looking for | 21:27 |
| sdague | oh... so you were *ALWAYS* doing it :) | 21:27 |
| sdague | jog0: so that's still wrong | 21:28 |
| jog0 | sdague: doh | 21:28 |
| *** rcleere has quit IRC | 21:28 | |
| *** rcleere_ is now known as rcleere | 21:28 | |
| *** Mithrandir has quit IRC | 21:28 | |
| jog0 | sdague: ohh I see | 21:28 |
| jog0 | derp | 21:28 |
| jog0 | $BASE/new/grenade/localrc | 21:28 |
| sdague | do the translation to the line in question you care about in -wrap | 21:28 |
| anteaya | jeblair: yes I am fine, missed be a whole 12 inches | 21:28 |
| *** sandywalsh has joined #openstack-infra | 21:29 | |
| sdague | then put that variable in the existing block | 21:29 |
| sdague | that's how we do everything else here | 21:29 |
| anteaya | otherwise I would be dead or no longer able to use the lower half of my body | 21:29 |
| *** dhellman_ has joined #openstack-infra | 21:29 | |
| jeblair | anteaya: i'm glad you're okay! | 21:29 |
| anteaya | I figure it was some high school good ole boy impressing his friends, passing a car with a pedestrian on teh other side | 21:29 |
| anteaya | jeblair: thanks, just in shock a bit | 21:29 |
| anteaya | clarkb: thanks | 21:30 |
| sdague | so actually what you are saying is you didn't actually change the partial upgrade case | 21:30 |
| * anteaya will get something to eat and read scrollback | 21:30 | |
| *** ildikov_ has quit IRC | 21:31 | |
| *** ildikov_ has joined #openstack-infra | 21:31 | |
| *** arborism has joined #openstack-infra | 21:31 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Write DO_NOT_UPGRADE_SERVICES to grenade's localrc not devstacks https://review.openstack.org/76658 | 21:32 |
| jog0 | sdague: third times a charm? | 21:32 |
| *** SumitNaiksatam has quit IRC | 21:32 | |
| *** coolsvap has joined #openstack-infra | 21:32 | |
| sdague | jog0: no, really, do the translation elsewhere, and lets only write to the grenade log once | 21:33 |
| jog0 | fungi: so logstash.o.o is unusable | 21:33 |
| jeblair | sdague: i think i need you to explain to me what was wrong with ps2 | 21:33 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool https://review.openstack.org/69510 | 21:33 |
| jog0 | sdague: not sure what you mean by do the translation elsewhere | 21:33 |
| jeblair | sdague, jog0: because i think it was correct and ps3 looks very wrong to me. | 21:33 |
| sdague | jeblair: it was not writing to grenade localrc | 21:33 |
| jeblair | sdague: it was writing to $BASE/new/grenade/localrc | 21:34 |
| jeblair | sdague: where should it have been writing? | 21:34 |
| fungi | clarkb: "If it helps, I received a 500 Internal Server Error from: api/search" | 21:34 |
| *** Mithrand1r has joined #openstack-infra | 21:34 | |
| sdague | jeblair: no, it was writing to >>localrc | 21:34 |
| *** gokrokve has quit IRC | 21:34 | |
| clarkb | fungi: ? | 21:34 |
| jeblair | sdague: ah you are right | 21:34 |
| *** dhellmann has quit IRC | 21:34 | |
| *** Mithrandir has joined #openstack-infra | 21:34 | |
| *** dhellmann has joined #openstack-infra | 21:34 | |
| *** mestery has quit IRC | 21:34 | |
| fungi | clarkb: that's the message kibana gives for any search at the moment | 21:34 |
| clarkb | oh scrollback | 21:34 |
| jog0 | https://review.openstack.org/#/c/76658/2/devstack-vm-gate.sh | 21:34 |
| *** _sirushti has joined #openstack-infra | 21:34 | |
| *** jhesketh has quit IRC | 21:34 | |
| *** dhellman_ has quit IRC | 21:34 | |
| jog0 | so the only issue is wrong file | 21:35 |
| *** coolsvap1 has quit IRC | 21:35 | |
| *** davidhadas has quit IRC | 21:35 | |
| *** jcooley_ has quit IRC | 21:35 | |
| *** david-lyle has quit IRC | 21:35 | |
| *** rossella_s has quit IRC | 21:35 | |
| *** apevec has quit IRC | 21:35 | |
| *** Mithrandir has quit IRC | 21:35 | |
| jog0 | but right block? | 21:35 |
| jeblair | sdague: but the correction to that should have just been to change "localrc" to "$BASE/new/grenade/localrc" on line 259 | 21:35 |
| jog0 | I thought sdague said move it somewhere else | 21:35 |
| jeblair | or... | 21:35 |
| *** rossella_s has joined #openstack-infra | 21:35 | |
| *** SumitNaiksatam has joined #openstack-infra | 21:35 | |
| sdague | jeblair: so I think we should be doing the translation in -wrap | 21:35 |
| jeblair | hehe, move the cd above the first heredoc and drop the long path from that one too. | 21:35 |
| sdague | and there should be | 21:35 |
| *** sirushti has quit IRC | 21:35 | |
| *** _sirushti is now known as sirushti | 21:35 | |
| *** zhiyan_ has quit IRC | 21:35 | |
| *** coolsvap_ has quit IRC | 21:35 | |
| sdague | DO_NOT_UPGRADE_SERVICES=$DO_NOT_UPGRADE_SERVICES | 21:35 |
| clarkb | fungi: logstash.o.o:/opt/kibana/kibana/KibanaConfig.rb | 21:35 |
| sdague | in the grenade block | 21:36 |
| fungi | jeblair: jog0: it sounded like the suggestion was to always put that bariable in the grenade localrc, but define the value of that variable in an earlier conditional prior to the heredoc | 21:36 |
| *** jcooley_ has joined #openstack-infra | 21:36 | |
| *** sandywalsh has quit IRC | 21:36 | |
| clarkb | fungi: notice it still has all of the nodes in that list. We may want to manually restrict it to the new ones for now | 21:36 |
| jog0 | sdague: ahh | 21:36 |
| jog0 | so even if its set to nothing we should include it? | 21:36 |
| *** dkliban has quit IRC | 21:36 | |
| *** zhiyan_ has joined #openstack-infra | 21:36 | |
| *** sandywalsh has joined #openstack-infra | 21:36 | |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Set Zuul gear server logs to debug" https://review.openstack.org/76657 | 21:36 |
| sdague | so all the grenade stuff happens at once, just to keep it easy to not run around realizing we modify it somewhere else | 21:36 |
| openstackgerrit | Elizabeth Krumbach Joseph proposed a change to openstack-infra/config: Add support for Fedora 20 to nodepool https://review.openstack.org/69510 | 21:36 |
| jeblair | sdague: wfm | 21:36 |
| sdague | jog0: yes, it should work | 21:36 |
| sdague | and if it doesn't, we have an issue | 21:36 |
| jog0 | thats the part I was missing | 21:37 |
| *** ildikov_ has quit IRC | 21:37 | |
| clarkb | fungi: did you want to stab at that? you will need to restart the kibana service afterwards too | 21:37 |
| fungi | clarkb: oh, it won't dynamically select just the cluster members which are up and responding? | 21:37 |
| *** ildikov_ has joined #openstack-infra | 21:37 | |
| clarkb | fungi: apparently not | 21:37 |
| sdague | the conditional write is logically fine, but I'm really concerned with the number of localrc files being written out that we'll miss something in review if grenade localrc get's written to more than once | 21:37 |
| *** oubiwann_ has joined #openstack-infra | 21:37 | |
| *** mestery_ is now known as mestery | 21:38 | |
| fungi | clarkb: should i slip in a change to remove all the old servers from the kibana config earlier than we remove them from everywhere else in that case? | 21:38 |
| sdague | honestly, bonus points for making the grenade_localrc write into a function, just to keep that all isolated | 21:38 |
| jog0 | sdague: thats why I put it right under the other block in https://review.openstack.org/#/c/76658/2/devstack-vm-gate.sh | 21:38 |
| clarkb | fungi: ya probably | 21:38 |
| clarkb | fungi: or, remove it by hand for now | 21:38 |
| fungi | clarkb: or is there a chance that it will try to reach a shard for which all replicas are on the old servers? | 21:38 |
| *** derekh has joined #openstack-infra | 21:38 | |
| clarkb | fungi: yes, but elasticsearch should do internal routing for us | 21:39 |
| sdague | jog0: well, the fact that jeblair missed the issue, means I want to be extra careful here, because it's easy to miss | 21:39 |
| openstackgerrit | afazekas proposed a change to openstack/requirements: Remove incompatible boto versions https://review.openstack.org/76663 | 21:39 |
| fungi | oh, got it | 21:39 |
| clarkb | so you can hit any node regardless of the shards it contains and it will figure it out for you | 21:39 |
| fungi | clarkb: in that case i'll stop puppet on logstash.o.o and make that config change manually on the server for the moment | 21:39 |
| clarkb | ok | 21:39 |
| *** rossella_s has quit IRC | 21:40 | |
| fungi | sudo vi /opt/kibana/kibana/KibanaConfig.rb | 21:40 |
| fungi | ha | 21:40 |
| fungi | waiting for the inevitable "fungi is not allowed to sudo on this host" joke | 21:40 |
| *** rossella_s has joined #openstack-infra | 21:40 | |
| jog0 | sdague: where should the translation be done? | 21:41 |
| fungi | clarkb: jog0: logstash queries seem to be working again | 21:42 |
| jog0 | fungi: yup | 21:42 |
| jog0 | thanks | 21:42 |
| sdague | jog0: I assume over in the -wrap script somewhere, that's where we do all the conditional transitions | 21:44 |
| *** rossella_ has joined #openstack-infra | 21:44 | |
| openstackgerrit | Bhuvan Arumugam proposed a change to openstack-infra/config: log analyzer for openstack IRC logs https://review.openstack.org/72445 | 21:46 |
| *** rossella_ has quit IRC | 21:47 | |
| *** rossella_s has quit IRC | 21:47 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Write DO_NOT_UPGRADE_SERVICES to grenade's localrc not devstacks https://review.openstack.org/76658 | 21:48 |
| jog0 | sdague: ^ | 21:48 |
| *** rossella_s has joined #openstack-infra | 21:48 | |
| sdague | jog0: tabs | 21:48 |
| *** prad has quit IRC | 21:48 | |
| jog0 | ohh pastemode | 21:49 |
| * sdague wonders how interested jeblair would be in bash8 running on d-g | 21:49 | |
| *** rossella_s has quit IRC | 21:50 | |
| *** sarob has quit IRC | 21:50 | |
| *** rossella_s has joined #openstack-infra | 21:50 | |
| *** prad has joined #openstack-infra | 21:51 | |
| *** sarob has joined #openstack-infra | 21:51 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Write DO_NOT_UPGRADE_SERVICES to grenade's localrc not devstacks https://review.openstack.org/76658 | 21:51 |
| jeblair | sdague: probably ok | 21:51 |
| jog0 | sdague: working back down the stack - filename:"console.html" AND message:"+ echo \'The following services are still running:Â Â nova-compute\'" AND NOT build_name:"check-grenade-dsvm-partial-ncpu" | 21:51 |
| jog0 | that is the issue | 21:52 |
| *** oubiwann_ has quit IRC | 21:52 | |
| harlowja | hi guys, for the following, i think a reverify would be correct right | 21:52 |
| harlowja | http://logs.openstack.org/20/54220/77/check/gate-taskflow-tox-py27-sa8-mysql/374be3e/console.html | 21:52 |
| jog0 | that I introduced | 21:52 |
| harlowja | "git.openstack.org: Temporary failure in name resolution" ? | 21:52 |
| sdague | jog0: sure, but that's not voting, right? | 21:52 |
| jog0 | sdague: not the AND NOT | 21:52 |
| jeblair | harlowja: yeah, i think there's already a bug for that | 21:52 |
| harlowja | kk | 21:52 |
| harlowja | thx jamespage | 21:53 |
| jog0 | I only see two hits so far | 21:53 |
| harlowja | oops jeblair | 21:53 |
| jog0 | so not sure | 21:53 |
| sdague | oh | 21:53 |
| sdague | wait, why is that not shutting down NCPU | 21:53 |
| jeblair | harlowja: 1270382 | 21:53 |
| jog0 | sdague: exactly | 21:54 |
| harlowja | yup thx jeblair found it | 21:54 |
| jog0 | sdague: for some reason that bug happens a lot more on the current partial-ncpu job | 21:54 |
| jog0 | which wasn't running partial-ncpu | 21:54 |
| sdague | heh | 21:55 |
| sdague | oh, well, remember es is still in world of hurt | 21:55 |
| *** e0ne has joined #openstack-infra | 21:55 | |
| jog0 | sdague: true | 21:55 |
| sdague | so yuo just might be missing data | 21:55 |
| *** sarob has quit IRC | 21:55 | |
| jog0 | but look at filename:"console.html" AND message:"+ echo \'The following services are still running:Â Â nova-compute\'" AND build_name:"check-grenade-dsvm-partial-ncpu" | 21:55 |
| jog0 | (without the NOT) | 21:55 |
| sdague | which makes me feel like we might want to revert the patch you think caused this | 21:56 |
| *** mfer has quit IRC | 21:56 | |
| *** jhesketh has joined #openstack-infra | 21:56 | |
| *** jhesketh has quit IRC | 21:56 | |
| jog0 | anyway I will continue to babysit this, and if it is bad we revert until understand what went wrong | 21:56 |
| *** jhesketh has joined #openstack-infra | 21:56 | |
| *** sarob has joined #openstack-infra | 21:56 | |
| *** e0ne has quit IRC | 21:56 | |
| *** jhesketh__ has joined #openstack-infra | 21:56 | |
| sdague | sure | 21:56 |
| sdague | I'm going to wait for results on your patch before voting on it | 21:56 |
| *** e0ne has joined #openstack-infra | 21:57 | |
| sdague | to make sure that's working in the base grenade case | 21:57 |
| jeblair | fungi, clarkb: i'm trying a new thing with this zuul restart | 21:57 |
| *** markwash has joined #openstack-infra | 21:57 | |
| jeblair | fungi, clarkb: i've shut down the mergers already so that no new builds will start | 21:57 |
| fungi | oh! neat idea | 21:57 |
| jeblair | fungi, clarkb: i'm thinking that will help stabilize things a little bit during the actual restart so that, say, the gate queue doesn't launch a bunch of jobs right before i kill it | 21:58 |
| *** e0ne has quit IRC | 21:58 | |
| *** e0ne has joined #openstack-infra | 21:58 | |
| *** CaptTofu has quit IRC | 21:59 | |
| clarkb | jeblair: will that cause those builds to fail instead because the merger can't merge anything? | 21:59 |
| jeblair | i'm also going to "update node set state=4 where state=3;" to speed things along a bit. | 21:59 |
| clarkb | if it works \o/ | 21:59 |
| jeblair | clarkb: no because gerrit's the thing that merges things. | 22:00 |
| Sukhdev | Folks we are running devstack (stack.sh) and are hitting following issue - wonder if somebody could shed some light | 22:00 |
| Sukhdev | 2014-02-26 13:46:05.451 | + git clone git://git.openstack.org/openstack/nova.git /opt/stack/nova | 22:00 |
| Sukhdev | 2014-02-26 13:46:26.358 | Cloning into '/opt/stack/nova'... | 22:00 |
| Sukhdev | 2014-02-26 13:47:26.365 | fatal: The remote end hung up unexpectedly | 22:00 |
| Sukhdev | 2014-02-26 13:47:26.369 | fatal: early EOF | 22:00 |
| sdague | so does anyone know if git.openstack.org is getting overloaded, because that's been happening a bit in logs | 22:01 |
| *** sdake_ has quit IRC | 22:01 | |
| fungi | sdague: it's pretty loaded. i'm adding a fifth git server to the farm and rebuilding the other 4 on performance+pvhvm servers | 22:01 |
| *** sarob has quit IRC | 22:01 | |
| sdague | ok | 22:01 |
| clarkb | sdague: are you behind a proxy? | 22:01 |
| sdague | not I | 22:02 |
| clarkb | gah Sukhdev ^ | 22:02 |
| sdague | I mean I've seen a bunch of people reporting the issue | 22:02 |
| jeblair | restarting zuul | 22:02 |
| clarkb | sdague: right there are two incarnations and one is be behind a proxy and hit the problem 100% of the time | 22:02 |
| *** julim has quit IRC | 22:02 | |
| clarkb | or not be behind a proxy and hit it occasionally | 22:02 |
| *** mgagne has quit IRC | 22:02 | |
| sdague | this was proposed - https://review.openstack.org/#/c/74910/ | 22:02 |
| sdague | to address it | 22:02 |
| *** mgagne has joined #openstack-infra | 22:03 | |
| *** melwitt1 has joined #openstack-infra | 22:03 | |
| *** e0ne has quit IRC | 22:03 | |
| sdague | I guess it just makes me wonder if we are doing ourselves a service by pointing most people to git.o.o vs. github here, if we are hitting load issues. | 22:03 |
| Sukhdev | clarkb: ? | 22:04 |
| jeblair | sdague: we are doing people a service because we are able to expand the capacity, which we're doing now. | 22:04 |
| clarkb | Sukhdev: are you behind a proxy? | 22:04 |
| *** mrmartin has joined #openstack-infra | 22:04 | |
| Sukhdev | clarkb: what do you mean by that? - firewal? | 22:04 |
| clarkb | Sukhdev: yes firewall or proxy | 22:05 |
| *** mrmartin has quit IRC | 22:05 | |
| *** melwitt has quit IRC | 22:05 | |
| Sukhdev | clarkb: yes, we are - but, we checked with the firewall folks, they claim that they do not see any drops of connection resets | 22:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove deprecated elasticsearch nodes from kibana https://review.openstack.org/76669 | 22:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Remove old elasticsearch cluster members https://review.openstack.org/76051 | 22:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Update logstash doc for an elasticsearch cluster https://review.openstack.org/76574 | 22:06 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-infra/config: Move primary elasticsearch discover node https://review.openstack.org/76575 | 22:06 |
| fungi | clarkb: is https://review.openstack.org/76051 there safe? | 22:06 |
| clarkb | Sukhdev: is port 9418 allowed through? | 22:06 |
| sdague | jeblair: ok | 22:06 |
| jeblair | zuul is restarted | 22:07 |
| fungi | clarkb: er, sorry, meant https://review.openstack.org/76669 | 22:08 |
| Sukhdev | clarkb: I believe so. I ran a script doing git clone the same repository for over 100 times - no failures. But, when I am running devstack (stack.sh) I see the ocassional failures | 22:08 |
| jeblair | Sukhdev: that's really interesting | 22:08 |
| *** CaptTofu has joined #openstack-infra | 22:08 | |
| Sukhdev | jeblair: yes it is I am so confused | 22:09 |
| *** Mithrand1r is now known as Mithrandir | 22:09 | |
| *** smarcet has quit IRC | 22:10 | |
| *** mgagne has quit IRC | 22:10 | |
| fungi | clarkb: jeblair: after a bit of playing around with gerrit replication, it does indeed seem that adding a new replication target is going to require a gerrit restart | 22:11 |
| *** rossella_s has quit IRC | 22:11 | |
| jeblair | fungi: i think we should go ahead and restart it whenever you are ready | 22:11 |
| jeblair | it's fast so i think we can just announce it in irc | 22:12 |
| clarkb | wfm | 22:12 |
| *** openstackstatus has joined #openstack-infra | 22:12 | |
| *** Sukhdev has quit IRC | 22:12 | |
| fungi | #status alert gerrit service on review.openstack.org will be down momentarily for a restart to add an additional git server | 22:13 |
| openstackstatus | NOTICE: gerrit service on review.openstack.org will be down momentarily for a restart to add an additional git server | 22:13 |
| *** ChanServ changes topic to "gerrit service on review.openstack.org will be down momentarily for a restart to add an additional git server" | 22:13 | |
| *** mgagne has joined #openstack-infra | 22:13 | |
| *** wchrisj has joined #openstack-infra | 22:14 | |
| *** DuncanT has joined #openstack-infra | 22:14 | |
| fungi | #status ok | 22:14 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 22:14 | |
| *** wchrisj is now known as Guest35609 | 22:14 | |
| *** wchrisj_ has quit IRC | 22:17 | |
| clarkb | fungi: es cluster looks green again. I think you can stop es on 04 whenever you are ready | 22:19 |
| fungi | we're closer--now it's at least trying to replicate, though we never puppeted the known_hosts for the gerrit2 user on review.o.o | 22:20 |
| jeblair | fungi: interesting problem since each of the git servers has a different key | 22:21 |
| fungi | jeblair: agreed. we'd presumably need to puppet each in advance of adding to replication configuration | 22:21 |
| *** sarob has joined #openstack-infra | 22:21 | |
| *** Ryan_Lane has joined #openstack-infra | 22:22 | |
| clarkb | we could cave and use exported resources now that we have puppetdb | 22:22 |
| jeblair | fungi: yeah. it's not ideal. you'll just 'ssh' by hand now? | 22:22 |
| fungi | however, i wonder whether gerrit caches the known_hosts file... | 22:22 |
| fungi | already did that part | 22:22 |
| clarkb | exported resources can copy the key from gitXX to puppetdb when puppet runs on gitXX then when puppet runs on gerrit grab that value from puppetdb and put it in place | 22:22 |
| fungi | but it still seems to be logging Feb 26 22:21:23 git05 sshd[18001]: Received disconnect from 198.101.231.251: 3: com.jcraft.jsch.JSchException: reject HostKey: git05.openstack.org | 22:22 |
| jeblair | clarkb: that actually sounds like the correct solution. | 22:22 |
| clarkb | exported resources tend to be a bit ugly because you ahve to double tap puppet | 22:22 |
| clarkb | and require puppetdb but we have one of those now | 22:23 |
| jeblair | clarkb: but are eventually consistent | 22:23 |
| clarkb | jeblair: correct | 22:23 |
| fungi | grr! https://review.openstack.org/Documentation/config-replication.html "Host keys for any destination SSH servers must appear in the user’s ~/.ssh/known_hosts file, and must be added in advance, before Gerrit starts. If a host key is not listed, Gerrit will be unable to connect to that destination, and replication to that URL will fail." | 22:24 |
| * fungi should memorize all of gerrit's documentation, clearly | 22:24 | |
| fungi | _must be added in advance, before Gerrit starts_ | 22:24 |
| jeblair | oh well. let's kick it again. | 22:25 |
| clarkb | that caching | 22:25 |
| fungi | yep | 22:25 |
| *** Guest35609 has quit IRC | 22:25 | |
| fungi | #status alert gerrit service on review.openstack.org will be down momentarily for a another brief restart--apologies for the disruption | 22:25 |
| openstackstatus | NOTICE: gerrit service on review.openstack.org will be down momentarily for a another brief restart--apologies for the disruption | 22:25 |
| *** ChanServ changes topic to "gerrit service on review.openstack.org will be down momentarily for a another brief restart--apologies for the disruption" | 22:25 | |
| *** pdmars has quit IRC | 22:26 | |
| *** mrodden has joined #openstack-infra | 22:26 | |
| *** Ryan_Lane has quit IRC | 22:26 | |
| *** sarob has quit IRC | 22:26 | |
| *** wchrisj_ has joined #openstack-infra | 22:26 | |
| fungi | #status ok | 22:27 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 22:27 | |
| *** mrodden1 has quit IRC | 22:27 | |
| dhellmann | is the status notice supposed to go out to all of the channels where the logging bot runs, or are those separate bots? | 22:27 |
| fungi | Feb 26 22:27:43 git05 sshd[20928]: Accepted publickey for cgit from 198.101.231.251 port 37644 ssh2 | 22:27 |
| fungi | dhellmann: only where the status bot (openstackstatus) runs | 22:28 |
| dhellmann | fungi: ok, thanks | 22:28 |
| fungi | dhellmann: we originally had it notifying in here and in #-dev since, you know, everyone was in there ;) | 22:28 |
| *** sarob has joined #openstack-infra | 22:28 | |
| dhellmann | fungi: now it's a ghost town over there | 22:28 |
| *** hashar has quit IRC | 22:28 | |
| fungi | too bad people flee channels where we notify important things ;) | 22:29 |
| jeblair | we'll put it everywhere when https://bugs.launchpad.net/openstack-ci/+bug/1190296 is complete | 22:29 |
| *** sarob_ has joined #openstack-infra | 22:29 | |
| *** weshay has quit IRC | 22:29 | |
| fungi | hey, lookit! http://git05.openstack.org:8080/cgit/openstack-infra/zuul/ | 22:29 |
| dhellmann | ok, I was just trying to figure out if something was broken or if I was just in a channel that didn't see the notices, so not a problem | 22:29 |
| *** markwash has quit IRC | 22:29 | |
| jeblair | fungi: yay! so we can trigger a full run and then approve that other change now | 22:30 |
| fungi | yep | 22:30 |
| clarkb | that other change? | 22:30 |
| clarkb | the one that adds 05? | 22:30 |
| jeblair | t | 22:30 |
| fungi | though rebuilding the other git servers will be touchy. i may want to backup and replace their ssh host keys | 22:30 |
| clarkb | fungi: good idea | 22:30 |
| jeblair | good point | 22:31 |
| fungi | so as to prevent needing multiple gerrit restarts | 22:31 |
| *** CaptTofu has quit IRC | 22:31 | |
| fungi | this one was unavoidable, but the others we can solve i think | 22:31 |
| fungi | gerrit seems to have done a full replicate on its own, maybe triggered by the restart | 22:32 |
| fungi | but i'll spot check anyway | 22:32 |
| *** sarob__ has joined #openstack-infra | 22:33 | |
| *** sarob_ has quit IRC | 22:33 | |
| openstackgerrit | Matt Riedemann proposed a change to openstack-infra/elastic-recheck: Add query for bug 1248757 https://review.openstack.org/76677 | 22:33 |
| *** thomasem has quit IRC | 22:34 | |
| *** denis_makogon_ has quit IRC | 22:34 | |
| sarob | Guys have moment to look at https://review.openstack.org/#/c/76419/ | 22:35 |
| fungi | looks like all the replicate processes in gerrit's queue have finished | 22:35 |
| *** changbl has quit IRC | 22:35 | |
| fungi | kicking off one manually now just to be sure | 22:35 |
| sarob | New stackforge project with no ops for tests | 22:35 |
| sarob | But running python tests anyway and failing | 22:35 |
| fungi | sarob: is it intended to eventually have tests? | 22:36 |
| *** pdmars has joined #openstack-infra | 22:36 | |
| sarob | In a few months | 22:37 |
| fungi | if so, adding a simple tox.ini that just runs /bin/true or something would be an easy way to make your test addition changes self-testing | 22:37 |
| sarob | No ops for right now | 22:37 |
| sarob | Or at least I thought | 22:37 |
| * fungi doesn't know what "ops" means in this context | 22:37 | |
| *** sarob__ has quit IRC | 22:37 | |
| *** pdmars_ has joined #openstack-infra | 22:38 | |
| clarkb | gate-noop | 22:38 |
| fungi | switching zuul to run gate-noop for that project in the place of actual jobs is an alternative, but does require a change to openstack-infra/config to enact that | 22:39 |
| sarob | Noop | 22:39 |
| sarob | Yup | 22:39 |
| sarob | Spellchk | 22:39 |
| *** thuc has quit IRC | 22:39 | |
| sarob | Fungi It's set that way now | 22:39 |
| *** thuc has joined #openstack-infra | 22:40 | |
| fungi | sarob: okay, please let us know the name of the project, or provide a url to a change where you're seeing the broken behavior | 22:40 |
| fungi | oh, you did | 22:41 |
| *** mbacchi_ has quit IRC | 22:41 | |
| fungi | looking at the layout.yaml to confirm | 22:41 |
| *** pdmars has quit IRC | 22:41 | |
| *** sarob_ has joined #openstack-infra | 22:41 | |
| *** mrda_away is now known as mrda | 22:41 | |
| *** Sukhdev has joined #openstack-infra | 22:41 | |
| sarob | layout has it as milk | 22:42 |
| fungi | sarob: as i suspected, stackforge/mil is set in layout.yaml for a variety of jobs, like the ones you see failing on your change... http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml#n3553 | 22:42 |
| sarob | With noon | 22:42 |
| fungi | er, stackforge/milk | 22:42 |
| sarob | Arrg | 22:42 |
| sarob | Right stackforge/milk | 22:42 |
| Sukhdev | jeblair clarkb: any idea? | 22:43 |
| fungi | sarob: so you'll want to submit a change to openstack-infra/config:modules/openstack_project/files/zuul/layout.yaml around line 3553 to change that so that it runs gate-noop in the check and gate pipelines, and probably get rid of the post pipeline entirely for now | 22:44 |
| sarob | I just did a pull and I show stackforge/milk | 22:44 |
| *** thuc has quit IRC | 22:44 | |
| sarob | Set for check and gate noop | 22:44 |
| *** mriedem has quit IRC | 22:44 | |
| sarob | Projects and python-jobs has no entry | 22:45 |
| *** dangers is now known as dangers_away | 22:46 | |
| fungi | sarob: please show me what you're talking about on http://git.openstack.org/cgit/openstack-infra/config/tree/ (i think maybe you're not looking at the layout.yaml, or you have a locally-modified branch you've been pulling into for some reason) | 22:46 |
| sarob | I checked github and you are correct | 22:47 |
| sarob | I thought I was losing my mind | 22:47 |
| fungi | sarob: you probably have a dirty master branch you modified at some point in the past and then pulled new remote changes into | 22:48 |
| sarob | Yup | 22:48 |
| fungi | sarob: you may want to reset it to origin/master | 22:48 |
| sarob | Dirty is bad | 22:48 |
| sarob | Will check | 22:48 |
| jeblair | Sukhdev: i don't. if you are willing to do some serious debugging, you could to a tcpdump on your host and if that fails, compare it with a packet capture or session log from the firewall. if you get that far and suspect that the problem is on the git.o.o side, we can schedule a time to do packet captures there as well. | 22:50 |
| *** mayu_ has joined #openstack-infra | 22:51 | |
| jeblair | Sukhdev: but honestly, we know that git.o.o is slightly overloaded at this point (we're almost done adding a new server), so it could just be bad luck. | 22:51 |
| *** dkranz has quit IRC | 22:51 | |
| mayu_ | jaypipes: hi, there | 22:51 |
| jeblair | clarkb, fungi: logstash queue seems on a decidedly downward trend. | 22:52 |
| fungi | jeblair: agreed, it seems to have made up its mind about that | 22:52 |
| jeblair | fungi: is replication complete? | 22:53 |
| *** hogepodge has quit IRC | 22:53 | |
| clarkb | fungi: jeblair yup | 22:53 |
| fungi | clarkb: jeblair: okay, the re-run of a full replicate has completed all pending tasks, but i'm testing git/http/https direct cloning now just to make sure nothing's screwy | 22:53 |
| mayu_ | clarkb: help, http://paste.openstack.org/show/69992/ | 22:53 |
| fungi | then we can merge 76650 as soon as that's done | 22:53 |
| clarkb | fungi: roger | 22:54 |
| jeblair | fungi: cool. i +2d, ready for your go on merging. | 22:54 |
| mayu_ | clarb: Following jaypipes's blog to construct third-party ci, there is an error on the link http://paste.openstack.org/show/69992/ | 22:55 |
| *** lnxnut has quit IRC | 22:55 | |
| fungi | testing by cloning nova is a bit slow, but if anything's going to break that's where we'd see it first i suspect, so good canary | 22:55 |
| clarkb | mayu_: it looks like you cut off the traceback, was there more to the traceback? | 22:55 |
| mayu_ | clarkb: no, remote hung up | 22:56 |
| jeblair | mayu_: we think the git servers are overloaded, we're adding a new one now. however, if you are behind a firewall or proxy, that could be the problem as well. | 22:56 |
| jeblair | there has been a significant increase it git.o.o traffic at around 19:15 today. | 22:58 |
| mayu_ | the other project code git clone sucessful, It is not firewall problem | 22:58 |
| jeblair | mayu_: just try again | 22:58 |
| *** e0ne has joined #openstack-infra | 22:58 | |
| fungi | jeblair: clarkb: interestingly, i got this from git05, even though i'm probably the only client hitting it... http://paste.openstack.org/show/69998/ | 22:59 |
| clarkb | huh | 23:00 |
| fungi | maybe we have apache tuning we need to do, or does haproxy time out active sockets under some circumstances? | 23:00 |
| clarkb | fungi: does the apache error log say anything interesting? | 23:00 |
| mayu_ | I did, exec install_slave.sh, but it is not trusted. | 23:00 |
| fungi | i'm digging now | 23:00 |
| clarkb | also the haproxy log should tell you stuff | 23:00 |
| jeblair | that's not going through haproxy | 23:00 |
| fungi | oh, wait, i'm also not going through haproxy in this case | 23:00 |
| *** Ryan_Lane has joined #openstack-infra | 23:00 | |
| fungi | right | 23:00 |
| jeblair | [Wed Feb 26 22:58:27 2014] [warn] [client 2001:470:8:d2f:96de:80ff:feec:f9e7] Timeout waiting for output from CGI script /usr/libexec/git-core/git-http-backend | 23:01 |
| jeblair | [Wed Feb 26 22:58:27 2014] [error] [client 2001:470:8:d2f:96de:80ff:feec:f9e7] (70007)The timeout specified has expired: ap_content_length_filter: apr_bucket_read() failed | 23:01 |
| fungi | yep | 23:01 |
| mayu_ | clarkb: exec install_slave.sh again, this is the output http://paste.openstack.org/show/69999/. it seems not trusted. | 23:02 |
| fungi | http://httpd.apache.org/docs/2.0/mod/core.html#timeout | 23:02 |
| *** e0ne has quit IRC | 23:03 | |
| *** ociuhandu has joined #openstack-infra | 23:03 | |
| jeblair | fungi: we should pack refs | 23:03 |
| jeblair | fungi: we should run the git repo maintenance crons manually on git05 | 23:03 |
| fungi | jeblair: running them now | 23:03 |
| *** dims_ has quit IRC | 23:04 | |
| *** harlowja is now known as harlowja_away | 23:05 | |
| mayu_ | jeblair: exec install_slave.sh again, this is the output http://paste.openstack.org/show/69999/. it seems not trusted. | 23:05 |
| clarkb | mayu_: it looks like it worked | 23:05 |
| *** bknudson has quit IRC | 23:05 | |
| *** yamahata has quit IRC | 23:06 | |
| *** ianw has quit IRC | 23:07 | |
| *** ianw has joined #openstack-infra | 23:07 | |
| fungi | clarkb: that paste looks like it failed out on a git clone retry which didn't work | 23:08 |
| fungi | remember recently we tweaked that script to retry to clone once if the first attempt fails | 23:08 |
| clarkb | right the first half of the paste is the old failure | 23:08 |
| clarkb | the second half was successful | 23:08 |
| fungi | oic | 23:09 |
| openstackgerrit | Sean Roberts proposed a change to openstack-infra/config: modified stackforge/milk project in zuul/layout.yaml https://review.openstack.org/76684 | 23:09 |
| mayu_ | clarkb: It seems that the downloading work not continue as the last exception. | 23:09 |
| fungi | yes, re-running does seem to have worked | 23:09 |
| *** jamielennox|away is now known as jamielennox | 23:10 | |
| fungi | oh, actually no the install_slave.sh worked, the prep_node.sh did not | 23:10 |
| fungi | if i'm interpreting that correctly | 23:10 |
| *** jgrimm has quit IRC | 23:11 | |
| jeblair | i just spot checked a devstack-precise node from each region, and they are not cloning repos. so i don't think the additional traffic is because we broke caching there. | 23:12 |
| *** doddstack has quit IRC | 23:12 | |
| fungi | jeblair: i suspect if we do some log analysis on the haproxy server we'll find a recent uptick in utilization from some third-party ci | 23:13 |
| mayu_ | fungi: I will launch dsvm-tempest-full, expecting it runs well | 23:14 |
| clarkb | fungi: jeblair 192.237.223.224 is persisntent | 23:14 |
| clarkb | which is haproxy | 23:15 |
| clarkb | so I don't think we are getting out of band requests | 23:15 |
| Sukhdev | jeblair: I did the tcpdump yesterday as well - Had the firewall guys look at the traffic as well - Did not see any issues | 23:15 |
| clarkb | grep -v makes it seem like most of the traffic is from the load balancer | 23:15 |
| jeblair | Sukhdev: i'd recommend seeing if the problem persists until after we add the new git server to the pool, and if so, let's dig into it more then. | 23:17 |
| jeblair | s/until// | 23:17 |
| Sukhdev | jeblair: when do you plan on adding additional server? | 23:17 |
| jeblair | Sukhdev: moments from now | 23:18 |
| *** pmathews has quit IRC | 23:18 | |
| *** pmathews1 has joined #openstack-infra | 23:18 | |
| openstackgerrit | A change was merged to openstack/requirements: Upgrade six to 1.5.2 https://review.openstack.org/68424 | 23:18 |
| Sukhdev | jeblair: Ah then it makes sense to wait it out | 23:18 |
| fungi | the repack is almost done with nova, looks like. that one takes quite a while | 23:18 |
| jeblair | grep 18:..:.. haproxy.log|wc -l == 92118 | 23:19 |
| jeblair | grep 20:..:.. haproxy.log|wc -l == 151808 | 23:19 |
| jeblair | there definitely seems to be a jump in number of requests | 23:19 |
| fungi | http://paste.openstack.org/show/70006/ | 23:20 |
| fungi | bluebird.ibm.com | 23:20 |
| *** dims_ has joined #openstack-infra | 23:20 | |
| lifeless | word | 23:20 |
| jeblair | fungi wins | 23:20 |
| *** VijayT has joined #openstack-infra | 23:21 | |
| * fungi has had far too many years analyzing service logs to spot denial of service attacks | 23:21 | |
| fungi | i suppose we could correlate with gerrit logs to see whether there are any service accounts authenticating from ip addresses with similar reverse dns entries | 23:22 |
| jeblair | fungi: i think they only account for an increase of 600 hits from the 18 hour to 20 | 23:23 |
| fungi | ahh | 23:23 |
| fungi | we can narrow that then | 23:23 |
| clarkb | vmware | 23:23 |
| *** markwash has joined #openstack-infra | 23:23 | |
| jeblair | fungi: btw, i think if this approach produces a negative result, then it's likely our own ddos that we call "the gate". | 23:24 |
| jeblair | since we cycle through ips pretty quickly | 23:24 |
| SpamapS | Updating cache of https://git.openstack.org/openstack/keystone.git in /root/.cache/image-create/repository-sources/6bd7a | 23:24 |
| SpamapS | 6223fe50b4a267bc924641d331df94c833e with ref master | 23:24 |
| SpamapS | error: RPC failed; result=7, HTTP code = 0 | 23:24 |
| SpamapS | Isolated incident or onging problem? | 23:24 |
| clarkb | jeblair: yes the log file is 35XMB of that 40MB appears to be not gate | 23:24 |
| jeblair | SpamapS: seems decreasingly isolated | 23:24 |
| *** dstanek is now known as dstanek_afk | 23:25 | |
| openstackgerrit | Jenkins proposed a change to openstack-dev/hacking: Updated from global requirements https://review.openstack.org/76687 | 23:25 |
| *** shashank_ has quit IRC | 23:25 | |
| *** eharney has quit IRC | 23:26 | |
| *** VijayT has left #openstack-infra | 23:26 | |
| sdague | krtaylor: is there any way to deal with mimetypes on softlayer's swift to make it so you can view those files directly in browser, instead of them pushing as attachment? | 23:26 |
| SpamapS | jeblair: would it be helpful if I report a bug on launchpad? Or just watch for another one? | 23:26 |
| *** VijayT has joined #openstack-infra | 23:26 | |
| jeblair | SpamapS: we're almost done adding another git server to the pool, that should help. | 23:27 |
| SpamapS | ahh backscroll.. moar servers ... ok :) | 23:27 |
| jeblair | clarkb, fungi: i think 1915 is about when i started doing zuul restarts. that makes me think this is mostly us. | 23:28 |
| fungi | and then hopefully identifying the recent uptick in utilization so we can thwappp it solidly | 23:28 |
| fungi | thwappp ourselves then | 23:28 |
| clarkb | jeblair: just queued load that is hammering the gate? | 23:28 |
| fungi | repacks just finished. retesting real quick | 23:28 |
| *** vkozhukalov has quit IRC | 23:28 | |
| anteaya | what will it take for git05 to show up in the git farm cacti grouping? http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=2 | 23:29 |
| jeblair | clarkb: i think so. i'm glad we found this now though since we're getting better at sustaining this kind of load in the zuul-nodepool-jenkins. it's looking more like git.o.o is not overpowered anymore. | 23:29 |
| jeblair | anteaya: some gui clicking. i'll do it | 23:30 |
| anteaya | jeblair: k | 23:30 |
| Sukhdev | jeblair: can you ping me as soon additional server come on line - I would like to try it then and report back the findings | 23:30 |
| anteaya | Sukhdev: I will let you know | 23:30 |
| *** shashank_ has joined #openstack-infra | 23:30 | |
| Sukhdev | anteaya: cool - thanks | 23:30 |
| mayu_ | anteaya: hi | 23:30 |
| anteaya | mayu_: hello | 23:30 |
| *** bhuvan has quit IRC | 23:31 | |
| Sukhdev | anteaya: BTW, in the morning discussion, you were correct. | 23:31 |
| anteaya | okay | 23:31 |
| mayu_ | anteaya: things goes not as expected | 23:31 |
| anteaya | I haven't gotten back to that yet | 23:31 |
| jeblair | anteaya: done | 23:32 |
| *** bhuvan_ has quit IRC | 23:32 | |
| anteaya | mayu_: waht are you expecting | 23:32 |
| * anteaya refreshes cacti git farm page | 23:32 | |
| *** thuc has joined #openstack-infra | 23:32 | |
| anteaya | jeblair: thanks | 23:32 |
| jeblair | np | 23:32 |
| *** thuc has quit IRC | 23:32 | |
| reed | in case you like to know how things end, I went and bought the Nexus 5 from a tmobile store (google store had a 2-3 weeks wait) | 23:33 |
| *** bhuvan__ has joined #openstack-infra | 23:33 | |
| *** bhuvan_ has joined #openstack-infra | 23:33 | |
| clarkb | reed: it was the correct choice | 23:33 |
| *** thuc has joined #openstack-infra | 23:33 | |
| clarkb | moto Gs lack of LTE is annoying | 23:33 |
| anteaya | reed: are you happy? | 23:33 |
| reed | clarkb, I follow your orders :) | 23:33 |
| clarkb | perfect cheap smartphone though | 23:33 |
| reed | anteaya, happy is a big word... no, I'm not: I am $396 poorer and some jerk has my perfectly find Nexus 4 | 23:34 |
| *** lcheng__ has joined #openstack-infra | 23:34 | |
| reed | s/find/fine | 23:34 |
| anteaya | reed: well at least you are honest | 23:34 |
| anteaya | taht scores big points with me | 23:34 |
| *** lcheng__ has quit IRC | 23:35 | |
| fungi | jeblair: clarkb: successfully cloned nova via git and http directly, so approving 76650 now | 23:35 |
| *** lcheng_ has quit IRC | 23:36 | |
| anteaya | git03 is still being worked very hard | 23:36 |
| anteaya | reed: and I understand how you feel | 23:36 |
| *** ryanpetrello has quit IRC | 23:36 | |
| mayu_ | Sukhdev: how to judge third-party work well with openstack ci ? | 23:37 |
| clarkb | jeblair: when you have a second can you hop on logstash.o.o and see if the log-gearman-client behavior is geard related? | 23:37 |
| *** bhuvan has joined #openstack-infra | 23:38 | |
| *** bhuvan___ has joined #openstack-infra | 23:38 | |
| clarkb | it still has a cpu pegged | 23:38 |
| mayu_ | anteaya: how to judge third-party work well with openstack ci ? | 23:39 |
| jeblair | clarkb: ack | 23:39 |
| *** eharney has joined #openstack-infra | 23:39 | |
| *** rlandy has quit IRC | 23:39 | |
| mayu_ | anteaya: how to judge third-party ci work well with openstack ci ? | 23:39 |
| jeblair | clarkb: it could be. it's sending a lot of data. | 23:40 |
| *** bhuvan__ has quit IRC | 23:40 | |
| *** bhuvan_ has quit IRC | 23:40 | |
| mayu_ | jaypipes: how to judge third-party ci work well with openstack ci ? | 23:40 |
| anteaya | mayu_: okay okay slow down | 23:40 |
| anteaya | we all can see the questions, sometimes it takes a minute to get to them | 23:41 |
| mayu_ | sorry | 23:41 |
| openstackgerrit | A change was merged to openstack-infra/config: Add git05 to the git.openstack.org haproxy farm https://review.openstack.org/76650 | 23:41 |
| *** lcheng__ has joined #openstack-infra | 23:41 | |
| anteaya | it is okay, you are learning | 23:41 |
| anteaya | mayu_: okay so right now your system can listen to changes to openstack/sandbox correct? | 23:42 |
| mayu_ | yes | 23:42 |
| anteaya | okay so post a change to openstack/sandbox repo | 23:42 |
| anteaya | and post the url for that change | 23:42 |
| anteaya | and then we will watch what happens | 23:42 |
| mayu_ | https://review.openstack.org/#/c/75953/ | 23:43 |
| *** bhuvan_ has joined #openstack-infra | 23:43 | |
| *** bhuvan__ has joined #openstack-infra | 23:43 | |
| anteaya | can you post a new patchset to that change, please? | 23:43 |
| anteaya | let's see what your system is doing right now | 23:43 |
| mayu_ | ok | 23:44 |
| *** bhuvan___ has quit IRC | 23:44 | |
| jeblair | clarkb: the stats run does a full iteration over all the jobs, so performance should improve as the queue goes down. i'm kind of inclined to not apply any emergency fixes at the moment since i'm about to go afk for 2 days.... | 23:44 |
| clarkb | jeblair: wfm | 23:45 |
| jeblair | clarkb: but i think we could switch geard to just using some internal counters so it doesn't have to do that | 23:45 |
| *** bhuvan has quit IRC | 23:45 | |
| clarkb | it appeared a bit odd so I asked | 23:45 |
| anteaya | Sukhdev: you can try running .stack.sh again | 23:45 |
| clarkb | but it iterating over the list each time makes sense about why it is slow | 23:45 |
| jeblair | clarkb: and if you decide you'd rather shut it off, removing the env vars from the defaults file and restarting should do the trick | 23:45 |
| clarkb | jeblair: ok | 23:45 |
| clarkb | thank you for looking | 23:45 |
| jeblair | "tcpdump port 8125 -A" on logstash is pretty cool, btw. :) | 23:46 |
| * clarkb tries that | 23:46 | |
| clarkb | oh neat you can watch the waiting metric go crazy | 23:47 |
| dtroyer | clarkb: I just left a response to your question in https://review.openstack.org/#/c/74910/, if you're happy it can go in. | 23:47 |
| *** bhuvan has joined #openstack-infra | 23:48 | |
| *** bhuvan___ has joined #openstack-infra | 23:48 | |
| clarkb | dtroyer: and responded I think I can live with it as is aftr you point that out | 23:48 |
| *** bhuvan__ has quit IRC | 23:49 | |
| *** bhuvan_ has quit IRC | 23:49 | |
| dtroyer | cool, thanks | 23:50 |
| jeblair | clarkb: ? | 23:51 |
| jeblair | clarkb: devstack does fetches in the gate? | 23:51 |
| jeblair | clarkb: it shouldn't do _any_ git operations | 23:51 |
| *** cadenzajon has quit IRC | 23:51 | |
| clarkb | jeblair: I think at least one of those fetches was outside of the erroronclone condition | 23:51 |
| clarkb | I didn't look at it too long /em looks again | 23:51 |
| dtroyer | one is outside, but inside RECLONE=True | 23:52 |
| clarkb | ah | 23:52 |
| clarkb | ok then no fetches in the gate | 23:52 |
| *** dolphm is now known as dolphm_503 | 23:52 | |
| fungi | jeblair: clarkb: i think i agree that we're the ones ddosing ourselves... over 50% of the entries in the haproxy.log are from rackspace's ipv6 assignment | 23:53 |
| fungi | for the past hour | 23:53 |
| *** bhuvan_ has joined #openstack-infra | 23:53 | |
| *** bhuvan__ has joined #openstack-infra | 23:53 | |
| *** nati_ueno has joined #openstack-infra | 23:53 | |
| *** mayu__ has joined #openstack-infra | 23:53 | |
| *** bhuvan has quit IRC | 23:53 | |
| jeblair | cool, so this is the new normal. :) | 23:53 |
| mayu__ | anteaya: https://review.openstack.org/#/c/76711/ | 23:54 |
| *** bhuvan___ has quit IRC | 23:54 | |
| *** mrodden has quit IRC | 23:55 | |
| krtaylor | sdague, I am looking into it, I agree it would be much nicer, it may be possible if I dig deeper | 23:55 |
| mayu__ | anteaya:it is the new patch that submitted right now. | 23:55 |
| anteaya | mayu__: okay so lets look at the comments your system leaves on that patch | 23:55 |
| anteaya | first of all it comments, that is a big thing to notice, well done | 23:55 |
| *** lcheng__ has quit IRC | 23:56 | |
| fungi | clarkb: jeblair: another 25% is from hp's class-a ipv4 assignment (and i'm willing to bet most of the remaining 25% are from other smaller hp allocations) | 23:56 |
| anteaya | that is the hard part | 23:56 |
| anteaya | mayu__: now you have to fine tune it so that it comments only when you want it to | 23:56 |
| anteaya | and the comments contain useful information | 23:56 |
| mayu__ | ok | 23:56 |
| anteaya | mayu__: so it leaves this comment Starting check jobs. http://10.250.201.20/zuul/status | 23:57 |
| clarkb | fungi: ya | 23:57 |
| anteaya | you need to ensure it does not leave that comment, mayu__ | 23:57 |
| anteaya | mayu__: yours is one of about 40 ci systems | 23:57 |
| *** e0ne has joined #openstack-infra | 23:57 | |
| fungi | jeblair: clarkb: though it's worth pointing out that i'm only analyzing the number of requests, so it's entirely possible that these are tiny fetches and someone else is doing whopping large reclones over and over which account for far fewer actual calls but much more system load | 23:57 |
| *** nati_uen_ has quit IRC | 23:57 | |
| *** nati_ueno has quit IRC | 23:57 | |
| clarkb | fungi: good point | 23:57 |
| anteaya | mayu__: and if all 40 ci systems left that message when they are starting jobs, that is just a log of noise | 23:58 |
| *** bhuvan has joined #openstack-infra | 23:58 | |
| *** bhuvan___ has joined #openstack-infra | 23:58 | |
| clarkb | fungi: there is an haproxy top thing we might want to look into using | 23:58 |
| *** nati_ueno has joined #openstack-infra | 23:58 | |
| clarkb | fungi: but I am pretty sure it isn't packaged anywhere | 23:58 |
| anteaya | mayu__: the only system that can leave the starting jobs message is our system, jenkins | 23:58 |
| clarkb | which is why I ginored it in the past | 23:58 |
| anteaya | mayu__: does that make sense? | 23:58 |
| mayu__ | not clear | 23:59 |
| fungi | clarkb: hatop? | 23:59 |
| *** bhuvan__ has quit IRC | 23:59 | |
| *** bhuvan_ has quit IRC | 23:59 | |
| fungi | clarkb: http://feurix.org/projects/hatop/ | 23:59 |
| *** sarob has quit IRC | 23:59 | |
| clarkb | fungi: yeah | 23:59 |
| anteaya | mayu__: your system leaves a comment on the patch to say it is starting jobs | 23:59 |
| anteaya | mayu__: you need to ensure it does not leave that message | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
