| jeblair | fungi: excellent, i'll leave 58106 for you to aprv when you feel like this conversation has successfully concluded :) | 00:00 |
|---|---|---|
| *** dstanek has quit IRC | 00:00 | |
| fungi | jeblair: my pleasure | 00:00 |
| fungi | zaro: your pending python-jenkins contribution shouldn't hold up approval of https://review.openstack.org/56412 right? | 00:01 |
| mordred | lifeless: that's right | 00:01 |
| *** loq_mac has quit IRC | 00:01 | |
| lifeless | mordred: that seems undesirable | 00:01 |
| mordred | lifeless: but I hope to soon get some build advice to the pip guys to let them know if a release will break us | 00:01 |
| lifeless | mordred: perhaps telling it to install from the mirror would avoid this? | 00:01 |
| mordred | lifeless: the version of pip you should always use is the latest version - all old versions of pip are broken | 00:01 |
| mordred | lifeless: perhaps | 00:02 |
| lifeless | though mirror as version lock is an antipattern | 00:02 |
| mordred | yup | 00:02 |
| mordred | we could tell git pip to install pip==1.4.1 if we wanted | 00:02 |
| mordred | and point that at the mirror | 00:02 |
| lifeless | anyhow, thanks for explaining. This isn't at all obvious, and I think you should capture that somewhere in the patch. | 00:02 |
| lifeless | for our future sanity. | 00:02 |
| mordred | well, the real guts are in devstack, not here | 00:02 |
| mordred | this is just enabling the flag | 00:03 |
| mordred | I agree though - I believe that more sanity should be captured | 00:03 |
| *** rongze has quit IRC | 00:04 | |
| fungi | completely unrelated, btw, but i deleted several vacated cinder volumes from static.o.o a little while ago, replaced those with larger ones, and so far nothing seems to have broken horribly so the plan there is going well (continuing to shuffle extents to the larger volumes but i expect it to be going on for days) | 00:05 |
| jeblair | fungi: thank you | 00:05 |
| fungi | linux lvm2 is a marvellous work of engineering | 00:06 |
| *** dcramer__ has quit IRC | 00:06 | |
| lifeless | we layer lvm on top of cinder vols? | 00:06 |
| fungi | yup | 00:06 |
| lifeless | do we do redundancy that way too? | 00:07 |
| fungi | nope, just jbod | 00:07 |
| lifeless | so - cinder then mdadm then lvm ? | 00:07 |
| *** datsun180b has quit IRC | 00:07 | |
| lifeless | or we trust that cinder will be available ? | 00:07 |
| fungi | nope, no md-raid, just lvm2 on top of cinder volumes | 00:07 |
| lifeless | ok, and just linear in lvm2? | 00:08 |
| fungi | yup | 00:08 |
| lifeless | thanks | 00:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Add gate-solum-devstack job https://review.openstack.org/57098 | 00:08 |
| openstackgerrit | A change was merged to openstack-infra/config: Add tooz https://review.openstack.org/56927 | 00:08 |
| *** flaper87 is now known as flaper87|afk | 00:09 | |
| fungi | lifeless: rackspace limits the size of a cinder volume to 1tb, their underlying tech (xen) limits block devices per vm to 16 (two of which are already preclaimed by their model), and they limit (presumably artifically) aggregate cinder storage to 10tb per account | 00:09 |
| jeblair | lifeless: it seems likely that raid is happening behind cinder; but we aren't sure, so attempting to raid on top of cinder (without knowing what's under it) could mean we detune the storage instead of tuning it. | 00:09 |
| *** reaper has quit IRC | 00:10 | |
| *** ^d has joined #openstack-infra | 00:10 | |
| jeblair | fungi: oh, are we going to hit the 10tb limit? i'm assuming we could get that raised. | 00:10 |
| fungi | right, at $oldjob-2 we always had to explain to our cloud customers that doing raid-1 on top of our san raid-1 wasn't especially sane | 00:10 |
| jeblair | fungi: exactly :) | 00:11 |
| fungi | jeblair: i only need ~9tb to reach the discussed goal, but yeah their faq makes it sound like a simple quota | 00:11 |
| *** wenlock has quit IRC | 00:11 | |
| jgriffith | fungi: not to jump in but mdraid between two cinder volumes is something I believe RAX recommends for certain cases | 00:12 |
| fungi | jgriffith: oh, cool. however in our case i think we just need $lots_of_hopefully_reliable_storage until we develop a better object storage solution to swap it into | 00:13 |
| *** rcleere has quit IRC | 00:13 | |
| jgriffith | fungi: ahhh got ya | 00:14 |
| jgriffith | fungi: stick with block :) | 00:14 |
| *** boris-42 has quit IRC | 00:14 | |
| fungi | so aggregating with lvm2 is an easy solution for now | 00:14 |
| jgriffith | fungi: totally makes sense | 00:14 |
| fungi | jgriffith: i don't think we'll ever be getting away from block storage enturely, just some bulk storage for applications where it makes sense to use object stores | 00:15 |
| jgriffith | FWIW I have a bp somewhere to create that as an optional volume type in Cinder | 00:15 |
| fungi | oh, that would be nifty | 00:15 |
| jgriffith | fungi: Yeah... and if I could get descent perf out of clustered LVM or LINBIT I'd like to make progress on that as well | 00:15 |
| jgriffith | but anywho | 00:15 |
| *** melwitt has quit IRC | 00:16 | |
| *** nati_ueno has joined #openstack-infra | 00:18 | |
| *** nati_uen_ has quit IRC | 00:21 | |
| jeblair | clarkb: yes, i think it's that simple. (requirements update) | 00:21 |
| *** slong has quit IRC | 00:22 | |
| *** slong_ has joined #openstack-infra | 00:22 | |
| openstackgerrit | A change was merged to openstack-infra/gear: geard: wrong reference to setup_logging() https://review.openstack.org/58824 | 00:26 |
| *** vipul is now known as vipul-away | 00:27 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix merging transitively-connected change queues. https://review.openstack.org/59865 | 00:29 |
| openstackgerrit | A change was merged to openstack-infra/config: Send clouddocs-maven-plugin events to doc channel https://review.openstack.org/58872 | 00:29 |
| lifeless | mordred: https://review.openstack.org/#/c/48803/ | 00:31 |
| *** matsuhashi has joined #openstack-infra | 00:31 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove remote-directory from publishing in api-jobs https://review.openstack.org/59538 | 00:31 |
| *** sarob has quit IRC | 00:32 | |
| *** wabat has left #openstack-infra | 00:33 | |
| *** nati_ueno has quit IRC | 00:33 | |
| *** sarob has joined #openstack-infra | 00:33 | |
| *** ^d has quit IRC | 00:33 | |
| openstackgerrit | A change was merged to openstack-infra/config: Move ttx/os-releasing to os-infra/release-tools https://review.openstack.org/59156 | 00:35 |
| openstackgerrit | A change was merged to openstack-infra/config: Enable #openstack-climate IRC channel logging https://review.openstack.org/57675 | 00:37 |
| *** sarob has quit IRC | 00:37 | |
| *** dstanek has joined #openstack-infra | 00:38 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add oslo.messaging to release status tracking https://review.openstack.org/59374 | 00:40 |
| *** vipul-away is now known as vipul | 00:40 | |
| *** balar has joined #openstack-infra | 00:41 | |
| clarkb | I am no longer afk | 00:43 |
| clarkb | woo | 00:43 |
| clarkb | EmilienM: for grenade no, we haven;t sorted that out yet | 00:43 |
| clarkb | I think grenade is still in a state of flux, we should sort it out, but I am not going to -1 changes to grenade jobs that don't follow the devstack job templating | 00:44 |
| clarkb | looking at the zuul fix now | 00:45 |
| clarkb | fungi: yes thank you (re commit message update) | 00:46 |
| *** openstack has joined #openstack-infra | 00:46 | |
| -hobana.freenode.net- [freenode-info] channel flooding and no channel staff around to help? Please check with freenode support: http://freenode.net/faq.shtml#gettinghelp | 00:46 | |
| openstackgerrit | A change was merged to openstack-infra/gear: Correctly set job known/running attributes https://review.openstack.org/59064 | 00:47 |
| openstackgerrit | A change was merged to openstack-infra/git-review: Ignore content of merge commits in reporting https://review.openstack.org/50401 | 00:49 |
| *** salv-orlando has quit IRC | 00:53 | |
| clarkb | jeblair: I am trying to understand the zuul change. I grok the intent, but I don't see where the shared state is carried across each iteration of the new loop | 00:55 |
| clarkb | jeblair: does the argument to combineChangeQueues need to be last_change_queues instead of change_queues? | 00:55 |
| *** sarob has joined #openstack-infra | 00:55 | |
| *** openstackgerrit has quit IRC | 00:56 | |
| *** openstackgerrit has joined #openstack-infra | 00:56 | |
| clarkb | jeblair: I think maybe the code works there because one extra pass is made? | 00:57 |
| clarkb | I will write notes shortly | 00:57 |
| *** mrodden1 has joined #openstack-infra | 01:02 | |
| *** mrodden has quit IRC | 01:03 | |
| *** dims has quit IRC | 01:04 | |
| jeblair | clarkb: i think you're on to something. :) | 01:10 |
| clarkb | jeblair: notes just about done | 01:10 |
| clarkb | jeblair: and posted | 01:11 |
| *** sarob has quit IRC | 01:11 | |
| *** sarob has joined #openstack-infra | 01:12 | |
| *** reed has quit IRC | 01:12 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix merging transitively-connected change queues. https://review.openstack.org/59865 | 01:13 |
| jeblair | clarkb: thanks, nice catch | 01:13 |
| clarkb | jeblair: I think there is still a small problem in that patchset :) | 01:14 |
| jeblair | clarkb: hehe | 01:15 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Fix merging transitively-connected change queues. https://review.openstack.org/59865 | 01:15 |
| clarkb | and posted a review | 01:15 |
| clarkb | there we go | 01:15 |
| clarkb | I am +2 as soon as jenkins is happy with it | 01:16 |
| *** sarob has quit IRC | 01:16 | |
| *** mfer has joined #openstack-infra | 01:18 | |
| clarkb | fungi: any reason for not approving https://review.openstack.org/#/c/54639/6 ? I am going to approve it shortly if you don't have strong feelings about it | 01:18 |
| *** mfer has quit IRC | 01:18 | |
| *** dims has joined #openstack-infra | 01:19 | |
| fungi | clarkb: since it was something mordred had originally requested i figured he might want to have a look too, but i'm good with merging it | 01:19 |
| *** krotscheck has quit IRC | 01:19 | |
| clarkb | fungi: I really like the info it gives us, and we can tweak later if it needs to do more or different things | 01:20 |
| fungi | also, i mostly don't aprv patches to pbr which mordred has not at least +2'd, but this is tests only | 01:20 |
| jerryz | clarkb: i notice that latest elastic- recheck's comment will include recheck/reverify bug #, won't this trigger a rebuild automatically? | 01:21 |
| clarkb | ya if this was pbr proper I probably would defer to him but its in the test and informational only so +A | 01:21 |
| clarkb | jerryz: no, the strings to recheck/reverify are very specific and must be 'recheck bug XXXXXX' basically | 01:21 |
| fungi | jerryz: zuul only considers it if it's the *only* text in the comment | 01:21 |
| fungi | the elastic-recheck comments are taking that into consideration by design, because they want an actual human to make sure that's sane before adding it as a separate comment | 01:22 |
| *** julim has quit IRC | 01:22 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/zuul: Ignore .venv directory https://review.openstack.org/58080 | 01:24 |
| clarkb | I am going to self approve ^ it is a simple tiny change that needed to be detached from its parent so that gate testing could happen | 01:24 |
| clarkb | trivial rebase detection picked up on that \o/ | 01:24 |
| fungi | i can review | 01:24 |
| fungi | lgtm, approved | 01:25 |
| jerryz | fungi clarkb: i see. thanks | 01:25 |
| clarkb | fungi: danke | 01:25 |
| fungi | nothing like approving a one-line change to make me feel productive, after all ;) | 01:26 |
| *** nati_ueno has joined #openstack-infra | 01:26 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Enable requirement update proposal jobs https://review.openstack.org/59855 | 01:28 |
| openstackgerrit | A change was merged to openstack-infra/zuul: Ignore .venv directory https://review.openstack.org/58080 | 01:29 |
| *** nati_ueno has quit IRC | 01:32 | |
| *** nati_ueno has joined #openstack-infra | 01:32 | |
| *** nati_ueno has quit IRC | 01:33 | |
| *** kui has joined #openstack-infra | 01:33 | |
| *** nati_ueno has joined #openstack-infra | 01:33 | |
| *** nosnos has joined #openstack-infra | 01:35 | |
| *** jamesmcarthur has joined #openstack-infra | 01:35 | |
| *** nosnos_ has joined #openstack-infra | 01:37 | |
| *** nosnos has quit IRC | 01:40 | |
| *** wenlock has joined #openstack-infra | 01:47 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Run the mirror if we land changes to pypi-mirror https://review.openstack.org/56777 | 01:48 |
| clarkb | another change that needed rebasing to be mergable | 01:48 |
| clarkb | I am trying to get changes that have been approved through | 01:48 |
| fungi | approving that one once jenkins reports back | 01:49 |
| clarkb | woot thanks (I don't think it was a trivial rebase according to gerrit but really it is) | 01:49 |
| fungi | yeah. it's a 3-line change no matter how you slice it | 01:49 |
| fungi | sorry, 4 line | 01:49 |
| *** mfer has joined #openstack-infra | 01:50 | |
| clarkb | jog0: http://logs.openstack.org/03/48803/23/gate/gate-tempest-dsvm-full/b5ae916/console.html is that problem familiar? I am not finding a bug for it (but haven't looked beyond console log yet) | 01:51 |
| *** Ryan_Lane has joined #openstack-infra | 01:51 | |
| openstackgerrit | A change was merged to openstack-infra/config: Run the mirror if we land changes to pypi-mirror https://review.openstack.org/56777 | 01:53 |
| *** dstanek_afk has joined #openstack-infra | 01:55 | |
| *** Ryan_Lane has quit IRC | 01:56 | |
| fungi | i've similarly been trying to set first priority on changes which already have at least one +2 so we can get them cleared out | 02:00 |
| *** rongze has joined #openstack-infra | 02:01 | |
| *** rongze has quit IRC | 02:01 | |
| clarkb | looks like maybe tempest is failing to see a server as active? I am going to do that sad thing and submit a bug against tempest | 02:01 |
| *** rongze has joined #openstack-infra | 02:02 | |
| *** bingbu has joined #openstack-infra | 02:04 | |
| clarkb | I have reverified 48803 with a bug. Going to walk home now | 02:07 |
| fungi | ahh, bug 1257561 | 02:07 |
| uvirtbot | Launchpad bug 1257561 in tempest "test_create_backup test not noticing that server has become active" [Undecided,New] https://launchpad.net/bugs/1257561 | 02:07 |
| fungi | exciting | 02:08 |
| fungi | clarkb: https://review.openstack.org/59855 is not going to merge. i'll rebase it | 02:09 |
| fungi | er, i meant 48803 | 02:09 |
| fungi | (stupid paste buffers) | 02:10 |
| clarkb | gah, I should've checked that... oh well and thanks | 02:10 |
| clarkb | now I am really walking home | 02:10 |
| fungi | yeah, it's hanging out at the back of the gate until i get this rebase submitted, then i'll approve once it passes check | 02:11 |
| fungi | specifically, it's conflicting with 54639 (currently ahead of it in the gate_ | 02:14 |
| zaro | fungi: approve away on 56412 | 02:14 |
| fungi | zaro: thanks--done | 02:15 |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: Deprecate projectName setting of CopyArtifact https://review.openstack.org/56412 | 02:15 |
| *** ruhe has joined #openstack-infra | 02:18 | |
| *** elasticio has joined #openstack-infra | 02:18 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 02:20 |
| *** dims has quit IRC | 02:23 | |
| *** Abhishek has joined #openstack-infra | 02:24 | |
| *** fungi has quit IRC | 02:24 | |
| Abhishek | hey guys, I am trying to write a new gating job | 02:24 |
| Abhishek | how do I install a python module for the job? | 02:24 |
| Abhishek | is it okay to just do a pip install in a bash script? | 02:25 |
| *** fungi has joined #openstack-infra | 02:26 | |
| *** guohliu has joined #openstack-infra | 02:27 | |
| mestery | Howdy folks, anyone else noticing the openstack-dev and openstack lists appear to be borked? | 02:27 |
| mestery | I haven't received emails on either since approx. 2:35PM CST or so. | 02:27 |
| mestery | For reference, this is the last email I received from openstack-dev today: http://lists.openstack.org/pipermail/openstack-dev/2013-December/021046.html | 02:31 |
| fungi | mestery: i've received messages from them as recently as 2013-12-04 02:04:41 utc | 02:31 |
| mestery | fungi: If you look at the archives, since the email I just referenced there have only been about 20 more. | 02:31 |
| mestery | That's curiously low for openstack-dev to me. | 02:31 |
| mestery | fungi: I checked with one other Cisco person who also hasn't received any since the same time as me. | 02:32 |
| mestery | So unsure if it's a Cisco problem or not ... | 02:32 |
| mestery | fungi: Likely a Cisco problem of some sort perhaps. | 02:32 |
| fungi | i've been receiving quite a few on both lists you mentioned since that time | 02:32 |
| mestery | fungi: thanks for the point of reference! | 02:32 |
| * mestery keeps digging and makes a note to move his list traffic off of cisco's servers going forward. | 02:32 | |
| fungi | yeah, the most recent messages in the -dev archive look familiar (though i've already deleted my copies, i could match message ids from my mta logs) | 02:33 |
| * fungi checks the delivery queue on the listserv for cisco domains | 02:34 | |
| mestery | fungi: Thanks for the help! | 02:34 |
| *** rwsu has quit IRC | 02:35 | |
| *** esker has joined #openstack-infra | 02:35 | |
| fungi | mestery: aha! | 02:39 |
| fungi | 2013-12-04 02:23:03 1Vo27D-0003zX-IO ** kmestery@cisco.com R=dnslookup T=remote_smtp: SMTP error from remote mail server after RCPT TO:<kmestery@cisco.com>: host alln-mx-01.cisco.com [2001:420:1201:6::ad25:93e6]: 550 Connections from the host lists.openstack.org (2001:4800:780e:510:3bc3:d7f6:ff04:b736), originating from SenderBase Network Owner ID: None, are being rejected due to a low SenderBase | 02:39 |
| fungi | Reputation Score. See http://www.senderbase.org for more information or contact your IT support team. | 02:39 |
| *** rwsu has joined #openstack-infra | 02:39 | |
| mestery | fungi: So it looks like we're blocking it, thanks for the tip. | 02:39 |
| fungi | mestery: senderbase was bought by ironport was bought by cisco, right? | 02:39 |
| fungi | or am i misremembering? | 02:40 |
| mestery | fungi: No, you've remembered correctly. | 02:40 |
| mestery | fungi: I'll file an IT case internally. *sigh* | 02:40 |
| * fungi goes back to not being a network ops guy | 02:40 | |
| * mestery grumbles about just wanted to get work done. | 02:40 | |
| mestery | fungi: Heh, thanks for the help! | 02:40 |
| fungi | np | 02:40 |
| jamesmcarthur | sdague: hi - we're currently trying to troubleshoot the issues you're having with the openstack.org profile page. | 02:41 |
| fungi | if your internal ticket happens to mention that senderbase should give openstack servers a better reputation, especially the ones hosting technical mailing lists which have never sent me any spam, that would also rock | 02:41 |
| fungi | mestery: ^ | 02:41 |
| mestery | fungi: Got it, will do. | 02:41 |
| jamesmcarthur | sdague: was wondering if you could provide any other info besides what you sent to Jonathan and Todd. | 02:42 |
| fungi | jamesmcarthur: excellent--we've gotten several reports, of broken in the affiliations/contact profile bits, so it's appreciated | 02:42 |
| jamesmcarthur | sdague: having trouble replicating the problem on all browsers without actually cutting off our internet connection | 02:42 |
| jamesmcarthur | fungi: cool. well i'm happy to field questions/take info directly. | 02:42 |
| jamesmcarthur | fungi: we'd like to get this nailed down ASAP | 02:43 |
| fungi | jamesmcarthur: i'll try to remember to send them your way. there were a couple who said they were unable to change their contact addresses without first adding an affiliation in their profile. that's teh main one which comes to mind | 02:43 |
| jamesmcarthur | fungi: we did find another error with the DOM that we're updating tomorrow, but I'm not 100% that's related to our issue here. | 02:44 |
| fungi | jamesmcarthur: great. i'll keep an ear to the ground | 02:44 |
| jamesmcarthur | fungi: the best thing to do is submit a ticket to https://bugs.launchpad.net/openstack-org/+bugs | 02:44 |
| fungi | and will send them there. i didn't know about that lp project existing, so thanks for the tip! | 02:44 |
| jamesmcarthur | fungi: we've got our eyes on this now and will be monitoring/reviewing/addressing bugs that come in for openstack.org and will be trying to stay on top of things from here forward | 02:45 |
| jamesmcarthur | fungi: also, do you sleep? you're online all the time ;) | 02:45 |
| fungi | jamesmcarthur: i'm working on jacking an internet into my skull so i can irc WHILE i sleep. putting off further sleep until that's done | 02:46 |
| *** marun has joined #openstack-infra | 02:46 | |
| mgagne | jamesmcarthur: he installed an irc client for google glass | 02:46 |
| jamesmcarthur | mgagne: is that true? because that would be quite cool. | 02:46 |
| fungi | mgagne: close. i have my tmux session via mosh from a beagleboard plugged into a pair of video goggles | 02:46 |
| fungi | (no joke) | 02:47 |
| mgagne | see? :D | 02:47 |
| jamesmcarthur | yes | 02:47 |
| jamesmcarthur | i feel like i'm not trying hard enough now ;) | 02:47 |
| * mgagne got lucky | 02:47 | |
| jamesmcarthur | but seriously, that's awesome | 02:48 |
| fungi | you couldn't try less hard than me. i'm currently watching star trek on blu-ray | 02:48 |
| fungi | it's slack time for me right now | 02:48 |
| *** MarkAtwood has quit IRC | 02:49 | |
| *** Ryan_Lane has joined #openstack-infra | 02:51 | |
| *** Abhishek has quit IRC | 02:52 | |
| *** Ryan_Lane has quit IRC | 02:55 | |
| *** jaypipes has quit IRC | 02:57 | |
| openstackgerrit | Mathieu Gagné proposed a change to openstack-infra/config: Remove puppet-openstack_dev_env from StackForge https://review.openstack.org/59888 | 03:00 |
| *** ruhe has quit IRC | 03:01 | |
| *** vkozhukalov has joined #openstack-infra | 03:09 | |
| *** elasticio has quit IRC | 03:12 | |
| *** MarkAtwood has joined #openstack-infra | 03:12 | |
| *** MarkAtwood has quit IRC | 03:13 | |
| *** MarkAtwood has joined #openstack-infra | 03:13 | |
| *** vipul has quit IRC | 03:15 | |
| *** vipul has joined #openstack-infra | 03:16 | |
| *** soren has quit IRC | 03:17 | |
| *** rainya has quit IRC | 03:17 | |
| *** adam_g has quit IRC | 03:17 | |
| *** antonym has quit IRC | 03:18 | |
| *** harlowja has quit IRC | 03:19 | |
| *** rainya has joined #openstack-infra | 03:19 | |
| *** Shrews has quit IRC | 03:19 | |
| *** pvo has quit IRC | 03:19 | |
| *** Shrews has joined #openstack-infra | 03:19 | |
| *** antonym has joined #openstack-infra | 03:20 | |
| *** pvo has joined #openstack-infra | 03:22 | |
| *** soren has joined #openstack-infra | 03:23 | |
| clarkb | I think we need to make hpcloud region a and b devstack-precise-check only | 03:27 |
| fungi | slow still? | 03:28 |
| *** mfer has quit IRC | 03:29 | |
| clarkb | fungi: ya | 03:30 |
| clarkb | fungi: https://jenkins02.openstack.org/job/gate-tempest-dsvm-full/262/ is holding up the gate | 03:30 |
| clarkb | slightly afraid it is going to hit a timeout | 03:31 |
| clarkb | mordred: ^ how do we give that feedback to HP? | 03:31 |
| *** esker has quit IRC | 03:34 | |
| fungi | ick | 03:34 |
| *** rcleere has joined #openstack-infra | 03:34 | |
| clarkb | gah it timed out | 03:34 |
| clarkb | it almost made it through | 03:35 |
| *** gyee has quit IRC | 03:35 | |
| clarkb | going to propose the change to check pipeline only them | 03:35 |
| clarkb | that is frustrating | 03:35 |
| fungi | well, rs iad didn't prove out faster either | 03:37 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Don't use new HPCloud regions in the gate. https://review.openstack.org/59891 | 03:38 |
| clarkb | fungi: right, but this is a step backwards for hpcloud | 03:38 |
| fungi | or a step forward, if one of the goals was to prevent tenants from consuming too many resources | 03:39 |
| clarkb | stepbackward for us as customers :) | 03:40 |
| fungi | so true | 03:45 |
| fungi | clouds | 03:45 |
| *** loq_mac has joined #openstack-infra | 03:45 | |
| zaro | fungi, clarkb : i'm thinking about doing gerrit upgrade tomorrow. i think i need access to review-dev right? | 03:46 |
| *** Ryan_Lane has joined #openstack-infra | 03:51 | |
| fungi | we have review-dev under bup, right? so if an in-place upgrade via puppet went horribly wrong, we could presumably restore it in the worst possible case | 03:53 |
| clarkb | yes I think so | 03:54 |
| clarkb | zaro: no, it can all be done via puppet | 03:54 |
| fungi | zaro: what is the upgrade path from 2.4 to 2.8? | 03:54 |
| fungi | incremental/stepwise? | 03:54 |
| zaro | great! i'm pretty sure you can go 2.4 -> 2.8 | 03:54 |
| zaro | i'll double check though. | 03:55 |
| *** amotoki has joined #openstack-infra | 03:55 | |
| zaro | what's the restore mechanism? | 03:55 |
| *** Ryan_Lane has quit IRC | 03:56 | |
| fungi | the restore mechanism is one of the root users invoking bup restore, then moving files into place and reloading the db from the last mysqldump in the backup | 03:57 |
| fungi | (in other words, manual) | 03:58 |
| zaro | what's the script in puppet that can do a gerrit upgrade? | 03:59 |
| clarkb | zaro: there isn't a script, you just change the war file | 03:59 |
| clarkb | it is all done through puppet | 04:00 |
| zaro | need to do more than just change out the file. i think you need to run a command to do the upgrade. | 04:01 |
| clarkb | puppet takes care of it | 04:01 |
| clarkb | you have to run the command that does DB migrations | 04:02 |
| *** herndon_ has quit IRC | 04:06 | |
| *** nati_ueno has quit IRC | 04:08 | |
| zaro | i see the gerrit wars are hosted on tarballs.o.o | 04:09 |
| clarkb | zaro: right so that is why jeblair was talking about creating a 2.8 branch and putting your patch on it so that we would get a war built for it | 04:09 |
| zaro | clarkb, fungi : do i have access to put new files on tarballs? | 04:09 |
| clarkb | zaro: no, they are put there by jenkins when changes are proposed to and merge on our gerrit repo | 04:10 |
| * zaro gets it now | 04:11 | |
| zaro | ohh wait. 2.8 branch would require new tooling to build. | 04:11 |
| clarkb | because of buck? | 04:12 |
| zaro | newer gerrit uses buck instead of maven | 04:12 |
| clarkb | so we may need to tackle that first | 04:12 |
| zaro | would that mean new job to build as well? | 04:12 |
| zaro | so new job that builds gerrit 2.8 branch? | 04:14 |
| zaro | i believe buck is build from source as well. i don't think there are packages for buck yet. | 04:14 |
| *** CaptTofu has quit IRC | 04:15 | |
| *** CaptTofu has joined #openstack-infra | 04:16 | |
| *** basic` has quit IRC | 04:17 | |
| *** basic` has joined #openstack-infra | 04:18 | |
| *** basic` has joined #openstack-infra | 04:18 | |
| *** MarkAtwood has quit IRC | 04:23 | |
| *** jerryz has quit IRC | 04:23 | |
| clarkb | zaro: it might require different jobs, zuul can figure out which one to run based on the branch | 04:28 |
| *** prad has quit IRC | 04:33 | |
| *** chandankumar has joined #openstack-infra | 04:36 | |
| *** loq_mac has quit IRC | 04:37 | |
| *** nosnos has joined #openstack-infra | 04:38 | |
| *** nosnos_ has quit IRC | 04:42 | |
| mordred | maybe we should try just using the upstream war? | 04:45 |
| *** rongze has quit IRC | 04:45 | |
| *** StevenK has joined #openstack-infra | 04:45 | |
| *** Ryan_Lane has joined #openstack-infra | 04:51 | |
| *** FallenPegasus has joined #openstack-infra | 04:52 | |
| clarkb | mordred: we could try that with a build off of master | 04:52 |
| *** Ryan_Lane has quit IRC | 04:56 | |
| *** SergeyLukjanov has joined #openstack-infra | 04:56 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 04:57 | |
| *** _SergeyLukjanov is now known as SergeyLukjanov | 04:57 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack https://review.openstack.org/58702 | 05:00 |
| * zaro likes mordred's idea | 05:03 | |
| *** guohliu has quit IRC | 05:03 | |
| *** rcleere has quit IRC | 05:05 | |
| zaro | time for some zzzzzzz.. | 05:05 |
| *** jamesmcarthur has quit IRC | 05:06 | |
| SergeyLukjanov | clarkb, hi | 05:08 |
| SergeyLukjanov | clarkb, you've placed comment in https://review.openstack.org/#/c/58702/4/modules/openstack_project/files/jenkins_job_builder/config/climate.yaml | 05:08 |
| SergeyLukjanov | to name gate-climate-devstack-dsvm | 05:08 |
| SergeyLukjanov | why devstack-dsvm if dsvm == devstack-vm? | 05:09 |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack https://review.openstack.org/58702 | 05:09 |
| clarkb | because of the way zuul knows that a test is running on a single use VM. it is just a regex on the job name | 05:14 |
| clarkb | you can see this in the layout.yaml file under the jobs list | 05:14 |
| SergeyLukjanov | name: ^.*(dsvm|devstack-vm).*$ | 05:15 |
| clarkb | ya that | 05:15 |
| SergeyLukjanov | it looks like it'll not accept devstack-dsvm | 05:15 |
| clarkb | it will .*dsvm.* matches | 05:15 |
| *** mrodden has joined #openstack-infra | 05:15 | |
| *** rongze has joined #openstack-infra | 05:16 | |
| SergeyLukjanov | oops, incorrect wording, I mean that -dsvm is enough and so why to name it "-devstack-devstack-vm"? | 05:16 |
| *** yjiang5_1 has joined #openstack-infra | 05:17 | |
| *** guitarza1 has joined #openstack-infra | 05:17 | |
| clarkb | oh because it is running the devstack exercisea. should be gate-climate-devstack-dsvm | 05:17 |
| *** gfa_ has joined #openstack-infra | 05:19 | |
| SergeyLukjanov | clarkb, ok, got it, thanks | 05:19 |
| *** rainya has quit IRC | 05:20 | |
| *** LinuxJedi_ has joined #openstack-infra | 05:20 | |
| StevenK | clarkb: Was it you that mentioned the Git for 4 years and up talk yesterday? I watched the LCA 2013 recording of it | 05:21 |
| *** michchap_ has joined #openstack-infra | 05:21 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack https://review.openstack.org/58702 | 05:21 |
| clarkb | StevenK: yes, Schwern is a friends roommate and I have seen the talk too. I find it really helpful understanding what git is doing | 05:21 |
| *** marun has quit IRC | 05:22 | |
| *** mrodden1 has quit IRC | 05:22 | |
| *** guitarzan has quit IRC | 05:22 | |
| *** stevebaker has quit IRC | 05:22 | |
| *** sandywalsh_ has quit IRC | 05:22 | |
| *** yjiang5 has quit IRC | 05:22 | |
| openstackgerrit | Sergey Lukjanov proposed a change to openstack-infra/config: Add gate-climate-devstack-dsvm https://review.openstack.org/58702 | 05:22 |
| *** michchap has quit IRC | 05:22 | |
| *** gfa has quit IRC | 05:22 | |
| *** LinuxJedi has quit IRC | 05:22 | |
| *** kpepple has quit IRC | 05:22 | |
| *** ttx has quit IRC | 05:22 | |
| *** pvo has quit IRC | 05:22 | |
| clarkb | makes understanding the differences between branches and tags and detached heads easy as well as rebase and other things | 05:22 |
| *** kpepple has joined #openstack-infra | 05:22 | |
| *** antonym has quit IRC | 05:22 | |
| *** ttx has joined #openstack-infra | 05:22 | |
| StevenK | I don't think I want to build a model of an openstack repo, though ... | 05:22 |
| *** stevebaker has joined #openstack-infra | 05:22 | |
| *** marun has joined #openstack-infra | 05:22 | |
| *** pvo has joined #openstack-infra | 05:22 | |
| *** ttx has quit IRC | 05:22 | |
| *** ttx has joined #openstack-infra | 05:22 | |
| *** rainya has joined #openstack-infra | 05:22 | |
| *** rongze has quit IRC | 05:28 | |
| openstackgerrit | A change was merged to openstack-infra/devstack-gate: Use get-pip for pip installation https://review.openstack.org/58106 | 05:28 |
| *** amotoki has quit IRC | 05:30 | |
| *** sandywalsh_ has joined #openstack-infra | 05:30 | |
| *** MarkAtwood has joined #openstack-infra | 05:36 | |
| openstackgerrit | A change was merged to openstack-dev/pbr: show diff between pip installed packages and requirements https://review.openstack.org/54639 | 05:36 |
| *** FallenPegasus has quit IRC | 05:38 | |
| chandankumar | Hello | 05:44 |
| chandankumar | when i send a review to openstack review. | 05:44 |
| chandankumar | when jenkins review it , i am not getting any mails in my mailbox. | 05:45 |
| clarkb | chandankumar: under your gerrit settings have you configured gerrit to send you email for things? | 05:45 |
| clarkb | also double check your prefered email address is correct | 05:45 |
| chandankumar | clarkb, yes | 05:46 |
| chandankumar | clarkb, the email is configured there. | 05:47 |
| chandankumar | clarkb, it is correct. | 05:48 |
| chandankumar | clarkb, it is happening from that day when i use my laptop to send reviews to gerrit | 05:48 |
| chandankumar | generally i use my desktop to send review | 05:48 |
| chandankumar | clarkb, i am not figuring out what is happening with it. | 05:50 |
| clarkb | chandankumar: do you have an example of a change that worked and a change that did not work? | 05:50 |
| chandankumar | clarkb, for thsi review https://review.openstack.org/#/c/59903/ | 05:52 |
| chandankumar | *this | 05:52 |
| chandankumar | when tom review my patch i have not got the mail | 05:52 |
| *** nicedice has quit IRC | 05:53 | |
| clarkb | I am still getting email so it isn't breaking everywhere. However I stopped sending gerrit mail to my gmail account | 05:55 |
| clarkb | chandankumar: have you ever received email from gerrit? I think you may need to confirm the address with gerrit (it has been so long I am trying to remember if that was requried) | 05:56 |
| chandankumar | clarkb, confirmation email i have not received from gerrit | 05:57 |
| *** dstanek has quit IRC | 05:57 | |
| clarkb | let me look in my email history to see if I got one of those | 05:57 |
| *** rongze has joined #openstack-infra | 05:58 | |
| clarkb | I got one when I added my second address but don't see one for the original | 05:59 |
| clarkb | chandankumar: preferred email is set correctly at https://review.openstack.org/#/settings/contact ? | 05:59 |
| *** dstanek has joined #openstack-infra | 06:00 | |
| chandankumar | clarkb, it is setted there. | 06:00 |
| chandankumar | clarkb, no idea what is happening with my gerrit account. | 06:02 |
| clarkb | is it possible that the mail from review@openstack.org is being filter? | 06:03 |
| openstackgerrit | A change was merged to openstack-dev/pbr: Use wheels for installation https://review.openstack.org/48803 | 06:03 |
| chandankumar | clarkb, after 29 nov , i have not received mail from review@openstack.org | 06:04 |
| *** SergeyLukjanov has quit IRC | 06:04 | |
| clarkb | chandankumar: can you link to a change where you did receive email? | 06:04 |
| *** SergeyLukjanov has joined #openstack-infra | 06:05 | |
| *** paul-- has quit IRC | 06:07 | |
| clarkb | did you get email for https://review.openstack.org/#/c/58990/ ? | 06:07 |
| *** SergeyLukjanov has quit IRC | 06:07 | |
| *** paul-- has joined #openstack-infra | 06:10 | |
| clarkb | my ability to debug right now is limited (almost bed time) so trying to gather as much info as possible | 06:10 |
| chandankumar | clarkb, no i have not received | 06:13 |
| clarkb | chandankumar: did it ever work? if so please link to a change where it did work | 06:14 |
| *** paul-- has quit IRC | 06:14 | |
| chandankumar | clarkb, up to this patch Fix the wrong PATCH body of v2 for image update | 06:14 |
| chandankumar | https://review.openstack.org/58437 | 06:14 |
| chandankumar | i have received the mails | 06:14 |
| clarkb | interesting so you did receive email for a change submitted by someone else that you commented on | 06:16 |
| clarkb | but not for changes submitted by you | 06:16 |
| chandankumar | yes | 06:16 |
| clarkb | ok good to know, I will try poking at it tomorrow | 06:17 |
| chandankumar | clarkb, Thank you | 06:18 |
| *** dstanek has quit IRC | 06:20 | |
| *** kui has quit IRC | 06:21 | |
| *** fbo_away is now known as fbo | 06:26 | |
| *** fifieldt_ has quit IRC | 06:27 | |
| *** rongze has quit IRC | 06:30 | |
| *** rongze has joined #openstack-infra | 06:30 | |
| *** pcrews has quit IRC | 06:32 | |
| *** nosnos has quit IRC | 06:34 | |
| *** nosnos has joined #openstack-infra | 06:35 | |
| *** jcooley_ has joined #openstack-infra | 06:41 | |
| *** sarob has joined #openstack-infra | 06:42 | |
| *** fbo is now known as fbo_away | 06:50 | |
| *** che-arne_ has quit IRC | 06:51 | |
| *** jcooley_ has quit IRC | 06:51 | |
| *** boris-42 has joined #openstack-infra | 06:51 | |
| *** vkozhukalov has quit IRC | 06:51 | |
| *** Ryan_Lane has joined #openstack-infra | 06:51 | |
| *** nati_ueno has joined #openstack-infra | 06:52 | |
| *** SergeyLukjanov has joined #openstack-infra | 06:55 | |
| *** Ryan_Lane has quit IRC | 06:56 | |
| *** marun has quit IRC | 06:57 | |
| *** DinaBelova has joined #openstack-infra | 06:57 | |
| *** denis_makogon has joined #openstack-infra | 06:58 | |
| *** yolanda has joined #openstack-infra | 06:58 | |
| *** MarkAtwood has quit IRC | 07:02 | |
| *** wenlock has quit IRC | 07:03 | |
| *** nosnos_ has joined #openstack-infra | 07:05 | |
| *** guohliu has joined #openstack-infra | 07:06 | |
| *** nosnos has quit IRC | 07:08 | |
| *** nati_ueno has quit IRC | 07:10 | |
| *** antonym has joined #openstack-infra | 07:17 | |
| *** rwsu has quit IRC | 07:24 | |
| *** sarob has quit IRC | 07:25 | |
| *** sarob has joined #openstack-infra | 07:25 | |
| *** yongli has quit IRC | 07:28 | |
| *** sarob has quit IRC | 07:30 | |
| *** amotoki has joined #openstack-infra | 07:33 | |
| *** pblaho has joined #openstack-infra | 07:37 | |
| jog0 | clarkb: I see a glance pull loop | 07:38 |
| jog0 | this *may* be related http://logs.openstack.org/03/48803/23/gate/gate-tempest-dsvm-full/b5ae916/logs/screen-g-api.txt.gz?level=TRACE | 07:39 |
| jog0 | clarkb: here is teh bug http://logs.openstack.org/03/48803/23/gate/gate-tempest-dsvm-full/b5ae916/logs/screen-n-cpu.txt.gz?level=TRACE#_2013-12-03_12_23_26_244 | 07:46 |
| jog0 | libvirt | 07:46 |
| jog0 | so yes we havea bug for it | 07:46 |
| jog0 | I think thast it at least | 07:47 |
| jog0 | I think we have a nasty libvirt bug(s) | 07:48 |
| *** SergeyLukjanov has quit IRC | 07:48 | |
| *** rwsu has joined #openstack-infra | 07:48 | |
| *** fifieldt has quit IRC | 07:49 | |
| *** Ryan_Lane has joined #openstack-infra | 07:51 | |
| *** cederb02 has joined #openstack-infra | 07:52 | |
| *** afazekas has quit IRC | 07:55 | |
| *** Ryan_Lane has quit IRC | 07:55 | |
| *** sarob has joined #openstack-infra | 07:56 | |
| *** flaper87|afk is now known as flaper87 | 07:59 | |
| *** cederb02 has quit IRC | 08:01 | |
| *** avishayb has joined #openstack-infra | 08:08 | |
| *** marun has joined #openstack-infra | 08:09 | |
| *** cederb02 has joined #openstack-infra | 08:12 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1257626 https://review.openstack.org/59919 | 08:14 |
| uvirtbot | Launchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"" [Critical,New] https://launchpad.net/bugs/1257626 | 08:14 |
| *** cederb02 has quit IRC | 08:16 | |
| *** guohliu has quit IRC | 08:17 | |
| *** sarob has quit IRC | 08:18 | |
| *** nosnos_ has quit IRC | 08:19 | |
| openstackgerrit | Sascha Peilicke proposed a change to openstack/requirements: Allow Django 1.5 and 1.6 https://review.openstack.org/59920 | 08:19 |
| *** nosnos has joined #openstack-infra | 08:19 | |
| *** inkerra has quit IRC | 08:20 | |
| *** metabro has quit IRC | 08:20 | |
| jog0 | clarkb: for bug 1257626, were there any changes to hpcloud recently? | 08:21 |
| uvirtbot | Launchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"" [Critical,New] https://launchpad.net/bugs/1257626 | 08:22 |
| *** GheRivero has quit IRC | 08:22 | |
| jog0 | and how we use it, if so that may trigger this | 08:22 |
| *** inkerra has joined #openstack-infra | 08:22 | |
| *** metabro has joined #openstack-infra | 08:22 | |
| *** Ryan_Lane has joined #openstack-infra | 08:29 | |
| *** markwash has quit IRC | 08:29 | |
| *** vkozhukalov has joined #openstack-infra | 08:30 | |
| *** markmcclain has joined #openstack-infra | 08:33 | |
| *** matsuhashi has quit IRC | 08:35 | |
| *** matsuhashi has joined #openstack-infra | 08:36 | |
| *** SergeyLukjanov has joined #openstack-infra | 08:38 | |
| *** boris-42 has quit IRC | 08:40 | |
| *** denis_makogon has quit IRC | 08:41 | |
| *** moted has quit IRC | 08:41 | |
| *** moted has joined #openstack-infra | 08:42 | |
| *** sileht has quit IRC | 08:43 | |
| *** sandywalsh_ has quit IRC | 08:44 | |
| *** sileht has joined #openstack-infra | 08:47 | |
| *** SergeyLukjanov has quit IRC | 08:49 | |
| *** nprivalova has joined #openstack-infra | 08:56 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/elastic-recheck: Add query for bug 1257641 https://review.openstack.org/59923 | 08:56 |
| uvirtbot | Launchpad bug 1257641 in tempest "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances" [Undecided,New] https://launchpad.net/bugs/1257641 | 08:56 |
| *** sandywalsh_ has joined #openstack-infra | 08:57 | |
| *** jcoufal has joined #openstack-infra | 08:58 | |
| *** hashar has joined #openstack-infra | 09:00 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1257626 https://review.openstack.org/59919 | 09:05 |
| uvirtbot | Launchpad bug 1257626 in nova "Timeout while waiting on RPC response - topic: "network", RPC method: "allocate_for_instance" info: "<unknown>"" [Critical,New] https://launchpad.net/bugs/1257626 | 09:05 |
| *** mkoderer_ has joined #openstack-infra | 09:05 | |
| *** zigo has joined #openstack-infra | 09:05 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: Add query for bug 1257641 https://review.openstack.org/59923 | 09:05 |
| uvirtbot | Launchpad bug 1257641 in tempest "Quota exceeded for instances: Requested 1, but already used 10 of 10 instances" [Undecided,New] https://launchpad.net/bugs/1257641 | 09:05 |
| *** branen__ has joined #openstack-infra | 09:05 | |
| *** primemin1sterp has joined #openstack-infra | 09:07 | |
| *** amotoki has quit IRC | 09:08 | |
| *** ritesh has joined #openstack-infra | 09:08 | |
| *** jpich has joined #openstack-infra | 09:08 | |
| *** fesp has joined #openstack-infra | 09:09 | |
| *** belliott_ has joined #openstack-infra | 09:11 | |
| *** SlickN1k has joined #openstack-infra | 09:11 | |
| *** branen_ has quit IRC | 09:12 | |
| *** zigo_ has quit IRC | 09:12 | |
| *** primeministerp has quit IRC | 09:12 | |
| *** mkoderer has quit IRC | 09:12 | |
| *** Alex_Gaynor has quit IRC | 09:12 | |
| *** flaper87 has quit IRC | 09:12 | |
| *** zaro has quit IRC | 09:12 | |
| *** clarkb has quit IRC | 09:12 | |
| *** belliott has quit IRC | 09:12 | |
| *** SlickNik has quit IRC | 09:12 | |
| *** che-arne has joined #openstack-infra | 09:12 | |
| *** fesp is now known as flaper87 | 09:12 | |
| *** belliott_ is now known as belliott | 09:12 | |
| *** SlickN1k is now known as SlickNik | 09:12 | |
| *** mkoderer_ is now known as mkoderer | 09:12 | |
| *** markmcclain has quit IRC | 09:13 | |
| *** flaper87 has quit IRC | 09:13 | |
| *** flaper87 has joined #openstack-infra | 09:13 | |
| *** derekh has joined #openstack-infra | 09:15 | |
| *** yongli has joined #openstack-infra | 09:17 | |
| *** sarob has joined #openstack-infra | 09:17 | |
| *** zaro has joined #openstack-infra | 09:17 | |
| *** Alex_Gaynor has joined #openstack-infra | 09:18 | |
| *** clarkb has joined #openstack-infra | 09:19 | |
| *** SergeyLukjanov has joined #openstack-infra | 09:22 | |
| *** lianglin has joined #openstack-infra | 09:29 | |
| lianglin | @fungi, do you know if community accept to run Integration in simulator or not? | 09:30 |
| *** yassine has joined #openstack-infra | 09:30 | |
| *** sarob has quit IRC | 09:31 | |
| *** romcheg has quit IRC | 09:32 | |
| *** salv-orlando has joined #openstack-infra | 09:33 | |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge https://review.openstack.org/59932 | 09:35 |
| *** afazekas has joined #openstack-infra | 09:36 | |
| lianglin | where can I find the criterion of post CI test (including gate and third-party testing)? I would like to see what kind of test must be executed, can it run against simulator or not, voting or non-voting... | 09:36 |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge https://review.openstack.org/59932 | 09:37 |
| *** bingbu has quit IRC | 09:40 | |
| *** markmc has joined #openstack-infra | 09:42 | |
| *** boris-42 has joined #openstack-infra | 09:46 | |
| *** dizquierdo has joined #openstack-infra | 09:49 | |
| *** yaguang has joined #openstack-infra | 09:56 | |
| *** sarob has joined #openstack-infra | 09:56 | |
| *** salv-orlando_ has joined #openstack-infra | 10:01 | |
| *** nosnos_ has joined #openstack-infra | 10:03 | |
| *** salv-orlando has quit IRC | 10:03 | |
| *** salv-orlando_ is now known as salv-orlando | 10:03 | |
| *** jorisroovers has joined #openstack-infra | 10:03 | |
| *** nosnos has quit IRC | 10:06 | |
| *** Ryan_Lane has quit IRC | 10:09 | |
| openstackgerrit | Valeriy Ponomaryov proposed a change to openstack-infra/config: Add gate-manila-devstack job https://review.openstack.org/57987 | 10:09 |
| *** BobBallAway is now known as BobBall | 10:10 | |
| *** DinaBelova has quit IRC | 10:10 | |
| *** mkerrin has joined #openstack-infra | 10:13 | |
| *** avishayb has quit IRC | 10:18 | |
| *** matsuhashi has quit IRC | 10:22 | |
| *** nprivalova has quit IRC | 10:22 | |
| *** markmcclain has joined #openstack-infra | 10:22 | |
| *** lianglin has quit IRC | 10:28 | |
| *** sarob has quit IRC | 10:30 | |
| *** yaguang has quit IRC | 10:34 | |
| *** ritesh has quit IRC | 10:34 | |
| *** markmcclain has quit IRC | 10:35 | |
| *** johnthetubaguy has joined #openstack-infra | 10:35 | |
| *** paul-- has joined #openstack-infra | 10:35 | |
| *** markmcclain has joined #openstack-infra | 10:35 | |
| *** rossella_s has joined #openstack-infra | 10:35 | |
| *** nati_ueno has joined #openstack-infra | 10:37 | |
| *** nprivalova has joined #openstack-infra | 10:41 | |
| *** matsuhashi has joined #openstack-infra | 10:45 | |
| *** dizquierdo has quit IRC | 10:52 | |
| *** rongze has quit IRC | 10:53 | |
| *** DinaBelova has joined #openstack-infra | 10:55 | |
| *** sarob has joined #openstack-infra | 10:56 | |
| *** rongze has joined #openstack-infra | 11:11 | |
| *** SergeyLukjanov has quit IRC | 11:12 | |
| *** jorisroovers has quit IRC | 11:12 | |
| *** DinaBelova has quit IRC | 11:15 | |
| *** rfolco_ has joined #openstack-infra | 11:15 | |
| *** avishayb has joined #openstack-infra | 11:24 | |
| *** yassine has quit IRC | 11:25 | |
| *** sarob has quit IRC | 11:28 | |
| *** DinaBelova has joined #openstack-infra | 11:41 | |
| *** SergeyLukjanov has joined #openstack-infra | 11:43 | |
| *** jergerber has quit IRC | 11:43 | |
| *** rongze has quit IRC | 11:45 | |
| anteaya | lianglin: have you looked at this page? http://ci.openstack.org/third_party.html | 11:48 |
| *** dizquierdo has joined #openstack-infra | 11:49 | |
| *** rongze has joined #openstack-infra | 11:51 | |
| *** tma996 has joined #openstack-infra | 11:51 | |
| *** nprivalova has quit IRC | 11:55 | |
| *** sarob has joined #openstack-infra | 11:56 | |
| *** jorisroovers has joined #openstack-infra | 12:03 | |
| BobBall | Do the check jobs set up a tempest.conf file, or just use the defaults from tempest? | 12:03 |
| *** paul-- has quit IRC | 12:12 | |
| sdague | BobBall: they setup tempest via devstack | 12:17 |
| sdague | devstack will configure a tempest that tests the features you've turned on in it | 12:17 |
| *** prad has joined #openstack-infra | 12:17 | |
| *** dims has joined #openstack-infra | 12:18 | |
| BobBall | ah - do you know where that script is? | 12:19 |
| BobBall | I want to make a test change and have something that'll be usabel in check jobs... | 12:19 |
| BobBall | basically I want a new smaller flavor for tempest to use - it currently uses 512M instances in my setup and I don't know if devstack is already fixing that to create+use a smaller flavor - but cirros only needs 20M so I was going to give it 128M | 12:20 |
| BobBall | Parallel tempest tests are failing for me because it wants to start too many VMs at once :) | 12:20 |
| BobBall | and I run out of memory | 12:20 |
| sdague | BobBall: so we already do that | 12:22 |
| sdague | lib/tempest - and there are tiny and nano there | 12:22 |
| *** paul-- has joined #openstack-infra | 12:23 | |
| BobBall | I'm not doing that in my CI so I wanted to set it up to do it :) | 12:23 |
| BobBall | thanks - I'll look there | 12:23 |
| sdague | basically you need to not define DEFAULT_INSTANCE_TYPE | 12:23 |
| BobBall | ahhhhh perfect!! | 12:24 |
| BobBall | indeed we are specifying it | 12:24 |
| BobBall | I knew someone would have fixed that one already... ;) | 12:25 |
| *** nprivalova has joined #openstack-infra | 12:28 | |
| *** sarob has quit IRC | 12:28 | |
| *** CaptTofu has quit IRC | 12:30 | |
| *** CaptTofu has joined #openstack-infra | 12:31 | |
| *** salv-orlando_ has joined #openstack-infra | 12:33 | |
| *** salv-orlando has quit IRC | 12:36 | |
| *** salv-orlando_ is now known as salv-orlando | 12:36 | |
| vkozhukalov | If I want to create project specific jobs (aka modules/openstack_project/files/jenkins_job_builder/config/project_name.yaml) where do I need to include this file to allow Jenkins Job Builder to know about this particular yaml file? | 12:39 |
| openstackgerrit | A change was merged to openstack/requirements: Allow Django 1.5 and 1.6 https://review.openstack.org/59920 | 12:39 |
| openstackgerrit | A change was merged to openstack/requirements: Bump python-savannaclient version to >= 0.4.0 https://review.openstack.org/59657 | 12:39 |
| vkozhukalov | or it includes all yaml files which present in that directory? | 12:40 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 12:42 | |
| *** _SergeyLukjanov has quit IRC | 12:42 | |
| *** dolphm has joined #openstack-infra | 12:46 | |
| anteaya | vkozhukalov: hi there | 12:47 |
| anteaya | can I ask you to expand with an example? | 12:47 |
| anteaya | what job are you trying to create? | 12:48 |
| *** dstanek has joined #openstack-infra | 12:48 | |
| *** sandywalsh_ has quit IRC | 12:49 | |
| *** smarcet has joined #openstack-infra | 12:50 | |
| openstackgerrit | Yuriy Taraday proposed a change to openstack-infra/pypi-mirror: Refactor if: nesting in build_mirror https://review.openstack.org/59961 | 12:50 |
| *** jcoufal has quit IRC | 12:52 | |
| anteaya | vkozhukalov: here is the layout.yaml file that calls the individual jenkins jobs: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/zuul/layout.yaml | 12:53 |
| anteaya | is that what you were looking for? | 12:53 |
| vkozhukalov | I am trying to create functional testing job for ironic. I am going to make one VM install ironic on it then make another VM, then boot second VM via PXE from the first one. Ok. Actully it does not matter how it is supposed to work. I just try to understand how to integrate that test into -infra. I can see project specific job in tripleo.yaml and can not understand how jenkins job builder is supposed to know about this file. | 12:54 |
| anteaya | okay | 12:56 |
| anteaya | let me see if I can help | 12:56 |
| *** sarob has joined #openstack-infra | 12:56 | |
| *** jcoufal has joined #openstack-infra | 12:57 | |
| anteaya | vkozhukalov: openstack-infra/config/modules/jenkins/files/slave_scripts/<script>.sh is picked up by openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/micros.yaml which is picked up by openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml which talks to | 13:00 |
| anteaya | openstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/python-jobs.yaml | 13:00 |
| anteaya | which is used by oenstack-infra/config/modules/openstack_project/files/jenkins_job_builder/config/projects.yaml | 13:01 |
| anteaya | a bit of a trail | 13:01 |
| anteaya | take some time with that | 13:01 |
| anteaya | and see if you have the path you need to do your work | 13:01 |
| *** matsuhashi has quit IRC | 13:03 | |
| *** sandywalsh_ has joined #openstack-infra | 13:03 | |
| vkozhukalov | anteaya, thanks. will go into it. | 13:03 |
| *** dprince has joined #openstack-infra | 13:03 | |
| *** SergeyLukjanov has joined #openstack-infra | 13:05 | |
| *** nosnos_ has quit IRC | 13:07 | |
| *** nosnos has joined #openstack-infra | 13:08 | |
| *** yamahata_ has joined #openstack-infra | 13:09 | |
| *** paul-- has quit IRC | 13:09 | |
| *** nprivalova has quit IRC | 13:11 | |
| *** paul-- has joined #openstack-infra | 13:14 | |
| jog0 | clarkb: I think ' Add HPCloud region-a to nodepool. | 13:15 |
| jog0 | Change-Id: I809303c5dd2f25f2b4dad638568fe4b8b1059818 | 13:15 |
| jog0 | may be breaking the large-ops tests | 13:15 |
| *** ruhe has joined #openstack-infra | 13:15 | |
| *** CaptTofu has quit IRC | 13:15 | |
| jog0 | this would be true if region-a is slower what we had a week ago | 13:15 |
| *** CaptTofu has joined #openstack-infra | 13:16 | |
| *** dims has quit IRC | 13:17 | |
| *** dims has joined #openstack-infra | 13:17 | |
| *** sandywalsh_ has quit IRC | 13:20 | |
| *** thomasem has joined #openstack-infra | 13:23 | |
| *** DinaBelova has quit IRC | 13:24 | |
| anteaya | vkozhukalov: k, hope it helps | 13:28 |
| *** sarob has quit IRC | 13:29 | |
| *** xchu has joined #openstack-infra | 13:29 | |
| *** yaguang has joined #openstack-infra | 13:30 | |
| *** fbo_away is now known as fbo | 13:31 | |
| *** nprivalova has joined #openstack-infra | 13:31 | |
| ArxCruz | jeblair: around ? | 13:32 |
| *** matsuhashi has joined #openstack-infra | 13:32 | |
| *** paul-- has quit IRC | 13:32 | |
| sdague | ArxCruz: it's 5:30am his time :) | 13:32 |
| sdague | so expect him up in a few more hours | 13:32 |
| ArxCruz | sdague: oh... | 13:33 |
| ArxCruz | sdague: https://review.openstack.org/#/c/55829/ | 13:33 |
| ArxCruz | would you like me to create new patches with dependencies of this? | 13:33 |
| *** herndon_ has joined #openstack-infra | 13:34 | |
| sdague | yes please | 13:34 |
| ArxCruz | ok | 13:34 |
| *** DinaBelova has joined #openstack-infra | 13:35 | |
| *** mriedem has joined #openstack-infra | 13:35 | |
| *** sandywalsh_ has joined #openstack-infra | 13:36 | |
| *** markmcclain has quit IRC | 13:38 | |
| *** paul-- has joined #openstack-infra | 13:39 | |
| *** nosnos has quit IRC | 13:39 | |
| *** CaptTofu has quit IRC | 13:39 | |
| *** ruhe is now known as ruhe_ | 13:41 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Adding an option to use qpid instead of rabbit or zeromq https://review.openstack.org/55829 | 13:41 |
| *** nprivalova has quit IRC | 13:43 | |
| *** ruhe_ is now known as ruhe | 13:44 | |
| *** jorisroovers has quit IRC | 13:48 | |
| *** jorisroovers has joined #openstack-infra | 13:48 | |
| *** jorisroovers has quit IRC | 13:50 | |
| *** jorisroovers has joined #openstack-infra | 13:52 | |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 13:55 |
| sdague | jeblair: it would be nice to coordinate moving the elastic_recheck console tree today | 13:55 |
| sdague | as there are a stack of fixes in there which would be handy to land | 13:55 |
| sdague | and not dual maintain | 13:56 |
| *** dkranz has joined #openstack-infra | 13:56 | |
| *** sarob has joined #openstack-infra | 13:56 | |
| *** markmcclain has joined #openstack-infra | 14:02 | |
| *** nprivalova has joined #openstack-infra | 14:02 | |
| *** paul-- has quit IRC | 14:05 | |
| *** julim has joined #openstack-infra | 14:05 | |
| *** bpokorny1 has joined #openstack-infra | 14:06 | |
| *** mfer has joined #openstack-infra | 14:06 | |
| *** yamahata_ has quit IRC | 14:07 | |
| *** paul-- has joined #openstack-infra | 14:07 | |
| *** yamahata_ has joined #openstack-infra | 14:07 | |
| openstackgerrit | Roman Prykhodchenko proposed a change to openstack-infra/config: Adds devstack-gate tests for Ironic https://review.openstack.org/53917 | 14:11 |
| *** heyongli has joined #openstack-infra | 14:11 | |
| *** markmcclain has quit IRC | 14:12 | |
| *** rongze has quit IRC | 14:14 | |
| *** mfer has quit IRC | 14:17 | |
| fungi | jog0: he and i were discussing it last night... the new hp regions are definitely slower, on par with rackspace's. the current suggestion is to switch them to check-only like we do with devstack nodes on rackspace | 14:17 |
| jog0 | fungi: ahh yeah | 14:17 |
| jog0 | that makes sense | 14:17 |
| *** paul-- has quit IRC | 14:18 | |
| *** mfer has joined #openstack-infra | 14:18 | |
| jog0 | that is causing at least two bugs | 14:18 |
| jog0 | (large-ops) | 14:18 |
| jog0 | and *maybe* the py27 nova issues | 14:18 |
| *** rongze has joined #openstack-infra | 14:18 | |
| *** dstanek has quit IRC | 14:19 | |
| fungi | oh, too fun | 14:19 |
| fungi | jog0: they wouldn't cause problems with nova unit testing, since they're only being used for integration jobs | 14:20 |
| jog0 | fungi: I just figured that out as well, was hoping to kill 3 birds with one stone I will have to settle for 2 | 14:21 |
| jog0 | anyway I have to go AFK For a bit | 14:21 |
| *** loq_mac has joined #openstack-infra | 14:23 | |
| *** loquacities has joined #openstack-infra | 14:29 | |
| *** nprivalova has quit IRC | 14:29 | |
| *** loq_mac has quit IRC | 14:30 | |
| *** sarob has quit IRC | 14:30 | |
| *** weshay has joined #openstack-infra | 14:31 | |
| *** loquacities has quit IRC | 14:35 | |
| *** dkliban has joined #openstack-infra | 14:42 | |
| *** guitarza1 is now known as guitarzan | 14:43 | |
| *** dkliban_ has joined #openstack-infra | 14:43 | |
| jog0 | fungi: for the large-ops issues, I am thinking we should drop the number of vms down to 100 (so a drop of 33%) | 14:45 |
| jog0 | that should help the issue | 14:45 |
| fungi | oh, so it's 150 right now? | 14:46 |
| fungi | you were originally able to ferret out interesting scale issues at 50 anyway, right? | 14:46 |
| jog0 | in part because that test originally failed for 50 instances and to fix things further in nova network is significant work (batching network commands etc) | 14:46 |
| fungi | so 100 still seems like a good safety margin to prevent those from returning | 14:46 |
| *** dkliban has quit IRC | 14:47 | |
| jog0 | fungi: yup it was root wrap was slow | 14:47 |
| jog0 | and we called it too much | 14:47 |
| jog0 | I'll file a bug with neutron and nova and then drop the value to 100 | 14:47 |
| ruhe | http://logs.openstack.org/59/59759/1/check/gate-tempest-docs/fe1a207/console.html see error msg in the bottom. does it mean that partition ran out of space? | 14:48 |
| mordred | fungi: we don't have any graphs or collected numbers do we? | 14:48 |
| *** pabelanger_ has joined #openstack-infra | 14:48 | |
| fungi | mordred: for the new region performance? i think we may have job run times in graphite but it would take some work to isolate and compare them against the old regions | 14:49 |
| *** pabelanger_ has quit IRC | 14:49 | |
| *** pabelanger_ has joined #openstack-infra | 14:49 | |
| *** pabelanger has quit IRC | 14:50 | |
| mordred | fungi: bleh. I hate work | 14:50 |
| *** pabelanger_ is now known as pabelanger | 14:50 | |
| *** pabelanger_ has joined #openstack-infra | 14:50 | |
| vkozhukalov | anteaya, ok. again about ironic functional testing. As far as i need to launch a couple of VMs and boot one of them via PXE, I found out how it is done in tripleo. They have shell code to deal with emulated bare metal (bm_poseurs), but this code in tripleo-incubator. There is the -infra/tripleo-ci project which uses code from tripleo-incubator to deal with bm_poseurs. It is supposed that in the future tripleo-ci will get emulated bare meta | 14:50 |
| vkozhukalov | l environments from https://etherpad.openstack.org/p/tripleo-test-cluster via API. So the question is should I create -infra/ironic-ci project and do the things as in tripleo-ci and should I use code from tripleo-incubator to launch bm_poseurs or I need to put the bm_poseurs code into ironic? | 14:50 |
| *** pabelanger has quit IRC | 14:51 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/devstack-gate: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER https://review.openstack.org/59993 | 14:52 |
| *** pabelanger has joined #openstack-infra | 14:53 | |
| fungi | vkozhukalov: i think those are questions you want to direct to #tripleo for the moment. the bare metal test cluster is not in production for upstream ci, and i'm unclear on their anticipated timeline | 14:53 |
| *** riteshnanda has joined #openstack-infra | 14:53 | |
| fungi | vkozhukalov: what we do have available for now is the ability to start up a vm running devstack on it, and then have it manage small (generally cirros) qemu paravirt guests | 14:54 |
| fungi | vkozhukalov: so if your testing can be limited to launching a couple of tiny cirros vms under qemu on a clound vm running devstack and have one of those (or the devstack context itself) provide the necessary pxe boot configuration for the cirros virtual machines, that might be something we could accommodate with the current implementation | 14:56 |
| *** sarob has joined #openstack-infra | 14:56 | |
| jog0 | fungi: we have some interesting findings with py27 failures inthe tripleo room | 14:57 |
| fungi | oho | 14:57 |
| * fungi /joins | 14:57 | |
| jog0 | it looks like its a performance issue, TL;DR there is a 160 second timeout on all nova tests and the migration tests are alwas slow | 14:58 |
| *** DinaBelova has quit IRC | 14:58 | |
| *** DennyZhang has joined #openstack-infra | 15:01 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER in jjb https://review.openstack.org/59995 | 15:01 |
| *** jamesmcarthur has joined #openstack-infra | 15:02 | |
| openstackgerrit | Arx Cruz proposed a change to openstack-infra/config: Replace DEVSTACK_GATE_ZEROMQ for DEVSTACK_GATE_MQ_DRIVER in jjb https://review.openstack.org/59995 | 15:02 |
| *** reaper has joined #openstack-infra | 15:03 | |
| vkozhukalov | fungi, I've already asked them. I know that bare metal test cluster is not ready, and I can launch bm_poseurs via virsh CLI. It is ok. I'am wondering if it is possible to start two VMs and boot the first VM from the PXE configured on the second one. Or I need to run VM and then run another qemu VM inside VM? | 15:04 |
| *** herndon_ has quit IRC | 15:05 | |
| vkozhukalov | fungi, I mean if it is possible to launch two VMs on jenkins bare metal node. | 15:05 |
| fungi | vkozhukalov: yes, right now we have a design limitation of one cloud vm per jenkins job. so inside that vm you can create more nested virtual machines, but those have to be paravirtualized because none of the donated cloud resources we use provide hardware support for nesting | 15:05 |
| vkozhukalov | fungi, ok I see. | 15:06 |
| fungi | vkozhukalov: i think the tripleo plan is to have a jenkins job correspond to a control node in the bare metal cluster instead, and it would then have the ability to launch virtual (or even physical) machines as needed for that particular job within an isolated tenant environment | 15:06 |
| fungi | but they can probably explain the details far better than i | 15:07 |
| *** thedodd has joined #openstack-infra | 15:07 | |
| *** markwash has joined #openstack-infra | 15:09 | |
| *** oubiwann_ has joined #openstack-infra | 15:10 | |
| *** markmcclain has joined #openstack-infra | 15:11 | |
| *** yaguang has quit IRC | 15:12 | |
| *** dstanek has joined #openstack-infra | 15:16 | |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 15:16 | |
| *** _SergeyLukjanov has quit IRC | 15:16 | |
| *** riteshnanda has quit IRC | 15:16 | |
| *** ywu has quit IRC | 15:16 | |
| *** denis_makogon has joined #openstack-infra | 15:19 | |
| *** heyongli has quit IRC | 15:19 | |
| anteaya | we are sitting here workong on neutron-grenade testing, EmilienM has a patch up here: https://review.openstack.org/#/c/59039/ that hopefully addresses jeblair's latest comments, does anyone have time for a review? | 15:20 |
| anteaya | currently that patch is a block for our work | 15:20 |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Drop TEMPEST_LARGE_OPS_NUMBER by 33% to 100 https://review.openstack.org/60000 | 15:22 |
| jog0 | fungi: ^ theres the patch | 15:22 |
| jog0 | ohh and I got a nice round number for my gerrit number too | 15:22 |
| pleia2 | woo 60k | 15:23 |
| *** matsuhashi has quit IRC | 15:23 | |
| * jog0 wonders why gerrit doesn't use hex numbers | 15:23 | |
| fungi | anteaya: technically i don't think it addresses jeblair's comment on the previous patchset, but it does seem to reflect the irc discussion EmilienM had with clarkb after that | 15:25 |
| *** riteshnanda has joined #openstack-infra | 15:26 | |
| anteaya | fungi: oh okay great | 15:27 |
| *** xchu has quit IRC | 15:27 | |
| fungi | sdague: dtroyer_zz: any idea whether adding (voting) grenade-neutron jobs to grenade and not to neutron is the right path forward there? i would have expected it to be applied to neutron only or both neutron and grenade, but not just grenade... | 15:27 |
| *** esker has joined #openstack-infra | 15:28 | |
| anteaya | fungi: EmilienM wants to know if he also needs to address periodic jobs due to this change | 15:28 |
| fungi | sdague: dtroyer_zz: https://review.openstack.org/59039 | 15:28 |
| *** sarob has quit IRC | 15:28 | |
| fungi | anteaya: i think clarkb said yesterday that he was taking a wait-and-see approach on grenade jobs before going into a refactoring similar to what he did with the devstack/tempest jobs | 15:29 |
| *** senk has joined #openstack-infra | 15:29 | |
| openstackgerrit | Joe Gordon proposed a change to openstack-infra/devstack-gate: Drop TEMPEST_LARGE_OPS_NUMBER by 33% to 100 https://review.openstack.org/60000 | 15:29 |
| anteaya | fungi: so we are waiting to hear back from clarkb? | 15:29 |
| sdague | fungi: actually, it should be both | 15:29 |
| fungi | sdague: thanks. i thought so. i'll -1 it with pointers | 15:30 |
| sdague | also, probably worth starting it as non-voting just to get the result stream | 15:30 |
| *** jcooley_ has joined #openstack-infra | 15:30 | |
| fungi | you read my mind | 15:30 |
| *** oubiwann_ has quit IRC | 15:31 | |
| *** rfolco has quit IRC | 15:32 | |
| *** dcramer_ has joined #openstack-infra | 15:33 | |
| *** wenlock has joined #openstack-infra | 15:34 | |
| *** datsun180b has joined #openstack-infra | 15:34 | |
| mordred | jog0: nice! | 15:34 |
| *** nprivalova has joined #openstack-infra | 15:34 | |
| jog0 | fungi: here is the bug for nova and neutron https://bugs.launchpad.net/neutron/+bug/1257788 | 15:35 |
| uvirtbot | Launchpad bug 1257788 in nova "nova-network doesn't handle large amount of concurrent booting instances well" [Low,New] | 15:35 |
| *** markwash has quit IRC | 15:35 | |
| *** markwash has joined #openstack-infra | 15:38 | |
| *** DennyZhang has quit IRC | 15:40 | |
| *** ruhe is now known as ruhe_ | 15:41 | |
| *** ruhe_ is now known as ruhe | 15:42 | |
| *** SergeyLukjanov has joined #openstack-infra | 15:44 | |
| openstackgerrit | Chuck Short proposed a change to openstack-dev/pbr: python3: Fix python3.2 installation https://review.openstack.org/60006 | 15:44 |
| ttx | mordred: hey, was wondering if I needed to do anything for oso.rootwrap to start showing up on PyPI once tagged | 15:46 |
| *** rongze has quit IRC | 15:46 | |
| ttx | mordred: i'm about to push a 1.0.0 tag there | 15:46 |
| fungi | ttx: i can check | 15:46 |
| ttx | oslo.rootwrap* | 15:47 |
| ttx | fungi: thx! | 15:47 |
| fungi | ttx: https://pypi.python.org/pypi/oslo.rootwrap returns "Not Found ()" so the project presumably needs to be registered | 15:47 |
| ttx | fungi: yeah, was wonderig if that happened automagically | 15:47 |
| ttx | fungi: let me know if I need to do anything, but i suspect not ? | 15:48 |
| jeblair | i'll do it real quick | 15:48 |
| mordred | it does not yet happen automagically | 15:48 |
| mordred | oh! look. it's jeblair | 15:48 |
| * mordred really wants to add that to manage-projects | 15:48 | |
| ttx | oh! | 15:48 |
| ttx | mordred: I'm now expecting automation by default | 15:49 |
| ttx | my mpcut.sh script that does MP branch creation, waits for the tarball build and tags all bugs fixreleased sort of works, btw | 15:49 |
| jeblair | done | 15:49 |
| ttx | jeblair: so I should be able to tag 1.0.0 and magic will happen, right ? | 15:50 |
| *** elasticio has joined #openstack-infra | 15:50 | |
| zul | is there a newer oslo.messaging on pypi? | 15:50 |
| jeblair | ttx: for future reference, i create the project manually using the web form, only filling out name and version (0.0), then add 'openstackci' as an owner | 15:50 |
| fungi | permissions seem to be in place, i'll check to make sure we're getting properly-versioned tarballs on prereleases | 15:50 |
| ttx | jeblair: ack | 15:50 |
| jeblair | ttx: yes, a tag should start the machinery now | 15:51 |
| *** syerrapragada1 has joined #openstack-infra | 15:51 | |
| fungi | well, we seem to be getting master branch tarballs at least, so your sdist is exercised | 15:51 |
| *** nprivalova has quit IRC | 15:51 | |
| fungi | i guess this will be the first ever tag, so no prerelease tarballs to check there | 15:51 |
| jog0 | so Django==1.6 just wedged the gate https://jenkins02.openstack.org/job/check-tempest-dsvm-full/buildTimeTrend | 15:52 |
| ttx | fungi, jeblair: tag on its way | 15:52 |
| fungi | ttx: looks like it's in teh status page now | 15:52 |
| fungi | tarballs.o.o looks right | 15:53 |
| *** vkozhukalov has quit IRC | 15:53 | |
| fungi | pypi looks right for the new release now | 15:53 |
| fungi | mirror rebuilds are happening, so you should be all set | 15:53 |
| *** dolphm has quit IRC | 15:54 | |
| *** syerrapragada has quit IRC | 15:54 | |
| *** rcleere has joined #openstack-infra | 15:54 | |
| *** elasticio has quit IRC | 15:54 | |
| *** elasticio has joined #openstack-infra | 15:54 | |
| *** oubiwann_ has joined #openstack-infra | 15:56 | |
| *** sarob has joined #openstack-infra | 15:56 | |
| *** tma996 has quit IRC | 15:57 | |
| anteaya | fungi: awesome, thanks for the comments on 59039 | 15:57 |
| *** jroovers has joined #openstack-infra | 15:58 | |
| *** tma996 has joined #openstack-infra | 15:58 | |
| *** jorisroovers has quit IRC | 15:58 | |
| *** elasticio has quit IRC | 15:58 | |
| *** elasticio has joined #openstack-infra | 15:59 | |
| *** tma996 has quit IRC | 15:59 | |
| jeblair | mordred: would like your help... | 16:00 |
| jeblair | mordred: i want to figure out why https://review.openstack.org/#/c/59920/1 passed | 16:00 |
| *** tma996 has joined #openstack-infra | 16:00 | |
| jeblair | mordred: fungi is working on reverting that change, because in fact after it landed and the mirrors rebuilt, things started failing | 16:01 |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Create and upload wheels https://review.openstack.org/56760 | 16:01 |
| *** senk has quit IRC | 16:01 | |
| *** tma996 has quit IRC | 16:01 | |
| *** senk has joined #openstack-infra | 16:01 | |
| mordred | jeblair: ok. looking | 16:02 |
| *** rongze has joined #openstack-infra | 16:02 | |
| *** tma996 has joined #openstack-infra | 16:02 | |
| *** jroovers has quit IRC | 16:02 | |
| *** Abhishek_ has joined #openstack-infra | 16:02 | |
| mordred | jeblair: it's the situation we were talking about yesterday | 16:03 |
| mordred | except a version of it we didn't consider | 16:03 |
| mordred | I think | 16:03 |
| mordred | hrm. unless we _used_ the mirror when we werent' supposed to | 16:03 |
| jeblair | mordred: it passed check, but shouldn't have | 16:04 |
| mordred | yeah. with you | 16:04 |
| *** tma996 has quit IRC | 16:04 | |
| jeblair | mordred: + '[' 0 -eq 0 -o openstack/requirements '!=' openstack/requirements ']' | 16:04 |
| *** tma996 has joined #openstack-infra | 16:04 | |
| jeblair | http://logs.openstack.org/20/59920/1/gate/gate-tempest-dsvm-full/339bdcb/logs/devstack-gate-setup-host.txt.gz | 16:04 |
| *** tma996 has quit IRC | 16:04 | |
| jeblair | mordred: so that first clause is "$DEVSTACK_GATE_REQS_INTEGRATION" -eq "0" | 16:05 |
| *** Abhishe__ has joined #openstack-infra | 16:05 | |
| *** tma996 has joined #openstack-infra | 16:05 | |
| jeblair | mordred: i think that always means that we use the mirror unless we're doing the requirements integration test | 16:06 |
| mordred | jeblair: I believe you are correct. which, it what fungi was saying as a feedback to the change to using select-mirror | 16:06 |
| mordred | and then I countered that that's not what we want | 16:06 |
| mordred | which we eventually came around to agreeing on | 16:07 |
| jeblair | mordred: so perhaps the logic is flawed, because what we really want is: use the mirror unless we're doing the requirements integration test _or_ this is a change to the requirements project | 16:07 |
| *** dstanek has quit IRC | 16:07 | |
| mordred | yes | 16:07 |
| mordred | I agree with your recent statement | 16:07 |
| *** dstanek has joined #openstack-infra | 16:07 | |
| jeblair | mordred: ok, what's the state of that change? | 16:07 |
| *** Abhishek_ has quit IRC | 16:08 | |
| mordred | jeblair: I need to fix https://review.openstack.org/#/c/58153/ then it can be re-reviewed | 16:08 |
| jeblair | mordred: https://review.openstack.org/#/c/48064/6/devstack-vm-gate-wrap.sh looks like it has the same conditional | 16:09 |
| anteaya | once the mirrors have been repopulated with django 1.5.5 again, can a core pull the lever on this add ironic puppet module to stackforge? 7 upvotes and a happy jenkins: https://review.openstack.org/#/c/59247/ | 16:09 |
| anteaya | fyi, I'm sitting in the eNovance office :D | 16:09 |
| mordred | jeblair: yah - but once the other two changes are in, the conditional is largely irrelevant, because the selection will key off of openstack/reuqirements not being in the projects.txt file | 16:10 |
| ttx | anteaya: most people would assume Montreal at that point. | 16:10 |
| *** tma996 has quit IRC | 16:10 | |
| anteaya | ttx good point | 16:10 |
| anteaya | Paris then for clarity | 16:10 |
| *** tma996 has joined #openstack-infra | 16:10 | |
| *** tma996 has quit IRC | 16:11 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack/requirements: Revert "Allow Django 1.5 and 1.6" https://review.openstack.org/60014 | 16:11 |
| *** tma996 has joined #openstack-infra | 16:11 | |
| *** Abhishe__ has quit IRC | 16:11 | |
| *** avishayb has quit IRC | 16:11 | |
| *** nprivalova has joined #openstack-infra | 16:11 | |
| fungi | markmcclain: thanks | 16:12 |
| jeblair | aprvd | 16:12 |
| *** osanchez has joined #openstack-infra | 16:12 | |
| *** tma996 has quit IRC | 16:12 | |
| *** tma996 has joined #openstack-infra | 16:13 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/config: Use requirements project list for mirror selection https://review.openstack.org/58153 | 16:14 |
| mordred | jeblair: ok. there's ^^ the pre-req | 16:15 |
| jeblair | mordred: okay, i agree that the patch series should fix this (i also agree with fungi that removing the '-o "$ZUUL_PROJECT" != "openstack/requirements"' bit would be nice) | 16:15 |
| *** jcoufal has quit IRC | 16:16 | |
| mordred | jeblair: shoudl I just go ahead and do that in the final patch? | 16:16 |
| jeblair | mordred: yeah, i think so. it's so confusing to have that there but not mean anything. | 16:17 |
| *** tma996 has quit IRC | 16:17 | |
| fungi | i believe the original logic there was that we should use upstream pypi for all requirements-integration jobs and also for all changes to the openstack/requirements project | 16:17 |
| jeblair | fungi: yeah, but the boolean logic is wrong, so it's just wrong currently, and a no-op in the new system. but it _looks_ really important. :) | 16:18 |
| mordred | jeblair: do we even need DEVSTACK_GATE_REQS_INTEGRATION anymore? | 16:18 |
| mordred | jeblair: if we havea genearlized trigger that means that all changes to ... oh, wait, we need to block it for that for pbr chagnes too | 16:18 |
| mordred | yeah. we nede it | 16:18 |
| *** markmcclain has quit IRC | 16:19 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-infra/devstack-gate: Use select-mirror in devstack-gate https://review.openstack.org/48064 | 16:19 |
| mordred | okie. there it is updated | 16:19 |
| jeblair | mordred: i don't think we need it | 16:19 |
| jeblair | mordred: grepping in infra/config says it's only used in the {pipeline}-requirements-integration-dsvm job, which is not in zuul's layout | 16:20 |
| mordred | jeblair: we do- we also run requirements integration on changes to pypi-mirror and pbr | 16:20 |
| mordred | oh. hrm. ok | 16:20 |
| mordred | what's the job we use in pbr, pypi-mirror and requirements to do that checking? | 16:20 |
| mordred | shouldn't that have the DEVSTACK_GATE_REQS_INTEGRATION override? | 16:21 |
| *** sarob has quit IRC | 16:21 | |
| jeblair | mordred: oh, wait, it is in layout.yaml | 16:21 |
| fungi | i wonder if it got lost in the d-g refactor | 16:21 |
| jeblair | sorry. i was wrong, it's still used and required. | 16:21 |
| jeblair | and i learned something about git grep. | 16:22 |
| *** oubiwann_ has quit IRC | 16:22 | |
| jeblair | mordred: so i just checked a random devstack-precise slave, and there is no /opt/requirements repo | 16:22 |
| fungi | jeblair: did you learn that it takes your cwd into account for tree recursion? | 16:22 |
| mikal | Is there a way to inject jobs into zuul without talking to it via gerrit? | 16:23 |
| mordred | jeblair: that's right - we haven't landed that patch yet | 16:23 |
| jeblair | fungi: no i learned it (understandably) doesn't follow symlinks. :) | 16:23 |
| mordred | jeblair: https://review.openstack.org/#/c/59191/ | 16:23 |
| *** danger_fo_away is now known as dangers | 16:23 | |
| fungi | oh, ha | 16:23 |
| mikal | i.e. it would be nice to be able to force a run of a pipeline without having to add gerrit comments | 16:23 |
| jeblair | mikal: https://review.openstack.org/#/c/58841/ | 16:23 |
| jeblair | mikal: that's admin-only though | 16:23 |
| mikal | jeblair: OMG< that's exactly what I want | 16:24 |
| mikal | jeblair: I'm an admin for turbo hipster though, so that would work for me | 16:24 |
| *** oubiwann_ has joined #openstack-infra | 16:24 | |
| fungi | mikal: the more recent trigger reworking in zuul is intended to make it possible to have other trigger types besides gerrit though, eventually | 16:24 |
| fungi | oh, and aside form periodic, which was what got added when that happened | 16:24 |
| jeblair | mordred: aprvd | 16:25 |
| mordred | jeblair: awesome | 16:25 |
| *** nprivalova has quit IRC | 16:25 | |
| fungi | i think i hadn't approved it because it depends on another change | 16:26 |
| mordred | it does | 16:26 |
| mordred | https://review.openstack.org/#/c/41268/3 | 16:26 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Fix pep8 tests https://review.openstack.org/60017 | 16:27 |
| jeblair | mordred: is that ready to land? | 16:27 |
| fungi | clarkb: had a list of the wheels-in-mirror changes with the order you wanted them merged, so was trying to follow that sequence though we may have lost track of the intended order for a couple since | 16:27 |
| mordred | jeblair: yes. it should be perfectly fine to land that. it will add the wheel mirrors to the list of things searched | 16:27 |
| *** ryanpetrello has joined #openstack-infra | 16:27 | |
| mordred | if a run does not have wheels enabled, that will be a no-op url scan | 16:27 |
| jeblair | fungi: are we okay on the lsb command thing? | 16:28 |
| jeblair | fungi: more words: do all our current slave platforms have the lsb_release command? | 16:28 |
| mordred | jeblair: we could rebase them off of each other though - they are not actually dependent - it was just a dependent workstream for me | 16:28 |
| fungi | jeblair: yeah, looks like it. the fedora-devstack people can just add the necessary additional package to the rpms list to make that possible i think | 16:28 |
| *** jamesmcarthur has quit IRC | 16:28 | |
| ryanpetrello | any chance I can convince somebody to take a look at https://review.openstack.org/#/c/59818/1 ?= | 16:28 |
| ryanpetrello | pretty straightforward review | 16:28 |
| ryanpetrello | just hoping to get the ceilometer tests passing so I can make a pecan release :) | 16:29 |
| ryanpetrello | whoa, that was fast | 16:29 |
| ryanpetrello | oh, wait, wrong email | 16:29 |
| ryanpetrello | :) | 16:29 |
| mordred | but I'd love to make having lsb_release a pre-req for our stuff working, since, you know, LSB | 16:29 |
| *** jamesmcarthur has joined #openstack-infra | 16:30 | |
| *** reaper has quit IRC | 16:30 | |
| fungi | ryanpetrello: out of curiosity, why does that job need to run on centos6 instead of ubuntu lts? | 16:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Add support for per-distro wheel mirrors https://review.openstack.org/41268 | 16:30 |
| jeblair | mordred: what's up with rax performance nodes? | 16:30 |
| openstackgerrit | A change was merged to openstack-infra/config: Install requirements repo on slaves https://review.openstack.org/59191 | 16:30 |
| ryanpetrello | fungi: it's running ceilometer tests that interact w/ mongodb | 16:30 |
| fungi | ryanpetrello: ahh, right! that old saw | 16:31 |
| *** ^d has joined #openstack-infra | 16:31 | |
| *** ^d has joined #openstack-infra | 16:31 | |
| *** markwash_ has joined #openstack-infra | 16:31 | |
| fungi | one of the few packages which centos has a newer version of than ubuntu | 16:31 |
| jeblair | never thought i'd see the day | 16:31 |
| *** markwash has quit IRC | 16:32 | |
| *** markwash_ is now known as markwash | 16:32 | |
| mordred | jeblair: uhm. I thought I had a patch up for that | 16:32 |
| mordred | jeblair: but it might be simply that I did the nodepool patch to allow that patch to be made | 16:33 |
| BobBall | Can I make tox abort on the first failure? | 16:33 |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge https://review.openstack.org/59932 | 16:33 |
| mordred | jeblair: which I think is landed now | 16:33 |
| mordred | jeblair: so now we just need to add some nodepool entries with some flavor name filters | 16:34 |
| ryanpetrello | fungi: thanks :) | 16:34 |
| fungi | ryanpetrello: if you want to see newer mongodb on ubuntu precise, bug 1228977 could use some attention | 16:35 |
| uvirtbot | Launchpad bug 1228977 in nova "n-cpu seems to crash when running with libvirt 1.0.6 from ubuntu cloud archive" [High,Confirmed] https://launchpad.net/bugs/1228977 | 16:35 |
| *** UtahDave has joined #openstack-infra | 16:35 | |
| openstackgerrit | A change was merged to openstack-infra/config: Run pecan-ceilometer gate tests on centos6 https://review.openstack.org/59818 | 16:35 |
| *** senk has quit IRC | 16:38 | |
| mikal | jeblair: for that client, what would a valid trigger look like? Is it just the trigger name? | 16:38 |
| mikal | i.e "zuul -v -c /etc/zuul/zuul.conf enqueue --trigger gerrit --pipeline check --project openstack/nova --change 59856 --patchset 1" doesn't seem to work for me. | 16:39 |
| *** CaptTofu has joined #openstack-infra | 16:39 | |
| mikal | jeblair: running that ends with the client saying "Waiting for job completion", but not entries in debug.log or zuul.log relating to a job being created | 16:40 |
| russellb | anyone want to play "spot the syntax error" with my hook function? This fails on the line defining the function: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/solum.yaml ... This one works: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/files/jenkins_job_builder/config/requirements.yaml | 16:40 |
| *** jgrimm has joined #openstack-infra | 16:41 | |
| jeblair | mikal: is the running server from that patchset too? (it adds both the client and necessary server support) | 16:41 |
| mikal | jeblair: ahh, no. I didn't restart zuul. | 16:42 |
| *** adam_g has joined #openstack-infra | 16:42 | |
| *** adam_g has quit IRC | 16:42 | |
| *** adam_g has joined #openstack-infra | 16:42 | |
| * mikal kicks self | 16:42 | |
| *** jcooley_ has quit IRC | 16:43 | |
| openstackgerrit | QianLin proposed a change to openstack-infra/nodepool: Fix the issue which raise FloatingIpPoolNotFound https://review.openstack.org/54331 | 16:43 |
| *** sHellUx has joined #openstack-infra | 16:44 | |
| *** Ryan_Lane has joined #openstack-infra | 16:46 | |
| *** UtahDave has quit IRC | 16:47 | |
| mikal | jeblair: the other gotcha was the new zuul_url config parameter | 16:47 |
| jeblair | mikal: yes, i'm a bad upstream and make that a breaking change. i need to add that to NEWS. | 16:48 |
| jeblair | mikal: good news is we're about to cut 2.0 and i'll try to stop doing that. :) | 16:48 |
| *** michchap_ has quit IRC | 16:48 | |
| *** dstanek has quit IRC | 16:49 | |
| mikal | jeblair: oh, I don't care. I'd much rather you move quickly | 16:49 |
| *** marun has quit IRC | 16:50 | |
| jeblair | mikal: usually it's not too much of a burden to be backwards compat and move quickly (especially since we CD it) | 16:50 |
| mikal | jeblair: is it expected that a zuul server restart would require a restart of all the workers? Because that's what happened to me... | 16:52 |
| zaro | good morning | 16:53 |
| jeblair | mikal: no, they should automatically reconnect. but we don't use turbo hipster :( | 16:53 |
| *** ArxCruz has quit IRC | 16:53 | |
| *** ArxCruz has joined #openstack-infra | 16:54 | |
| mikal | jeblair: huh, this might be a turbo hipster bug then | 16:54 |
| mikal | jeblair: I will hassle jhesketh about it | 16:54 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Remove braces from bash vars in solum job config https://review.openstack.org/60027 | 16:55 |
| *** pcrews has joined #openstack-infra | 16:55 | |
| *** sarob has joined #openstack-infra | 16:56 | |
| fungi | russellb: you might have also been able to fix that by doubling the ${{braces}} to escape them from the yaml parser | 16:57 |
| * fungi can't remember when that is and isn't necessary | 16:57 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/zuul: Document zuul_url breaking change https://review.openstack.org/60028 | 16:57 |
| russellb | fungi: ah, makes sense | 16:58 |
| zaro | would any infra-core have time to create a gerrit 2.8 branch so we can get the gerrit upgrade started? | 16:59 |
| *** LinuxJedi_ is now known as LinuxJedi | 16:59 | |
| fungi | zaro: i'll take a look | 16:59 |
| zaro | fungi: thnx | 17:00 |
| fungi | zaro: we've got it tracking upstream/stable-2.8 right now... you want an openstack/2.8.0 branch spun off from that? | 17:00 |
| jeblair | zaro: i thought you were going to install a war from master? | 17:00 |
| *** sarob has quit IRC | 17:00 | |
| jeblair | upstream master | 17:00 |
| *** oubiwann_ has quit IRC | 17:01 | |
| jeblair | zaro: or are you working on both things in parallel? | 17:01 |
| zaro | jeblair: i was thinking we want the change owner patch added on top, so we need the branch. | 17:01 |
| *** dcramer_ has quit IRC | 17:02 | |
| *** fbo is now known as fbo_away | 17:03 | |
| fungi | from a timeline perspective, if it turns out to take almost as long to set up the new build machinery for buck and 2.8 as it takes them to release 2.9, we may get better gains focusing on the upgrade path part and less on the additional features | 17:03 |
| zaro | jeblair: i initially thought install from master was better idea, but since we'll probably want to make some (UI) tweeks anyway so might as well go with our own fork. | 17:03 |
| zaro | yeah, it will be easier and faster to go with upstream master. | 17:04 |
| zaro | so if you prefer upstream master then maybe the 2.8 branch can wait. | 17:04 |
| fungi | for ui tweaks, if it's something which requires rebuilding teh warfile, we probably want to upstream patches to make those externally-configurable instead and wait for it to be in the following upgrade | 17:05 |
| *** rongze has quit IRC | 17:05 | |
| jeblair | fungi: that's preferable, but it could slow things down considerably; i think i like the idea of doing both in parallel... | 17:06 |
| * zaro agrees, lets do in parallel. | 17:06 | |
| *** rongze has joined #openstack-infra | 17:07 | |
| *** sHellUx has quit IRC | 17:07 | |
| jeblair | zaro: that way if you hit something that slows you down while working on master on review-dev, you can switch to getting the build machinery going | 17:07 |
| *** oubiwann_ has joined #openstack-infra | 17:07 | |
| jeblair | and we'll be ahead of the game if we need to use it | 17:07 |
| fungi | carrying patches to adjust the ui in ways which are accepted upstream simply gets us those changes a little faster at the expense of maintaining build infrastructure/tooling for gerrit, while obviously carrying patches which won't be accepted upstream gets us back to the "bad old days" of having a perpetual fork | 17:07 |
| jeblair | fungi: yeah, definitely don't want a perpetual fork | 17:07 |
| fungi | but if the effort to maintain build machinery is relatively minimal, then parallel effort seems fine to me | 17:07 |
| jeblair | i'd say focus on upstream master, and secondarily, hack on build machinery so it's there if we need it | 17:08 |
| zaro | cool. | 17:08 |
| *** yamahata_ has quit IRC | 17:08 | |
| *** jcooley_ has joined #openstack-infra | 17:08 | |
| zaro | fungi: so just go ahead and branch with latest upstream 2.8? | 17:09 |
| *** SergeyLukjanov is now known as _SergeyLukjanov | 17:09 | |
| fungi | zaro: call it openstack/2.8.0? | 17:09 |
| * fungi checks to see what's been tagged on the upstream branch | 17:10 | |
| *** jpich has quit IRC | 17:10 | |
| fungi | looks like the tag is just v2.8 (no micro version) | 17:10 |
| *** loq_mac has joined #openstack-infra | 17:11 | |
| fungi | 12 hours ago | 17:11 |
| zaro | fungi: gerrit channel says watch out for LDAP on upgrade. we not using that are we? | 17:11 |
| fungi | zaro: no ldap for us (yet anyway) | 17:11 |
| fungi | zaro: there is now an openstack/2.8 branch on the openstack-infra/gerrit project | 17:12 |
| BobBall | How do I find out the cinder config in check jobs? | 17:12 |
| zaro | great, thnx. | 17:12 |
| fungi | BobBall: in which check jobs? | 17:12 |
| BobBall | full tempest | 17:12 |
| fungi | BobBall: look at the logs linked for a completed run and the localrc used for devstack should be included | 17:13 |
| BobBall | oh, of course... duh... | 17:13 |
| BobBall | sorry | 17:13 |
| fungi | under the "logs" subdirectory there will be a localrc.gz or similar | 17:13 |
| *** dcramer_ has joined #openstack-infra | 17:14 | |
| fungi | zaro: http://git.openstack.org/cgit/openstack-infra/gerrit/log/?h=openstack/2.8 looks like there are even a couple of post-2.8 patches on that stable branch already | 17:14 |
| fungi | minor documentation fixes, it seems | 17:15 |
| *** jcooley_ has quit IRC | 17:15 | |
| *** dolphm has joined #openstack-infra | 17:17 | |
| *** dstanek has joined #openstack-infra | 17:17 | |
| zaro | fungi: interesting upstream gerrit site is on 2.8-rc3 but i don't see a tag for rc2. | 17:18 |
| fungi | zaro: yeah, i see tags for v2.8-rc0, v2.8-rc1 and v2.8 | 17:20 |
| fungi | zaro: maybe they incremented rc versions without corresponding tags, or deleted tags from the repo | 17:20 |
| *** loq_mac has quit IRC | 17:21 | |
| *** ruhe is now known as ruhe_ | 17:21 | |
| *** loq_mac has joined #openstack-infra | 17:21 | |
| *** ruhe_ is now known as ruhe | 17:21 | |
| *** mrodden has quit IRC | 17:22 | |
| zaro | fungi: would not surprise me. | 17:22 |
| fungi | the subway map for the gate is amusing at the moment | 17:24 |
| jeblair | fungi: lots of trains to nowhere | 17:24 |
| fungi | would make for a great twilight zone plot | 17:25 |
| fungi | "lots of trains to nowhere" | 17:25 |
| *** pblaho has quit IRC | 17:27 | |
| ryanpetrello | (╯°□°)╯︵ ┻━┻ | 17:27 |
| ryanpetrello | mongodb | 17:27 |
| mikal | jeblair: looking at the zuul report formatter, its not clear to me how you get the nice HTML style reports in gerrit. Our email reports are just the url. How are you turning that into a HTML link? | 17:28 |
| fungi | clearly my terminal font is lacking a few glyphs there | 17:28 |
| *** jasond has joined #openstack-infra | 17:28 | |
| jeblair | mikal: gerrit commentlink magic | 17:28 |
| ryanpetrello | fungi: don't mind me, just flipping a table over | 17:29 |
| mikal | jeblair: who the what now? | 17:29 |
| ryanpetrello | so running the ceilometer stuff on centos doesn't seem to have solved it either | 17:29 |
| ryanpetrello | different failure now, though | 17:29 |
| jeblair | mikal: so our reports are exacly as you are seeing, but gerrit does magic on them when displaying them | 17:29 |
| mikal | jeblair: oh, I see (a quick google helped!) | 17:29 |
| mikal | So, I need to trick you guys into letting me tweak the gerrit config as well? | 17:29 |
| fungi | mikal: http://git.openstack.org/cgit/openstack-infra/config/tree/modules/gerrit/templates/gerrit.config.erb#n78 | 17:30 |
| jeblair | mikal: you want your comments to be linkified in openstack gerrit when you post them? | 17:30 |
| jeblair | mikal: if so, you can just make your zuul comment syntax match what we're using, and it'll hit the same commentlink patterns | 17:30 |
| mikal | jeblair: yeah, that's what I want | 17:31 |
| * mikal looks at that template | 17:31 | |
| fungi | mikal: the patterns we match are listed at http://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/gerrit.pp#n128 | 17:31 |
| jasond | hi, i'm trying to understand why the gate failed with "Build timed out" in http://logs.openstack.org/51/59851/2/check/gate-heat-python26/b2d6d73/console.html i've been looking at what appears to be a debug log (http://logs.openstack.org/51/59851/2/check/gate-heat-python26/b2d6d73/tmpUBhIV8), but i'm not sure where to look. the tests for this patch pass locally. can someone advise me where to look next? | 17:31 |
| *** datsun180b is now known as datsun_F40PH | 17:31 | |
| *** hashar has quit IRC | 17:32 | |
| mikal | jeblair: so it looks to me based on that config that your results are in the form of a href? | 17:32 |
| *** chandankumar has quit IRC | 17:33 | |
| jeblair | mikal: i think if you set job_name_in_report=true in zuul.conf you'll get the right pattern | 17:33 |
| mikal | jeblair: oh, cool. I shall try that | 17:33 |
| *** mrodden has joined #openstack-infra | 17:34 | |
| *** loq_mac has quit IRC | 17:36 | |
| *** loq_mac has joined #openstack-infra | 17:36 | |
| mikal | OMG, I want an IRC reporter in zuul | 17:38 |
| mikal | That would be funky | 17:38 |
| *** senk has joined #openstack-infra | 17:38 | |
| *** vkozhukalov has joined #openstack-infra | 17:39 | |
| jeblair | mikal: :) | 17:39 |
| mikal | jeblair: heh, be gentle. This is my first day poking around zuul. | 17:40 |
| *** loq_mac has quit IRC | 17:41 | |
| mordred | mikal: you're a zuul | 17:41 |
| zaro | jeblair, fungi: an email just came out on gerrit mailing list. final 2.8 was just release today so i believe the v2.8 tag is the final. | 17:41 |
| jeblair | mikal: i'd accept that patch. also, keep in mind you could also write an irc trigger. :) | 17:41 |
| * jeblair will be amused if zuul's UI ends up being everything _except_ a web interface. | 17:42 | |
| fungi | jasond: i believe that's the raw subunit stream returned via testr during that job run | 17:42 |
| clarkb | jog0: yes region-* is breaking things, I have a change up to only use region-* on check tests | 17:42 |
| jeblair | clarkb: because it's slow? | 17:43 |
| *** elasticio has quit IRC | 17:43 | |
| jasond | fungi: oh ok, is it meant to help with debugging? | 17:43 |
| fungi | zaro: right, they tagged v2.8 ~13 hours ago, so sayeth teh git | 17:43 |
| clarkb | anteaya: correct I don't think grenade has established a job pattern yet so I don't want to require things in that space yet. Better to learn and see what is going on there | 17:43 |
| clarkb | jeblair: yes | 17:43 |
| *** senk has quit IRC | 17:43 | |
| mordred | fungi, zaro: that still means that 2.8 final doesn't have the zaro patch though, right? | 17:43 |
| mikal | jeblair: I can see a use case for an IRC trigger as well | 17:44 |
| mikal | jeblair: not that I will ever have the free time to implement such a thing | 17:44 |
| fungi | jasond: i think you may be able to feed it into a testtools something or other, someone else here may have a more accurate suggestion there ;) | 17:44 |
| fungi | mordred: i believe we don't expect that patch until 2.9 | 17:44 |
| clarkb | jeblair: fungi: mordred: https://review.openstack.org/#/c/59891/ | 17:45 |
| mordred | fungi: ok. cool | 17:45 |
| mordred | clarkb: looking | 17:45 |
| zaro | mordred: that's correct, no zaro patch. that will be in 2.9 | 17:45 |
| mordred | cool | 17:45 |
| fungi | gah! https://review.openstack.org/60014 failed gating on check-requirements-integration-dsvm | 17:45 |
| mordred | fungi: how did it fail there? | 17:46 |
| mordred | fungi: uhm. why is it pulling in d2to1 | 17:47 |
| fungi | looks like maybe oslo.messaging installation broke? | 17:47 |
| clarkb | jeblair: when I babysat the changes tests were taking longer but not too much longer 14 vs 18 minutes on swift functional and ~30 vs ~40 minutes for neutron tempest. But full tempest seems to do much worse | 17:49 |
| *** oubiwann_ has quit IRC | 17:49 | |
| *** derekh has quit IRC | 17:50 | |
| jeblair | clarkb: what were the full tempest times? i'd extrapolate 45 -> 60 mins based on those 2 data points (and the assumption that full tempest runs take ~45 mins currently) | 17:52 |
| *** rossella_s has quit IRC | 17:52 | |
| mordred | fungi: W T F? | 17:52 |
| clarkb | I saw one test run into the 90 minute timeout just as it was running the last cli tests. So I think just above 90 minutes is the worst case, but would need to look at more test runs to see if that is an outlier | 17:53 |
| clarkb | jeblair: ^ | 17:53 |
| jeblair | wow. that's definitely not linear. | 17:53 |
| *** oubiwann_ has joined #openstack-infra | 17:53 | |
| fungi | mordred: i'm trying to suss out if there's any obvious reference to d2to1 in either master or latest tag of oslo.messaging, but no | 17:54 |
| *** SergeyLukjanov has joined #openstack-infra | 17:55 | |
| *** gyee has joined #openstack-infra | 17:55 | |
| mordred | fungi: right. I also don't understand how that's getting in there, or what's going on in that moment where it's trying to install oslo.messaging and winding up re-installing pbr and finding d2to1 | 17:55 |
| *** ndipanov has joined #openstack-infra | 17:55 | |
| *** michchap has joined #openstack-infra | 17:56 | |
| fungi | mordred: did oslo.messaging 1.2.0 ever get released? tags seem to jump from 1.2.0a11 to 1.3.0a1 and the d2to1 setup_requires was removed between those | 17:57 |
| mordred | fungi: I do not belive it did | 17:57 |
| mordred | markmc: ^^ | 17:58 |
| *** riteshnanda has quit IRC | 17:58 | |
| *** SergeyLukjanov has quit IRC | 17:58 | |
| *** che-arne has quit IRC | 17:59 | |
| *** senk has joined #openstack-infra | 17:59 | |
| *** dcramer_ has quit IRC | 17:59 | |
| fungi | yeah, tarballs and pypi seem to confirm that as well | 17:59 |
| *** dizquierdo has quit IRC | 18:01 | |
| *** oubiwann_ has quit IRC | 18:02 | |
| *** michchap has quit IRC | 18:03 | |
| fungi | mordred: seems it built oslo.messaging==1.3.0a2.5.g566c40e at http://logs.openstack.org/14/60014/1/check/check-requirements-integration-dsvm/c9f065e/console.html#_2013-12-04_16_16_31_368 so presumably that's what it's installing again later for glance | 18:04 |
| ndipanov | hey guys - anyone have any idea who owns the mox3 package on pypi, and more importantly - where's the code kept? | 18:04 |
| *** senk has quit IRC | 18:04 | |
| mordred | ndipanov: I do! | 18:04 |
| *** reed has joined #openstack-infra | 18:05 | |
| ndipanov | mordred, awesome, thanks! | 18:05 |
| ndipanov | :D | 18:05 |
| mordred | ndipanov: https://github.com/emonty/pymox - although I tohught zul and I moved it into gerrit | 18:06 |
| mordred | zul: ^^ ? | 18:06 |
| *** wenlock has quit IRC | 18:06 | |
| ndipanov | mordred, ok that clears a lot of things up - it's not in gerrit btw... | 18:06 |
| clarkb | I thought we werent goign to put effort into mox3 because mock' | 18:07 |
| ndipanov | clarkb, porting for example nova to mock as a project is a huge waste of time imho | 18:07 |
| zul | mordred: i dont think so | 18:07 |
| clarkb | ndipanov: so is forking a project | 18:07 |
| ndipanov | clarkb, true - so I want to figure out what we should do | 18:08 |
| clarkb | at the summit I thought we decided new tests shoudl use mock, old tests will be converted over time | 18:08 |
| clarkb | zul: ^ ? | 18:08 |
| zul | thats right | 18:08 |
| ndipanov | clarkb, that works fine if we can move to mox3 - which no one seems to be doing | 18:08 |
| zul | ndipanov: keystoneclient has already switched over and there is a patch for oslo-incubator to start switching over | 18:10 |
| zul | we are focused on the clients right now | 18:10 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Nodepool: cache all gerrit repos https://review.openstack.org/60040 | 18:11 |
| ndipanov | zul, awesome - there were some concerns about some bug-fixes/divergence with upstream mox | 18:11 |
| ndipanov | if you could call it that | 18:11 |
| *** ruhe has quit IRC | 18:11 | |
| ndipanov | zul, it's be awesome if you would weigh in on the thread in dev right now | 18:11 |
| ndipanov | it would * | 18:11 |
| zul | ndipanov: im aware but upstream is basically dead right now | 18:11 |
| zul | ndipanov: on the ml? | 18:11 |
| ndipanov | zul, yeah | 18:11 |
| zul | ndipanov: ok | 18:11 |
| ndipanov | zul, any plans to backport stuff? | 18:12 |
| zul | ndipanov: i thought there was a thread last month ;) | 18:12 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Add zuul pack-refs cron https://review.openstack.org/59543 | 18:12 |
| ndipanov | that's the same thread :) | 18:12 |
| ndipanov | zul, I just dug it up since I had to -2 a patch converting stuff to mock | 18:12 |
| ndipanov | zul, and there does not seem to be a clear move forward at least in nova | 18:13 |
| clarkb | right I think we are rehashing old discussions | 18:13 |
| ndipanov | clarkb, definitely we are | 18:13 |
| clarkb | the clear move forward is start using mock today for new tests, work on converting old tests as part of the move to support python3 | 18:13 |
| ndipanov | clarkb, and... mova to mox3? | 18:14 |
| ndipanov | move | 18:14 |
| clarkb | no | 18:14 |
| ndipanov | no? | 18:14 |
| clarkb | at the summit there wasnt any desire to support mox3 | 18:14 |
| fungi | projects currently using mox and aiming for py3k compatibility will need to rewrite those tests to use mock instead of mox before they can reach that goal | 18:14 |
| *** wenlock has joined #openstack-infra | 18:15 | |
| ndipanov | I see | 18:15 |
| ndipanov | so mox3 is dead as far as you guys are concerned | 18:16 |
| fungi | afaik mox3 was an experimental fork of a dead project | 18:16 |
| fungi | while mock is actively maintained and works with both python 2.x and 3.x | 18:17 |
| clarkb | mimic was the other option | 18:18 |
| zul | ndipanov: no mox is dead imho | 18:18 |
| jeblair | fungi, mordred: what's up with requirements and https://review.openstack.org/#/c/60014/ ? | 18:18 |
| zul | ndipanov: i just commented on the list | 18:18 |
| mordred | jeblair: we're trying to figure it out | 18:18 |
| mordred | the failure there is WEIRD | 18:18 |
| clarkb | fungi: jeblair: any opposition to my approving https://review.openstack.org/#/c/59891/1 ? | 18:18 |
| fungi | jeblair: i'm still trying to dissect why it tries to go glance->oslo.messaging->d2to1 according to the log | 18:19 |
| clarkb | I think we will get better runtime numbers too since we don't cancel jobs in the check queue often | 18:19 |
| ndipanov | ok well thanks guys - that clears some of things up | 18:19 |
| *** dolphm has quit IRC | 18:19 | |
| jeblair | clarkb: aprvd | 18:19 |
| clarkb | ty | 18:20 |
| *** dolphm has joined #openstack-infra | 18:21 | |
| fungi | http://logs.openstack.org/14/60014/1/gate/gate-tempest-dsvm-full/dfc890c/console.html also bailed on a java exception collecting mysql logs from the slave workspace. haven't seen that one before | 18:22 |
| jeblair | fungi: timeout | 18:22 |
| fungi | aha | 18:22 |
| jeblair | fungi: was interrupted while uploading logs (so just barely timed out) | 18:23 |
| ndipanov | zul, any chance we'll see it in stackforge/gerrit soon? | 18:23 |
| clarkb | so uh the wheel stuff has broken the gate >_> | 18:23 |
| fungi | jeblair: right, and devstack-precise-hpcloud-region-b-788020 | 18:23 |
| clarkb | https://jenkins02.openstack.org/job/gate-keystone-python26/2544/console lots of that sort of thing | 18:23 |
| fungi | which explains the timeout | 18:23 |
| zul | mordred: ^^^ | 18:24 |
| mordred | how did that break the gate? | 18:24 |
| fungi | clarkb: well, the gate is also completely broken today because django 1.6 doesn't work with... something tempest-wise | 18:24 |
| mordred | *facepalm* | 18:24 |
| *** dstanek has quit IRC | 18:24 | |
| *** cederb02 has joined #openstack-infra | 18:25 | |
| openstackgerrit | A change was merged to openstack-infra/config: Don't use new HPCloud regions in the gate. https://review.openstack.org/59891 | 18:25 |
| markmc | mordred, oslo.messaging 1.2.0 was never released, right | 18:25 |
| markmc | mordred, what's up? | 18:25 |
| fungi | clarkb: currently trying to figure out why the revert gor the change bringing in new django won't pass the requirements-integration test | 18:25 |
| clarkb | mordred: "Add support for per-distro wheel mirrors" did it I think | 18:25 |
| reed | we need a gate before the gate | 18:25 |
| mordred | clarkb: right - but - that should be adding those as extra-index-urls - why is it not finding the things in the standard one | 18:26 |
| *** dolphm has quit IRC | 18:26 | |
| clarkb | mordred: oh gotcha, it should also look in the standard place | 18:26 |
| clarkb | pip bug? | 18:26 |
| mordred | clarkb: look before it - for pycrypto - we don't find it in wheel location - but we _do_ find it normally | 18:26 |
| *** cederb02 has quit IRC | 18:26 | |
| reed | with the gate borked I feel bad about asking for a fix to the planet cache ... right? | 18:27 |
| jeblair | mordred: it's throwing lots of _ecxeptions_! | 18:27 |
| mordred | clarkb: ok. let's revert it | 18:27 |
| *** oubiwann_ has joined #openstack-infra | 18:27 | |
| mordred | sorry guys - I blame dstufft | 18:27 |
| *** cederb02 has joined #openstack-infra | 18:27 | |
| reed | hi oubiwann | 18:27 |
| clarkb | oh lol the centos upgrade is causing the problems | 18:27 |
| clarkb | mordred: centos is now 6.5 | 18:27 |
| fungi | whee | 18:27 |
| mordred | that shouldn't matter | 18:27 |
| mordred | no | 18:27 |
| oubiwann_ | reed: hello! | 18:27 |
| clarkb | mordred: I wonder if we just rerun the mirror if it will fix it | 18:27 |
| mordred | it should not matter | 18:27 |
| jeblair | yeah, it needs to not break in this situation | 18:27 |
| mordred | nope. there are two things broken there | 18:27 |
| mordred | BUT | 18:27 |
| clarkb | mordred: oh wait | 18:28 |
| mordred | not having a wheel dir should not break anuthing | 18:28 |
| mordred | it's configured to look in both places | 18:28 |
| mordred | what it seems is happening | 18:28 |
| clarkb | mordred: http://pypi.openstack.org/openstack/CentOS%206.4/ is the dir not CentOS-6.5 and I agree it shouldn't completely break | 18:28 |
| mordred | is that something is causing case insensitivity searching to stop working | 18:28 |
| mordred | WHY IN THE WORLD that change is causing that behvior is gobsmackingly weird | 18:29 |
| clarkb | hmm sphinx vs Sphinx? | 18:29 |
| mordred | yah. that normally works | 18:29 |
| mordred | for some reasons, it's not working here | 18:29 |
| mordred | even though we did not change codepaths that shoudl affect that | 18:29 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Revert "Add support for per-distro wheel mirrors" https://review.openstack.org/60042 | 18:29 |
| *** ^d has quit IRC | 18:29 | |
| *** ^d has joined #openstack-infra | 18:30 | |
| mordred | +2 | 18:30 |
| clarkb | approved | 18:30 |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: upgrade review-dev from gerrit ver 2.4.x to ver 2.8 https://review.openstack.org/60043 | 18:31 |
| mordred | stab stab stab | 18:31 |
| jeblair | reed: what needs to happen? | 18:31 |
| *** jamesmcarthur has quit IRC | 18:32 | |
| openstackgerrit | Khai Do proposed a change to openstack-infra/config: upgrade review-dev from gerrit ver 2.4.x to ver 2.8 https://review.openstack.org/60043 | 18:32 |
| portante | mordred: our pep8 job failed with a UnicodeDecodeError, is that known? | 18:33 |
| openstackgerrit | A change was merged to openstack-infra/config: Revert "Add support for per-distro wheel mirrors" https://review.openstack.org/60042 | 18:33 |
| *** DennyZhang has joined #openstack-infra | 18:33 | |
| mordred | portante: it's not known to me - point me at it? | 18:33 |
| portante | "No distributions at all found for openstack.nose-plugin" | 18:33 |
| portante | https://jenkins02.openstack.org/job/gate-swift-docs/1615/console | 18:33 |
| mordred | oh - that's a whole other thing | 18:33 |
| *** jerryz has joined #openstack-infra | 18:33 | |
| mordred | that's known - we're fixing it | 18:33 |
| portante | k, it seems to be the cause of a whole bunch of gate failures | 18:33 |
| mordred | or, at least, we've reverted the thing that we think caused it | 18:33 |
| mordred | yeah | 18:34 |
| *** markmc has quit IRC | 18:34 | |
| portante | k thx | 18:34 |
| *** DennyZhang has quit IRC | 18:34 | |
| portante | is there a bug to recheck/reverify on? | 18:34 |
| *** loq_mac has joined #openstack-infra | 18:34 | |
| *** jamesmcarthur has joined #openstack-infra | 18:34 | |
| reed | jeblair, I misspelled Terri Yu's name, fixed the blog post but I fear the planet won't refresh unless it's manually triggered | 18:34 |
| portante | mordred: if so, just tell me yes, and I'll find it | 18:34 |
| *** DennyZhang has joined #openstack-infra | 18:34 | |
| jeblair | #status alert gate failures due to django incompatibility, pip bugs, and node performance problems | 18:34 |
| openstackstatus | NOTICE: gate failures due to django incompatibility, pip bugs, and node performance problems | 18:34 |
| *** Abhishek_ has joined #openstack-infra | 18:34 | |
| *** ChanServ changes topic to "gate failures due to django incompatibility, pip bugs, and node performance problems" | 18:34 | |
| *** fbo_away is now known as fbo | 18:35 | |
| *** Abhishek_ has quit IRC | 18:36 | |
| *** loquacities has joined #openstack-infra | 18:36 | |
| jeblair | reed: where is it incorrect? i see "As one of the recipient of the travel grant, Terri Yu told us:" which lgtm | 18:36 |
| reed | jeblair, oh, that's weird | 18:37 |
| jeblair | reed: (except "recipient" should be plural) | 18:37 |
| *** harlowja has joined #openstack-infra | 18:37 | |
| reed | never mind then | 18:37 |
| jeblair | reed: maybe it updated? shift-reload? | 18:37 |
| reed | maybe, yeah | 18:37 |
| reed | ok, issue solved | 18:37 |
| *** Abhishek_ has joined #openstack-infra | 18:37 | |
| *** loq_mac has quit IRC | 18:38 | |
| fungi | mordred: so, the line quoted in setup.py for that traceback was removed in change https://review.openstack.org/49196 which was merged between 1.2.0a11 and 1.3.0a1 | 18:39 |
| *** cederb02 has quit IRC | 18:41 | |
| fungi | mordred: and the requirement is for oslo.messaging>=1.2.0a11 so presumably it's building the minimum version matched there rather than the branch tip or latest tag | 18:41 |
| mordred | why would it build minimum ... wait | 18:41 |
| clarkb | could that possibly be related to the pip weirdness with wheels? | 18:41 |
| *** mancdaz is now known as mancdaz_away | 18:41 | |
| mordred | GAH | 18:42 |
| mordred | GAH GAH GAH | 18:42 |
| * mordred stab stab stab stab stab | 18:42 | |
| mordred | -f http://tarballs.openstack.org/oslo.messaging/oslo.messaging-1.2.0a11.tar.gz#egg=oslo.messaging-1.2.0a11 | 18:42 |
| mordred | oslo.messaging>=1.2.0a11 | 18:42 |
| fungi | mordred: at least i'm otherwise at a loss for how the line, as quoted from setup.py, could in any way be present | 18:42 |
| mordred | >= is a misnomer | 18:42 |
| fungi | oh, that | 18:42 |
| mordred | so that's why it's getting oslo.messaging old with d2to1 | 18:42 |
| mordred | NOW - why is that breaking then and not at other times | 18:42 |
| clarkb | because d2to1 is not in your local mirror | 18:43 |
| fungi | because we're urgently trying to un-break things and the universe hates us for trying to reduce its entropy quotient | 18:43 |
| clarkb | :) is it possible some other dep used to pull in d2to1 and now doesn't? | 18:43 |
| *** Ryan_Lane has quit IRC | 18:43 | |
| fungi | but clarkb's probably on the nose there | 18:43 |
| clarkb | this is why that change zaro wrote is handy, too bad we didn't have it in over the last few days | 18:43 |
| *** ewindisch_ has joined #openstack-infra | 18:44 | |
| *** nati_ueno has quit IRC | 18:45 | |
| *** fbo is now known as fbo_away | 18:45 | |
| *** loquacities has quit IRC | 18:46 | |
| *** osanchez has quit IRC | 18:46 | |
| *** jerryz_ has joined #openstack-infra | 18:46 | |
| portante | mordred: would that pip problem be fixed now? or does it take a while for that change to make it way through the system? | 18:49 |
| portante | See: https://jenkins02.openstack.org/job/gate-swift-python26/1740/console | 18:49 |
| jog0 | fungi: https://review.openstack.org/#/c/60014/ didn't merge | 18:49 |
| fungi | jog0: yes, we're just about done figuring out why | 18:49 |
| jog0 | clarkb: I have a seprate change that will help the region thing https://review.openstack.org/#/c/60000/ | 18:49 |
| *** jerryz has quit IRC | 18:50 | |
| mgagne | 60000! :O | 18:50 |
| clarkb | portante: it can take some time for it to get applied to all of the slaves. Change would've landed on puppet master ~5 minutes ago and puppet runs every 15 minutes so a little after the top of the hour we should be good | 18:50 |
| jog0 | mgagne: \o/ | 18:50 |
| clarkb | jog0: I don't think that change is necessary, we will stop running the large ops tests no those slaves instead | 18:50 |
| *** dolphm has joined #openstack-infra | 18:50 | |
| portante | okay, thanks | 18:50 |
| clarkb | jog0: but your change doesn't hurt either so I won't -1 it | 18:50 |
| jog0 | clarkb: I agree that both patches aren't necessary but I think its safer to lower that number | 18:51 |
| *** oubiwann_ has quit IRC | 18:51 | |
| jog0 | having at 150 isn't helping anything as there are no efforts to get nova and neutron to work better (at least not priority ones) | 18:51 |
| jog0 | and 100 is still a good regression test (we couldn't do 50 before) | 18:51 |
| fungi | mordred: so should the requirements-integration job, in the new landscape once wheels are being built, start ignoring -f lines while the projects work on removing them? | 18:51 |
| *** dolphm has quit IRC | 18:52 | |
| jog0 | oslo.messaging fun | 18:52 |
| *** dolphm has joined #openstack-infra | 18:52 | |
| mordred | fungi: yeah - but I don't think that's our problem right now | 18:52 |
| jerryz_ | clarkb jog0: since you are here, i want to check if elastic-recheck has an exception exit issue. My recheck bot sometimes will end with this log: http://paste.openstack.org/show/54448/ | 18:53 |
| *** mrodden has quit IRC | 18:53 | |
| fungi | jog0: yes, glance wants an older oslo.messaging version at the moment, which in turn wants d2to1 which we're not locally mirroring in the process of requirements-integration tests | 18:53 |
| clarkb | fungi: just add d2to1 to global reqs in that change? | 18:53 |
| jog0 | fungi: one step forward one step back | 18:53 |
| clarkb | then we can fix glance | 18:54 |
| clarkb | this skirts around the problem. but should move us forward | 18:54 |
| mordred | the real questoin is - what caused the local mirror to suddently stop containing d2to1 | 18:54 |
| mordred | I mean, give it a try | 18:54 |
| *** oubiwann_ has joined #openstack-infra | 18:54 | |
| jog0 | jerryz_: looks like an irc connection issue at first glance | 18:54 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Nodepool: cache all gerrit repos https://review.openstack.org/60040 | 18:54 |
| *** krotscheck has joined #openstack-infra | 18:55 | |
| jerryz_ | jog0: do we have a way to maintain the connection? | 18:55 |
| jerryz_ | or how the channel ends up being closed? | 18:55 |
| jog0 | jerryz_: I am not really sure actually I didn't work on the bot mtreinish did | 18:56 |
| jog0 | sorry | 18:56 |
| *** ryanpetrello has quit IRC | 18:56 | |
| *** michchap has joined #openstack-infra | 18:56 | |
| clarkb | jerryz_: I disconnected from freenode last night could that have been what you saw? | 18:56 |
| *** dolphm has quit IRC | 18:57 | |
| clarkb | also it depends on so many things, which server you are connecting to, local firewalls and so on | 18:57 |
| jerryz_ | server=irc.freenode.net | 18:57 |
| *** ryanpetrello has joined #openstack-infra | 18:58 | |
| jerryz_ | i just named a channel and let recheck join. | 18:58 |
| jerryz_ | there are only the bot and me in the channel | 18:58 |
| *** oubiwann_ has quit IRC | 18:58 | |
| *** mrodden has joined #openstack-infra | 18:59 | |
| openstackgerrit | Jeremy Stanley proposed a change to openstack/requirements: Revert "Allow Django 1.5 and 1.6" https://review.openstack.org/60014 | 18:59 |
| fungi | clarkb: mordred: ^ trying the suggestion | 18:59 |
| mordred | nod | 18:59 |
| jog0 | fungi: haha i like the hack | 19:00 |
| jeblair | fungi: er, isn't d2to1 in the mirror? | 19:00 |
| fungi | jeblair: but not in the "local" mirror the integration test builds on-demand for that job | 19:00 |
| mordred | jeblair: it's not showing up in the locally built mirror in in the requirements job | 19:00 |
| mordred | which is bonghits | 19:00 |
| mordred | but we don't know why yet, and we don't know why any changes to that process that caused it to not be there wound up passing tests | 19:01 |
| jeblair | mordred: okay, so how has _that_ not tripped us up before? | 19:01 |
| *** michchap has quit IRC | 19:01 | |
| mordred | right | 19:01 |
| mordred | that's my question | 19:01 |
| openstackgerrit | lifeless proposed a change to openstack-infra/reviewstats: Remove arata and devananda from TripleO core. https://review.openstack.org/60050 | 19:01 |
| mordred | unless we were not skipping mirror properly because of the other change we looked at | 19:01 |
| *** pete5 has quit IRC | 19:02 | |
| jog0 | lifeless: why not hook reviewstats into gerrit groups to know core? | 19:02 |
| *** pete5 has joined #openstack-infra | 19:02 | |
| *** pete5 has quit IRC | 19:02 | |
| fungi | jog0: gerrit 2.4.x provides no mechanism to query group membership | 19:02 |
| *** pete5 has joined #openstack-infra | 19:02 | |
| lifeless | jog0: patches appreciated etc etc etc | 19:02 |
| *** wenlock has quit IRC | 19:02 | |
| jog0 | fungi: lol besides screen scraping? | 19:02 |
| fungi | well, yeah... that *shudder* | 19:03 |
| lifeless | mikal: you get group infro from gerrit, don't you? | 19:03 |
| *** talluri has joined #openstack-infra | 19:04 | |
| lifeless | anyone have ideas about the py33 failure on https://review.openstack.org/#/c/60017/ checks ? | 19:04 |
| *** syerrapragada1 has quit IRC | 19:04 | |
| *** syerrapragada has joined #openstack-infra | 19:04 | |
| fungi | jog0: also, it's not as simple as "members of some group" (you need to collect the approval acl entries from the project(s) of interest, then descend recursively through the referenced groups since gerrit allows arbitrary group inclusion in other groups) | 19:04 |
| jog0 | fungi: ohh gerrit | 19:06 |
| openstackgerrit | A change was merged to openstack-infra/jeepyb: Always sleep after group creation https://review.openstack.org/59583 | 19:07 |
| fungi | lifeless: Non-zero exit code (2) from test listing. stdout=b'\xb3 `@R\x17text/plain;charset=utf8\rimport errors"reviewstats.tests.test_projectinfo\xc8\x19\xce\xfe', stderr=None | 19:08 |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Remove hpcloud region-a from nodepool. https://review.openstack.org/60051 | 19:08 |
| clarkb | mordred: ^ | 19:08 |
| fungi | lifeless: that's a new one on me | 19:08 |
| mordred | clarkb: thanks | 19:08 |
| jog0 | clarkb: just out of curiosity why remove new region all together? | 19:09 |
| jerryz_ | clarkb: i see irc has an ping-pong heartbeat mechanism. if the server doesn't get a pong for a ping request, the connection to the client will be closed. i suspect somehow either ping or pong didn't get through firewall. but e-r is not supposed to die in this scenario | 19:10 |
| fungi | clarkb: mordred: do we think region a is unsuitable but region b is fine? | 19:10 |
| clarkb | mordred: ^ | 19:10 |
| *** ^d has quit IRC | 19:10 | |
| lifeless | fungi: oh right, my eyes glazes on that | 19:10 |
| clarkb | jerryz_: right, it should probably try to reconnect with an expoential backoff | 19:10 |
| mordred | fungi: request from HP - region-a is under pressure and we should really just stick with region-b for the new cloud | 19:10 |
| lifeless | fungi: I have a patch to make that nicer, but it at least lets me know what to poke at | 19:11 |
| mordred | fungi: I probably should have known that already before we added a in - so it's my bad | 19:11 |
| fungi | mordred: worth noting, we're getting similarly slow performance on region b. presumably this is not intended to address that | 19:11 |
| lifeless | fungi: it means discover failed on py33 | 19:11 |
| mordred | it is not | 19:11 |
| lifeless | fungi: but where did stderr for it go? | 19:11 |
| jog0 | mordred: ahh | 19:11 |
| fungi | lifeless: that is indeed a very, very good question | 19:12 |
| lifeless | fungi: testr in that version calls subprocess with stderr undefined, so it will be on the processes fd 2 | 19:12 |
| jeblair | clarkb: fyi your change will cause nodepool to fail to clean up anything in region-a | 19:12 |
| jeblair | clarkb: so you may want to 'nova delete' things manually | 19:12 |
| lifeless | fungi: setup.py's hook does | 19:12 |
| lifeless | return commands.run_argv([sys.argv[0]] + list(args), | 19:13 |
| lifeless | sys.stdin, sys.stdout, sys.stderr) | 19:13 |
| lifeless | so it should also be wired directly to fd2 | 19:13 |
| clarkb | jeblair: ok, will the nodepool DB need cleaning up too? | 19:13 |
| clarkb | jeblair: should I just remove region-a from the jenkins masters and leave the creds there? | 19:13 |
| jeblair | clarkb: that would be the cleanest way | 19:13 |
| clarkb | I just -2'd the change, will update | 19:14 |
| fungi | clarkb: jeblair: could we reduce the quota to 0 with a patch first, and then remove once settled? | 19:14 |
| *** wenlock has joined #openstack-infra | 19:14 | |
| *** johnthetubaguy has quit IRC | 19:14 | |
| ArxCruz | jeblair: when you got time, check if https://review.openstack.org/#/c/55829/ is what you want :) | 19:14 |
| jeblair | fungi: i think removing it from the masters will be sufficient. _also_ setting quota to 0 shouldn't hurt. | 19:15 |
| *** ^d has joined #openstack-infra | 19:15 | |
| *** ^d has quit IRC | 19:15 | |
| *** ^d has joined #openstack-infra | 19:15 | |
| openstackgerrit | Clark Boylan proposed a change to openstack-infra/config: Stop using hpcloud region-a. https://review.openstack.org/60051 | 19:15 |
| jeblair | ArxCruz: will do; there's a lot going on right now. | 19:16 |
| ArxCruz | i know, when you got chance ;) | 19:17 |
| *** thomasem has quit IRC | 19:17 | |
| *** senk has joined #openstack-infra | 19:18 | |
| *** markwash has quit IRC | 19:19 | |
| *** senk has quit IRC | 19:20 | |
| *** senk has joined #openstack-infra | 19:20 | |
| *** fbo_away is now known as fbo | 19:21 | |
| openstackgerrit | John Dickinson proposed a change to openstack-infra/reviewstats: update swift core https://review.openstack.org/60053 | 19:23 |
| *** talluri has quit IRC | 19:23 | |
| clarkb | I need to AFK for a bit during the middle of the day. Going to visit the dentist. Will do my best to keep up with the fires | 19:24 |
| *** SergeyLukjanov has joined #openstack-infra | 19:24 | |
| *** SergeyLukjanov has quit IRC | 19:24 | |
| *** SergeyLukjanov has joined #openstack-infra | 19:25 | |
| fungi | yep, and i'm disappearing for a few hours starting a little after 2100 utc (monthly local osug meetup) | 19:25 |
| clarkb | fungi: I see tempest tests are running and passing. is django 1.6 only a problem occasionally? | 19:25 |
| fungi | clarkb: on full, postgres-full and neutron i think | 19:26 |
| jeblair | fungi, mordred: can you summarize where you are on the django thing? | 19:26 |
| sdague | clarkb: where do you see passes? | 19:26 |
| *** vkozhukalov has quit IRC | 19:26 | |
| clarkb | oh I see now | 19:26 |
| mordred | jeblair: fungi uploaded a modified change and we're watching to see if it passes build | 19:26 |
| clarkb | sdague: I was looking at largeops and things not realizing an actual test was failing | 19:26 |
| sdague | heh | 19:27 |
| clarkb | I thought it was pure dependency weirdness | 19:27 |
| *** nicedice has joined #openstack-infra | 19:27 | |
| sdague | ok, do we have a bug on this yet? | 19:27 |
| *** sarob has joined #openstack-infra | 19:27 | |
| sdague | I was going to write the er for it | 19:27 |
| clarkb | I didn't write one | 19:27 |
| fungi | though jenkins/zuul is estimating 51 minutes remaining on https://jenkins02.openstack.org/job/check-requirements-integration-dsvm/9/ | 19:27 |
| clarkb | mordred: want to approve https://review.openstack.org/#/c/60051/2 ? | 19:28 |
| mordred | clarkb: done. thanks | 19:29 |
| jeblair | i approved 60014 because it's silly to wait on it in check | 19:30 |
| fungi | fair enough | 19:31 |
| lifeless | russellb: python3 we'll need to make non-voting I think | 19:31 |
| sdague | https://bugs.launchpad.net/horizon/+bug/1257885 | 19:31 |
| uvirtbot | Launchpad bug 1257885 in horizon "horizon does not work with django 1.6 " [Undecided,New] | 19:31 |
| lifeless | russellb: paramiko :(. | 19:31 |
| russellb | lifeless: ah ok, fine with me | 19:31 |
| russellb | lifeless: thanks :) | 19:31 |
| openstackgerrit | A change was merged to openstack-infra/config: Stop using hpcloud region-a. https://review.openstack.org/60051 | 19:32 |
| jeblair | lifeless: why not remove it? | 19:32 |
| sdague | jeblair: agreed, honestly, you should kill everything else in the zuul queue | 19:32 |
| sdague | because they are all going to fail until that goes in | 19:32 |
| lifeless | jeblair: I've fixed all the other bits I could find, when paramiko merges the 3 series patch it should just work | 19:32 |
| fungi | sdague: we don't know for sure that it's going to succeed either | 19:32 |
| fungi | sdague: but yeah | 19:32 |
| lifeless | jeblair: and we have this openstack wide desire to support 3 well, don't we? | 19:32 |
| sdague | fungi: sure, but we know everything else is going to fail | 19:32 |
| jeblair | lifeless: oh, ok, so there is hope. nonvoting is fine then. | 19:32 |
| sdague | might as well get rid of it | 19:32 |
| *** ^demon has joined #openstack-infra | 19:33 | |
| *** rpodolyaka1 has joined #openstack-infra | 19:33 | |
| jeblair | sdague, fungi: i will make it happen | 19:33 |
| mgagne | jeblair: is /home/gerrit2/acls/inactive.config a built-in ACL? won't gerrit crash if acl config does not exist? | 19:35 |
| mgagne | jeblair: or is it handled by jeepyb? | 19:35 |
| openstackgerrit | lifeless proposed a change to openstack-infra/config: Make python3.3 nonvoting for reviewstats. https://review.openstack.org/60058 | 19:35 |
| jeblair | mgagne: doesn't exist yet. would you mind writing it as part of that change? | 19:35 |
| fungi | i suppose i could remove django 1.6 from pypi.openstack.org as a temporary measure to reduce pain | 19:35 |
| mgagne | jeblair: how are projects deprecated in openstack? | 19:35 |
| *** ^d has quit IRC | 19:36 | |
| sdague | clarkb: we don't index horizon logs yet eh? is that because of the os-loganalyze filter? | 19:36 |
| jeblair | mgagne: we haven't really done it yet. we should probably do something similar for melange. | 19:37 |
| jeblair | mgagne: that's the only one i think. | 19:37 |
| *** dcramer_ has joined #openstack-infra | 19:37 | |
| *** ^demon has quit IRC | 19:37 | |
| *** mrodden has quit IRC | 19:39 | |
| *** fbo is now known as fbo_away | 19:39 | |
| fungi | given that previous tests of 60014 passed tempest tests which were failing without it, i think it's safe to say it will solve the django situation once merged... so any reason not to delete static.o.o:/srv/static/pypi/openstack/Django/Django-1.6.tar.gz in the meantime? | 19:40 |
| jeblair | fungi: wfm | 19:40 |
| * fungi does | 19:40 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: add query for the horizon gate wedge https://review.openstack.org/60059 | 19:40 |
| jeblair | 60014,2 is a head now | 19:41 |
| sdague | jeblair: so do you think this is close enough to resolving that we don't need to land that er check? | 19:42 |
| sdague | http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkZBSUw6IHRlbXBlc3Quc2NlbmFyaW8udGVzdF9kYXNoYm9hcmRfYmFzaWNfb3BzLlRlc3REYXNoYm9hcmRCYXNpY09wcy50ZXN0X2Jhc2ljX3NjZW5hcmlvW2Rhc2hib2FyZF1cIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNDMyMDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg2MTg2MTQyMDQ1fQ== | 19:42 |
| jeblair | sdague: i'd do it; there's still a bunch of check jobs outstanding, and it may catch some of the gate jobs still running | 19:42 |
| *** mrodden has joined #openstack-infra | 19:42 | |
| sdague | jeblair: ok, landing | 19:43 |
| openstackgerrit | Ivan Kolodyazhny proposed a change to openstack-infra/config: Added Fuel Docs project to StackForge https://review.openstack.org/59932 | 19:45 |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: add query for the horizon gate wedge https://review.openstack.org/60059 | 19:45 |
| *** wenlock has quit IRC | 19:46 | |
| jerryz_ | fungi: did you ever turn up the flavor for devstack slaves? Recently I have been noticing in my own CI environment ,tempest is prone to failures possibly caused by race condition bugs. after i increase the number of cores for devstack slaves, most of the failures are gone. | 19:47 |
| *** bknudson has joined #openstack-infra | 19:47 | |
| jerryz_ | fungi: is tempest in parallel demanding more performance than ever? | 19:47 |
| fungi | jerryz_: i don't recall being asked, but we run 8gb flavor slaces | 19:48 |
| fungi | er, slaves | 19:48 |
| fungi | so that's 4 vcpus and 8gb ram | 19:49 |
| jerryz_ | jerryz: my ram is 8gb too. i just increased the number of cpus | 19:49 |
| jerryz_ | jerryz: so i wonder what is the type of physical cpu hp cloud uses? | 19:49 |
| *** Mithrandir has quit IRC | 19:49 | |
| fungi | jerryz_: sounds likely to affect it, sure. in this case i think we consider the sorts of race conditions we expose in openstack by putting resource pressure on the slave vms to represent actual errors in openstack we want to fix | 19:50 |
| jerryz_ | fungi: mine is 2.4Ghz intel 5620 | 19:51 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: create a selection of jobs to not use config drive https://review.openstack.org/60061 | 19:52 |
| *** dolphm has joined #openstack-infra | 19:53 | |
| *** Mithrandir has joined #openstack-infra | 19:53 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow configdrive vs. metadata server to be configured https://review.openstack.org/59303 | 19:54 |
| sdague | jeblair: so if 60061 lands first, then 59303 is actually self testing | 19:55 |
| *** dcramer_ has quit IRC | 19:56 | |
| *** miqui has joined #openstack-infra | 19:56 | |
| *** michchap has joined #openstack-infra | 19:57 | |
| *** dolphm has quit IRC | 19:57 | |
| miqui | zuul users.....seen this before http://15.185.238.106/show/30/ | 19:58 |
| mordred | fungi: failed | 19:58 |
| miqui | fungi or mordred...perhaps you have seen this one http://15.185.238.106/show/30/ | 19:58 |
| jeblair | miqui: perhaps it's a draft change | 19:59 |
| fungi | mordred: gah, so is that local mirror being built by installing the global requirements list, or by installing the integrated projects? | 20:00 |
| fungi | mordred: if the latter, then if no project is directly installing d2to1 perhaps it doesn't end up in the local mirror copy? | 20:00 |
| miqui | hmm ..jeblair... what is interesting is the KeyError: 'project' | 20:01 |
| mordred | fungi: it's built by running pypi-mirror against the requirements repo | 20:01 |
| clarkb | I don't see the failure | 20:01 |
| fungi | clarkb: http://logs.openstack.org/14/60014/2/check/check-requirements-integration-dsvm/7821a08/console.html | 20:01 |
| jeblair | clarkb: check failure | 20:01 |
| clarkb | gotcha | 20:02 |
| *** senk has quit IRC | 20:02 | |
| *** senk has joined #openstack-infra | 20:02 | |
| anteaya | clarkb: okay, fungi has left some comments so I'll talk to EmilienM about addressing those and then see where we are about a plan to move forward | 20:02 |
| *** michchap has quit IRC | 20:02 | |
| clarkb | another work around would be to remove newer django from the mirror (to get other changes to work) | 20:02 |
| jeblair | clarkb: fungi has done that | 20:03 |
| fungi | clarkb: i did | 20:03 |
| anteaya | grenade for Neutron is one of our must haves so we have to move forward on it as part of our testing mandate | 20:03 |
| fungi | clarkb: hopefully jobs starting after 1940 utc will get django 1.5.5 | 20:03 |
| fungi | clarkb: at least until the next mirror run, if we don't get this revert through | 20:03 |
| jeblair | since everyone is about to leave, how about we just merge the revert. | 20:04 |
| *** boris-42 has quit IRC | 20:04 | |
| fungi | jeblair: i'll upload a switch back to ps1 for it then | 20:04 |
| clarkb | wfm | 20:05 |
| mordred | ++ | 20:05 |
| openstackgerrit | Jeremy Stanley proposed a change to openstack/requirements: Revert "Allow Django 1.5 and 1.6" https://review.openstack.org/60014 | 20:05 |
| fungi | that's ^ back to just the plain revert, no d2to1 monkeybusiness | 20:05 |
| openstackgerrit | A change was merged to openstack/requirements: Revert "Allow Django 1.5 and 1.6" https://review.openstack.org/60014 | 20:05 |
| *** sarob has quit IRC | 20:06 | |
| *** herndon_ has joined #openstack-infra | 20:06 | |
| openstackgerrit | Monty Taylor proposed a change to openstack-dev/pbr: Revert "Use wheels for installation" https://review.openstack.org/60063 | 20:06 |
| *** sarob has joined #openstack-infra | 20:06 | |
| *** herndon_ has left #openstack-infra | 20:06 | |
| mordred | that ^^ is the only other change that's happened that I could see causing this | 20:06 |
| *** iv_m has joined #openstack-infra | 20:06 | |
| mordred | of course, why it itself wouldn't have failed is behond me | 20:06 |
| *** oubiwann_ has joined #openstack-infra | 20:07 | |
| *** dolphm has joined #openstack-infra | 20:08 | |
| ArxCruz | clarkb: fungi mordred is the check-tempest-dsvm-neutron always failing ? | 20:08 |
| fungi | ArxCruz: if it's the 500 internal server error pattern, recheck bug 1257809 | 20:09 |
| uvirtbot | Launchpad bug 1257809 in openstack-ci "Django 1.6 found incompatible during integration testing" [Critical,Fix released] https://launchpad.net/bugs/1257809 | 20:09 |
| *** derekh has joined #openstack-infra | 20:10 | |
| *** dstanek has joined #openstack-infra | 20:10 | |
| *** sarob has quit IRC | 20:10 | |
| fungi | though i think elastic-recheck may be starting to report it as bug 1257885 instead | 20:10 |
| uvirtbot | Launchpad bug 1257885 in horizon "horizon does not work with django 1.6 " [High,Triaged] https://launchpad.net/bugs/1257885 | 20:10 |
| mordred | fungi, jeblair: so - summary of things we've learned is, I guess, before we had a bug in requirements testing which caused 1.6 django to get through untested. somehwere in the midst of addressing that, something broke all requirements integration testing | 20:11 |
| openstackgerrit | Stuart Warren proposed a change to openstack-infra/jenkins-job-builder: Add Mercurial plugin feature to jenkins-job-builder https://review.openstack.org/60066 | 20:11 |
| mordred | so if we revert enough things, we should be back to just "reuiqrements testing is broken as before" | 20:11 |
| jeblair | fungi: i marked the bug as a dup. | 20:11 |
| fungi | k | 20:11 |
| mordred | does that sound right to you guys? | 20:11 |
| mordred | I'm thinking a next step, if it starts working again | 20:12 |
| zaro | is it possible to clone a non-master branch with gerrit-git-prep? | 20:12 |
| dkranz | Is the persistent "ImageKilledException: Image dc0db32f-5da6-4125-a143-39948c8b22be 'killed' while waiting for 'active'" on the radar? | 20:12 |
| *** sarob has joined #openstack-infra | 20:12 | |
| *** Abhishek_ has quit IRC | 20:12 | |
| fungi | mordred: yeah. i mean, if you revert enough things you'll be back to commit 0x0 | 20:12 |
| jeblair | mordred: yes. i'd like to get the mirror selection changes in. | 20:12 |
| mordred | maybe to merge just the select mirror logic _without_ the enable wheel change - thinking that perhaps the enable wheel change is what's making things bork | 20:12 |
| dkranz | I had not seen that before but it just started happening a huge amount | 20:12 |
| mordred | even though its ability to make things bork seems very strange | 20:12 |
| fungi | dkranz: have an example log? | 20:14 |
| jeblair | nnfi doesn't seem to play well with a merged change in the queue :( | 20:14 |
| dkranz | fungi: http://logs.openstack.org/41/54941/13/check/check-tempest-dsvm-full/fb7700c/console.html | 20:14 |
| jog0 | dkranz: in general lets try to keep the is bug x known in the qa room being its qa related | 20:14 |
| jog0 | dkranz: but while here | 20:14 |
| jog0 | dkranz: I was waiting for gate to unwedge before digging into that | 20:15 |
| dkranz | Actually perhaps it is the same as the http 500 failure which also shows in the log | 20:15 |
| jog0 | our baseline is very dirty now so figured better to wait a bit | 20:15 |
| *** dstanek has quit IRC | 20:15 | |
| jeblair | mordred, fungi, clarkb: please review https://review.openstack.org/#/c/59865/ so that it can get picked up in the zuul restart i'm about to do. | 20:15 |
| fungi | dkranz: so probably django 1.6 | 20:15 |
| mordred | jeblair: looking | 20:16 |
| dkranz | fungi: Perhaps. BUt the stacktrace is from a different test. Ugh | 20:16 |
| dkranz | fungi: But they started happening about the same time so probably related. | 20:17 |
| *** sarob has quit IRC | 20:17 | |
| *** dstanek has joined #openstack-infra | 20:17 | |
| dkranz | jog0: ok | 20:17 |
| jog0 | dkranz: yeah I am not convinced its related but too much noise right now so figured better wait | 20:17 |
| jog0 | I thought that a few hours ago though | 20:17 |
| jog0 | but we should be out of the water soon | 20:18 |
| sdague | the code size bar on new gerrit is super nice - https://gerrit-review.googlesource.com/#/q/status:open,n,z can't wait until we get new gerrit in infra | 20:18 |
| *** iv_m has left #openstack-infra | 20:18 | |
| mordred | jeblair: fyi - your lambda function can be replaced with key=operator.itemgetter('name') | 20:19 |
| mordred | jeblair: but I don't care | 20:19 |
| jog0 | sdague: that is really slick | 20:19 |
| mordred | jeblair: more of a "I learned about that function recently, so I thought I'd share" - +A | 20:19 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/elastic-recheck: remove space that fails to have this check match https://review.openstack.org/60070 | 20:20 |
| jog0 | so https://review.openstack.org/#/c/60014/ merged now we expect things to pass? | 20:20 |
| fungi | jog0: since deleting django 1.6 from the mirror at 1940 utc we expect jobs starting after that time to pass | 20:21 |
| *** boris-42 has joined #openstack-infra | 20:22 | |
| openstackgerrit | A change was merged to openstack-infra/elastic-recheck: remove space that fails to have this check match https://review.openstack.org/60070 | 20:23 |
| *** dstanek has quit IRC | 20:23 | |
| jog0 | fungi: cool I can see http://status.openstack.org/elastic-recheck/ dropping already | 20:23 |
| jeblair | mordred: lifeless pointed that out as well. it would require an import. also, it's attrgetter. tbh, i find the lambda easier in general, but will consider attrgetter in the future if i need performance or sorting on multiple attributes. | 20:24 |
| *** Abhishek_ has joined #openstack-infra | 20:24 | |
| *** che-arne has joined #openstack-infra | 20:24 | |
| mordred | jeblair: nod | 20:24 |
| jamesmcarthur | Hi - is sdague around today? | 20:26 |
| sdague | jamesmcarthur: yes | 20:26 |
| sdague | sorry, it was late last night when you were looking for me | 20:26 |
| jamesmcarthur | sdague: no worries! i'm troubleshooting the problem you reported with the member profile/affiliations on openstack.org | 20:27 |
| jamesmcarthur | I've had a lot of trouble replicating the errors you were seeing. However, after some analysis, we identified a couple of problems, which I believe we resolved early this morning. | 20:28 |
| *** melwitt has joined #openstack-infra | 20:28 | |
| jamesmcarthur | I'm not 100% they're related to the ajax timeouts you were seeing, but I was hoping to check with you again. | 20:28 |
| *** loq_mac has joined #openstack-infra | 20:29 | |
| *** dstanek has joined #openstack-infra | 20:29 | |
| jamesmcarthur | If your'e still running into issues, I'd like to set up a time to troubleshoot with you, at your convenience. | 20:29 |
| fungi | zaro: if nobody answered your question yet, gerrit-git-prep resets to whatever branch the change is being applied to, so if you submit a change to the openstack/2.8 branch of openstack-infra/gerrit for example, that's what it should check out to run jobs for it | 20:31 |
| sdague | jamesmcarthur: ok, let me give it a shot | 20:31 |
| jamesmcarthur | sdague: thanks! | 20:31 |
| *** loquacities has joined #openstack-infra | 20:31 | |
| fungi | zaro: more accurately, it should check out whatever ref zuul has constructed and tells it to use | 20:33 |
| *** ^d has joined #openstack-infra | 20:33 | |
| fungi | but zuul takes the gerrit branch for the change into account to determine how to build the ref | 20:33 |
| *** loq_mac has quit IRC | 20:33 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Fix merging transitively-connected change queues. https://review.openstack.org/59865 | 20:35 |
| miqui | @jeblair ---> <jeblair> miqui: perhaps it's a draft change | 20:35 |
| miqui | can you elaborate? | 20:35 |
| jeblair | miqui: it probably got bad data back from gerrit; that might happen if it's a draft change | 20:35 |
| *** sarob has joined #openstack-infra | 20:37 | |
| openstackgerrit | A change was merged to openstack-infra/zuul: Add a zuul client https://review.openstack.org/58841 | 20:37 |
| devananda | hi guys! how would i get ironic added to http://status.openstack.org/reviews/ ? | 20:37 |
| *** sarob_ has joined #openstack-infra | 20:38 | |
| *** sarob has quit IRC | 20:39 | |
| *** sarob_ has quit IRC | 20:40 | |
| *** Abhishek_ has quit IRC | 20:40 | |
| *** sarob has joined #openstack-infra | 20:40 | |
| miqui | ..thanks jeblair... | 20:40 |
| *** senk has quit IRC | 20:41 | |
| russellb | lifeless: did you already submit the change to make py33 non-voting for reviewstats? if not i can | 20:41 |
| jeblair | restarting zuul | 20:42 |
| *** _david_ has joined #openstack-infra | 20:42 | |
| *** loquacities has quit IRC | 20:42 | |
| fungi | noted | 20:43 |
| *** jerryz_ has quit IRC | 20:43 | |
| mgagne | devananda: my understanding is that this page is generated by https://github.com/openstack-infra/reviewday Project list is hardcoded here: https://github.com/openstack-infra/reviewday/blob/master/bin/reviewday#L27 | 20:44 |
| devananda | mgagne: perfect. thanks | 20:44 |
| SpamapS | 2013-12-04 18:47:54.077 | No distributions at all found for sphinx>=1.1.2 (from -r /home/jenkins/workspace/gate-os-apply-config-python26/test-requirements.txt (line 7)) ?? | 20:44 |
| jeblair | zuul is running, and i have re-enqueued the check and gate jobs using the new script (!) | 20:44 |
| *** sarob has quit IRC | 20:45 | |
| fungi | yummy, yummy dogfood! | 20:45 |
| *** sarob_ has joined #openstack-infra | 20:45 | |
| portante | jeblair: is there a bug number that I can recheck jobs against? | 20:45 |
| openstackgerrit | Devananda van der Veen proposed a change to openstack-infra/reviewday: Add Ironic projects https://review.openstack.org/60071 | 20:45 |
| portante | for the above "no distributions ..." | 20:45 |
| fungi | portante: for which specific failures? | 20:45 |
| mordred | SpamapS: yeah - that's evil we're working on fixing right now | 20:46 |
| mordred | fungi: I suppose maybe I should file a bug? | 20:46 |
| fungi | sounds like there isn't one yet | 20:46 |
| portante | almost all of our most recent jobs have been hosed by the no distributions | 20:46 |
| jeblair | mordred: is that the pbr revert? | 20:46 |
| *** derekh has quit IRC | 20:46 | |
| mordred | jeblair: yeah. at least I hope it is | 20:47 |
| jeblair | mordred: i put it at the top of the queue | 20:47 |
| *** sarob has joined #openstack-infra | 20:47 | |
| *** sarob_ has quit IRC | 20:48 | |
| mordred | SpamapS, portante: bug 1257905 | 20:48 |
| uvirtbot | Launchpad bug 1257905 in openstack-ci "mirror building broken requirements testing" [Undecided,New] https://launchpad.net/bugs/1257905 | 20:48 |
| portante | mordred: thanks, should I recheck or wait? | 20:49 |
| fungi | portante: i think we're still waiting for openstack-dev/pbr 60063,1 to merge | 20:49 |
| portante | okay, i'll hold off then | 20:49 |
| fungi | it's got 51 minutes by the status page estimate | 20:49 |
| fungi | and no changes ahead of it | 20:50 |
| portante | oh | 20:50 |
| *** dprince has quit IRC | 20:50 | |
| portante | once it merges, will it take long to get deployed? | 20:51 |
| portante | fungi? | 20:51 |
| *** vipul is now known as vipul-away | 20:51 | |
| SpamapS | mordred: ty | 20:51 |
| fungi | portante: i think once that change merges any jobs started after that should get the old (working) behavior again | 20:51 |
| mordred | fungi: if it's still broken at that point, I'm going to rewrite everytihng in go | 20:52 |
| clarkb | and rust | 20:53 |
| fungi | mordred: static linking will solve all our problems | 20:53 |
| *** MarkAtwood has joined #openstack-infra | 20:55 | |
| sdague | mordred: man, you should just do asm.js to get full baremetal performance | 20:55 |
| mordred | I don't even want to know what you just said | 20:57 |
| jeblair | mordred: help me out here... on https://jenkins01.openstack.org/job/gate-requirements-integration-dsvm/19/console | 20:58 |
| mordred | jeblair: looking | 20:58 |
| jeblair | mordred: why do i keep seeing 'pip wheel' commands? | 20:58 |
| *** hashar has joined #openstack-infra | 20:58 | |
| mordred | jeblair: that builds the wheels | 20:58 |
| mordred | which is a function in pypi-mirror itself | 20:58 |
| *** markwash has joined #openstack-infra | 20:58 | |
| jeblair | ah i see | 20:59 |
| mordred | jeblair: btw - I think there may be a bug somewhere in our mirror building | 20:59 |
| mordred | jeblair: like, the production mirror building | 20:59 |
| mordred | I need to go look at logs | 20:59 |
| mordred | but the fact that we do not have a centos 6.5 dir published on the mirror is concerning | 20:59 |
| openstackgerrit | A change was merged to openstack-infra/reviewday: Add Ironic projects https://review.openstack.org/60071 | 21:00 |
| clarkb | mordred why? didnt you add that a while back? | 21:01 |
| mordred | clarkb: yup. that's what it's concerning that it's not there | 21:02 |
| clarkb | oh that is because centos 6.5 is brand new | 21:02 |
| clarkb | released yesterday. probably need to rerun mirror job | 21:02 |
| *** wenlock has joined #openstack-infra | 21:05 | |
| *** SergeyLukjanov has quit IRC | 21:05 | |
| *** esker has quit IRC | 21:06 | |
| openstackgerrit | A change was merged to openstack-infra/config: Remove braces from bash vars in solum job config https://review.openstack.org/60027 | 21:06 |
| *** sandy__ has joined #openstack-infra | 21:08 | |
| *** sandywalsh_ has quit IRC | 21:08 | |
| *** harlowja has quit IRC | 21:09 | |
| *** katyafervent has quit IRC | 21:09 | |
| *** katyafervent has joined #openstack-infra | 21:09 | |
| *** harlowja has joined #openstack-infra | 21:09 | |
| *** DennyZhang has quit IRC | 21:10 | |
| *** smarcet has left #openstack-infra | 21:11 | |
| fungi | okay, heading out, but will be semi-on from the cisco offices in a bit | 21:15 |
| mordred | clarkb: ok. I'll hold off on panicing on that so far | 21:15 |
| *** denis_makogon has quit IRC | 21:17 | |
| *** paul-- has joined #openstack-infra | 21:17 | |
| *** loq_mac has joined #openstack-infra | 21:17 | |
| *** dcramer_ has joined #openstack-infra | 21:18 | |
| *** loquacities has joined #openstack-infra | 21:20 | |
| zaro | fungi: thnx for answering my question about gerrit-git-prep. | 21:21 |
| jeblair | mordred, fungi: so at least one gate job has fully succeeded (#3) | 21:21 |
| jeblair | (behind the pbr job) | 21:21 |
| mordred | jeblair: thank god | 21:21 |
| jeblair | s/job/change/ | 21:21 |
| *** loq_mac has quit IRC | 21:22 | |
| *** Mithrandir has quit IRC | 21:24 | |
| *** Mithrandir has joined #openstack-infra | 21:24 | |
| *** julim has quit IRC | 21:25 | |
| *** loquacities has quit IRC | 21:25 | |
| *** vipul-away is now known as vipul | 21:25 | |
| *** Abhishek_ has joined #openstack-infra | 21:27 | |
| *** dstanek has quit IRC | 21:28 | |
| *** dstanek has joined #openstack-infra | 21:29 | |
| *** dstanek_afk has quit IRC | 21:30 | |
| *** dkranz has quit IRC | 21:30 | |
| *** dstanek has quit IRC | 21:30 | |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Add secondary index support https://review.openstack.org/60080 | 21:31 |
| *** dstanek has joined #openstack-infra | 21:32 | |
| *** dstanek has quit IRC | 21:33 | |
| *** thomasem has joined #openstack-infra | 21:33 | |
| *** thomasem has quit IRC | 21:34 | |
| *** thomasem has joined #openstack-infra | 21:34 | |
| *** hashar has quit IRC | 21:35 | |
| *** dstanek has joined #openstack-infra | 21:36 | |
| *** loq_mac has joined #openstack-infra | 21:37 | |
| *** paul-- has quit IRC | 21:37 | |
| *** sarob has quit IRC | 21:39 | |
| *** sarob has joined #openstack-infra | 21:39 | |
| openstackgerrit | Steve Baker proposed a change to openstack-infra/config: Enable neutron on heat-slow experimental job https://review.openstack.org/60084 | 21:39 |
| *** dkliban_ has quit IRC | 21:39 | |
| jeblair | restarted nodepool to get that logging change in | 21:42 |
| *** sarob has quit IRC | 21:43 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Add grenade-dsvm-forward job template https://review.openstack.org/53879 | 21:44 |
| *** dolphm has quit IRC | 21:45 | |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/config: Add grenade-dsvm-forward job template https://review.openstack.org/53879 | 21:47 |
| *** rongze has quit IRC | 21:49 | |
| *** Abhishek_ has quit IRC | 21:51 | |
| *** Abhishek_ has joined #openstack-infra | 21:51 | |
| *** CaptTofu has quit IRC | 21:53 | |
| *** Abhishe__ has joined #openstack-infra | 21:54 | |
| lifeless | russellb: I did | 21:54 |
| lifeless | russellb: https://review.openstack.org/60058 | 21:55 |
| *** Abhishek_ has quit IRC | 21:55 | |
| fungi | zuul status is indeed looking far greener | 21:55 |
| *** loq_mac has quit IRC | 21:55 | |
| *** loq_mac has joined #openstack-infra | 21:55 | |
| *** jamesmcarthur has quit IRC | 21:56 | |
| sdague | fungi: jog0's fun graph is definitely a winner here - http://status.openstack.org/elastic-recheck/ | 21:58 |
| jeblair | mordred, fungi: good news and bad news. bad: the pbr change still failed gate-requirements-integration-dsvm good: it doesn't seem to be required for most changes | 21:59 |
| jeblair | mordred, fungi: so i think the current status is that openstack/requirements is hosed, but nothing else is. | 22:00 |
| portante | jeblair: is that one "timed out" doing pip install from pypi.openstack.org? | 22:01 |
| *** loq_mac has quit IRC | 22:01 | |
| *** loq_mac has joined #openstack-infra | 22:01 | |
| portante | is there a bug number for that that I can recheck against? | 22:01 |
| mordred | jeblair: and we force-landed the django change, yeah? | 22:02 |
| jeblair | mordred: yes | 22:02 |
| mordred | jeblair: so, for the moment, the fact that the requirements job is hosed is bad but not project-wide-critical | 22:02 |
| *** harlowja has quit IRC | 22:02 | |
| portante | the pbr failure for swift hit this one: http://logs.openstack.org/13/47713/7/check/check-swift-dsvm-functional/db9cc6c/console.html | 22:02 |
| openstackgerrit | Sean Dague proposed a change to openstack-infra/devstack-gate: allow configdrive vs. metadata server to be configured https://review.openstack.org/59303 | 22:03 |
| jeblair | mordred: yes, though i don't expect that to be the case for very long. | 22:03 |
| *** mfer has quit IRC | 22:03 | |
| mordred | me either | 22:03 |
| jeblair | portante: i don't know the answer to your question. perhaps mordred does. | 22:04 |
| jeblair | #status ok | 22:05 |
| *** ChanServ changes topic to "Discussion of OpenStack Project Infrastructure | Docs http://ci.openstack.org/ | Bugs https://launchpad.net/openstack-ci | Code https://git.openstack.org/cgit/openstack-infra/" | 22:05 | |
| ekarlso | any devstackers here ? http://paste.ubuntu.com/6522113/ | 22:05 |
| portante | jeblair, mordred, so then I'll file another bug the above pbr failure | 22:05 |
| ekarlso | is my local.conf, but it produces a " = " line in my nova.conf ?! | 22:05 |
| portante | filed: https://bugs.launchpad.net/devstack/+bug/1257933 | 22:08 |
| uvirtbot | Launchpad bug 1257933 in devstack "pbr fails to install during check-swift-dsvm-functional job" [Undecided,New] | 22:08 |
| *** mfer has joined #openstack-infra | 22:08 | |
| *** dims has quit IRC | 22:10 | |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Make havana devstack/tempest jobs non-voting https://review.openstack.org/60092 | 22:11 |
| *** dcramer_ has quit IRC | 22:13 | |
| *** pabelanger has quit IRC | 22:15 | |
| devananda | reviewday question -- are there separate per-project pages, or just the one giant shared page? | 22:15 |
| pleia2 | devananda: giant shared page, but we've been thinking about doing some per-project motifications now that it's so huge | 22:17 |
| pleia2 | you know, in our free time :) | 22:17 |
| devananda | pleia2: awesome. so, like, tomorrow, right? :) | 22:17 |
| pleia2 | maybe by j-1 | 22:18 |
| pleia2 | :D | 22:18 |
| devananda | hehe | 22:18 |
| pleia2 | actually, I should create a bug for it so it's more formally on my radar | 22:18 |
| *** rongze has joined #openstack-infra | 22:19 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add support for batch_tasks. https://review.openstack.org/57469 | 22:20 |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add seealso to batch_tasks from promoted_build. https://review.openstack.org/57473 | 22:20 |
| stevebaker | I wonder if I could get some eyeballs on this? https://review.openstack.org/#/c/60084/ | 22:20 |
| openstackgerrit | David Ostrovsky proposed a change to openstack-infra/config: Gerrit-2.8: Activate new change screen per default https://review.openstack.org/60094 | 22:21 |
| openstackgerrit | James E. Blair proposed a change to openstack-infra/config: Zuul: set gear log level to INFO https://review.openstack.org/60095 | 22:21 |
| Alex_Gaynor | sdague: If there's any particular django regerssion I could be hlpi9ng with, let me know | 22:21 |
| _david_ | Gerrit 2.8: i wonder if we want to extend Gerrit TopMenu? As of Gerrit 2.8, it is possible to extend Gerrit Top Menu, We could put there Zuul of whatever we want | 22:22 |
| *** loq_mac has quit IRC | 22:23 | |
| Alex_Gaynor | cc: mordred, fungi ^ :-) | 22:23 |
| _david_ | Example of gerrit Top-Menu extension in LibreOffice project: https://gerrit-test.libreoffice.org/#/c/6676/ Buildbot coming from our buildbot plugin | 22:24 |
| fungi | Alex_Gaynor: i suspect there are now extensive logs archived from our accidentally running tempest with django 1.6 today | 22:24 |
| Alex_Gaynor | Anyone have a link to one? :-) | 22:25 |
| fungi | i'll dig one out | 22:25 |
| *** dangers is now known as danger_fo_away | 22:25 | |
| Alex_Gaynor | thanks | 22:26 |
| *** harlowja has joined #openstack-infra | 22:27 | |
| fungi | Alex_Gaynor: i guess for starters, the horizon crowd are tracking horizon support for django 1.6 at https://blueprints.launchpad.net/horizon/+spec/django-1point6 | 22:27 |
| jeblair | _david_: the libreoffice logo owerwrites the search box if your screen is <1430 pixels | 22:27 |
| fungi | Alex_Gaynor: it sounds like the move to uncap it in requirements was probably premature | 22:27 |
| *** rongze has quit IRC | 22:28 | |
| Alex_Gaynor | seems so | 22:28 |
| _david_ | jeblair, yeah ,-) a known issue | 22:28 |
| _david_ | jeblair, back to top menu extension: do you think it would be helpful? | 22:28 |
| *** loq_mac has joined #openstack-infra | 22:28 | |
| _david_ | jeblair, to have it, we would need a trivial gerrit plugin, but it would open a door for future extensions as well, ... | 22:29 |
| jeblair | _david_: i don't think it's a priority. | 22:30 |
| clarkb | I am mostly back now (eating a late lunch but in front of a computer) | 22:31 |
| _david_ | jeblair, ok, just to let you know: you could put ACL logic on menu entries, so that only some groups can see some entries. | 22:31 |
| clarkb | we are reverting the pbr integration with wheels change? | 22:31 |
| jeblair | clarkb: it failed requirements integration test. | 22:32 |
| clarkb | jeblair: is that what was causing the d2to1 madness? | 22:32 |
| jeblair | clarkb: no idea. mordred: ? | 22:32 |
| Alex_Gaynor | fungi: do you know if there are issues besides the one review request that was merged? | 22:32 |
| mordred | clarkb: I do not know. none of the symptoms make any sense to me- I need to dive in much more deeply | 22:33 |
| _david_ | jeblair, mordred what is the status with WIP plugin and needed core patch for 2.8? | 22:33 |
| clarkb | mordred: ok | 22:33 |
| mordred | zaro: ^^ | 22:33 |
| clarkb | is there anything around the gate that I should be looking at or doing? or am I good to go back to reviews? | 22:33 |
| *** rpodolyaka1 has quit IRC | 22:33 | |
| *** gyee has quit IRC | 22:33 | |
| _david_ | if we don't want/need it i'm fine with it | 22:34 |
| clarkb | in other news, last night the author of this change https://review.openstack.org/#/c/59903/ says they did not receive any email when jenkins and tom commented on that change, but do receive emails when they comment on changes by other authors. Any idea why that may happen? | 22:34 |
| clarkb | _david_: the needed core patch merged the other day so will be in 2.9. I think our hope is that 2.9 will release relatively quickly and we can use that | 22:35 |
| fungi | Alex_Gaynor: so, the issue was that we spontaneously started trying to use django 1.6 in tempest tests, if that's what you mean by "the one review request that was merged". the end result was a lot of 500 errors from the dashboard as seen in http://logs.openstack.org/00/60000/2/check/check-tempest-dsvm-full/c80ec07/console.html | 22:35 |
| clarkb | _david_: otherwise we might use a build off of master or a minimal fork of 2.8 (but really want to use 2.9) | 22:35 |
| Alex_Gaynor | fungi: is there a way to see the horizon logs for the 500s in that log? | 22:35 |
| clarkb | Alex_Gaynor: yes http://logs.openstack.org/00/60000/2/check/check-tempest-dsvm-full/c80ec07/logs/screen-horizon.txt.gz | 22:36 |
| Alex_Gaynor | clarkb: danke | 22:36 |
| _david_ | clarkb, i guess we are talking about different patches? you mean zaro's change owner patch. It is entirely unrelated to my WIP plugin and needed core patch to make it work, that was rejected by mfick | 22:37 |
| openstackgerrit | Clint "SpamapS" Byrum proposed a change to openstack/requirements: Add markdown to requirements https://review.openstack.org/60100 | 22:37 |
| clarkb | fungi: Alex_Gaynor looking at https://review.openstack.org/#/c/58561/1 I don't understand why we couldn't get that stuff in before we do django 1.6 | 22:37 |
| clarkb | _david_: oh I thought your change for WIP is already in, I am talking about different things | 22:38 |
| clarkb | sory | 22:38 |
| _david_ | clarkb, np, yeah the plugin is in (because /me have a +2 on gerrit-review for this plugin) | 22:38 |
| *** ryanpetrello has quit IRC | 22:39 | |
| openstackgerrit | Jaroslav Henner proposed a change to openstack-infra/jenkins-job-builder: Add seealso to batch_tasks from promoted_build. https://review.openstack.org/57473 | 22:40 |
| *** loquacities has joined #openstack-infra | 22:42 | |
| *** loq_mac has quit IRC | 22:42 | |
| *** fifieldt has joined #openstack-infra | 22:42 | |
| *** dkliban_ has joined #openstack-infra | 22:44 | |
| openstackgerrit | Douglas Mendizabal proposed a change to openstack-infra/config: Initial configuration for python-barbicanclient https://review.openstack.org/60101 | 22:44 |
| *** miqui has quit IRC | 22:45 | |
| *** wenlock has quit IRC | 22:47 | |
| clarkb | mordred: shoudl I trigger the python26 mirror job so that we can see if a centos 6.5 dir is created? | 22:48 |
| *** Ryan_Lane has joined #openstack-infra | 22:49 | |
| *** rossella_s has joined #openstack-infra | 22:49 | |
| *** sarob has joined #openstack-infra | 22:50 | |
| SpamapS | 1386197303.447 15015 192.168.122.1 TCP_MISS_ABORTED/000 0 GET http://pypi.openstack.org/openstack/pbr/ - HIER_NONE/- - | 22:51 |
| *** loquacities has quit IRC | 22:51 | |
| SpamapS | 1386197318.458 15007 192.168.122.1 TCP_MISS_ABORTED/000 0 GET http://pypi.openstack.org/openstack/ - HIER_NONE/- - | 22:51 |
| SpamapS | is pypi.openstack.org just down? | 22:51 |
| *** loq_mac has joined #openstack-infra | 22:51 | |
| SpamapS | 1386197323.436 4974 192.168.122.1 TCP_MISS/503 3723 GET http://pypi.openstack.org/openstack/pbr/ - HIER_DIRECT/2001:4800:7813:516:3bc3:d7f6:ff05:4882 text/html | 22:51 |
| clarkb | SpamapS: can you link to the job that resulted in that | 22:53 |
| clarkb | or is this local to you? | 22:53 |
| *** yjiang5_1 has quit IRC | 22:53 | |
| clarkb | nevermind 192.168/16 would be local to you | 22:53 |
| SpamapS | clarkb: thats just me doing a 'tox -r -epep8' in openstack/tripleo-image-elements through my proxy with PIP_INDEX_URL=http://pypi.openstack.org/openstack/ | 22:53 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Fix CSV output https://review.openstack.org/60102 | 22:53 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Remove text blobs from CSV output https://review.openstack.org/60103 | 22:53 |
| clarkb | seems up for me | 22:53 |
| SpamapS | clarkb: yeah it isn't doing it now | 22:54 |
| *** sarob has quit IRC | 22:54 | |
| * fungi blames internets | 22:54 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=314&rra_id=all is empty | 22:54 |
| *** rongze has joined #openstack-infra | 22:55 | |
| lifeless | clarkb: can I get an APRV on https://review.openstack.org/#/c/60058/ ? | 22:55 |
| *** dkranz has joined #openstack-infra | 22:55 | |
| clarkb | lifeless: sure; hwoever, will we have to revert in an hour when paramiko is happy? | 22:56 |
| lifeless | clarkb: no | 22:56 |
| lifeless | clarkb: weeks probably | 22:56 |
| clarkb | ok | 22:56 |
| lifeless | clarkb: the patch set upstream is under review | 22:56 |
| *** loq_mac has quit IRC | 22:56 | |
| lifeless | https://github.com/paramiko/paramiko/issues/16 | 22:56 |
| clarkb | I see thanks | 22:56 |
| clarkb | (change has been approved) | 22:56 |
| SpamapS | clarkb: probably related to this though: https://bugs.launchpad.net/openstack-ci/+bug/1257905 | 22:57 |
| uvirtbot | Launchpad bug 1257905 in openstack-ci "mirror building broken requirements testing" [Undecided,New] | 22:57 |
| SpamapS | clarkb: I pasted the lines with data in the comments in hopes it helps :) | 22:57 |
| clarkb | SpamapS: it could be however that was a failure local to test slaves and had nothing to do with pypi.o.o | 22:58 |
| *** rfolco_ has quit IRC | 22:58 | |
| clarkb | (could be because I think those local test slaves use apache now) | 22:58 |
| *** Ryan_Lane has quit IRC | 22:58 | |
| clarkb | mordred: is it possible that your apache vhost needs updating? | 22:58 |
| openstackgerrit | A change was merged to openstack-infra/config: Make python3.3 nonvoting for reviewstats. https://review.openstack.org/60058 | 22:58 |
| *** bpokorny1 has quit IRC | 22:59 | |
| jeblair | clarkb: what are you talking about? | 22:59 |
| jeblair | clarkb: what test slaves run apache? | 22:59 |
| clarkb | jeblair: the pbr integration test slaves serve the local mirror out of apache iirc | 22:59 |
| lifeless | clarkb: thanks | 23:00 |
| clarkb | there was some reason mordred stopped using file:/// URIs for the local mirror. I would need to look at git logs | 23:00 |
| *** rongze has quit IRC | 23:00 | |
| lifeless | the file:// codepaths are not the same as the http:// codepaths | 23:00 |
| lifeless | we were not getting representative behaviour out of them | 23:00 |
| mordred | that is correct. it turns out that pip does not do case insensitive searching on wheels via file:/// urs | 23:00 |
| mordred | urls | 23:00 |
| mordred | but yes. lifeless said it more succinctly | 23:00 |
| *** sarob has joined #openstack-infra | 23:00 | |
| *** nati_ueno has joined #openstack-infra | 23:01 | |
| jeblair | clarkb: okay, so what failure do you think is related to that? | 23:01 |
| *** thomasem has quit IRC | 23:01 | |
| clarkb | jeblair: SpamapS thought his pypi.o.o failures may be related to bug 1257905. I only think that is possible if the apache on pypi.o.o and on the test slaves is exhibiting a common problem | 23:02 |
| uvirtbot | Launchpad bug 1257905 in openstack-ci "mirror building broken requirements testing" [Undecided,New] https://launchpad.net/bugs/1257905 | 23:02 |
| clarkb | because 1257905 is occuring without talking to pypi.o.o | 23:03 |
| jeblair | clarkb: i thought 1257905 was related to case mismatches, not connection errors. | 23:04 |
| SpamapS | true, sounds like coincidence and not truly related | 23:04 |
| SpamapS | (I saw pbr problems in both and thought too much) | 23:04 |
| clarkb | jeblair: that a theory anyways. case mismatches don;t happen when wheels are not enabled (fwiw I don't think spamaps problem is related at all, I am just trying to sort out if they could be) | 23:05 |
| jeblair | i believe portante reported some pypi.o.o connection problems earlier. perhaps there are rackspace network glitches. | 23:05 |
| jeblair | clarkb: but then, it may have been on a requirements job, so i guess all bets are off. | 23:05 |
| portante | jeblair: manifested as pbr install failures | 23:06 |
| fungi | i wonder whether we wouldn't be better served by normalizing all requirements lists to match the capitalization used on pypi, so that file:// urls will "just work" to local trees | 23:06 |
| SpamapS | portante: same for me. | 23:06 |
| fungi | (and then enforcing that capitalization in requirements gating) | 23:07 |
| *** _david_ has quit IRC | 23:08 | |
| *** rfolco_ has joined #openstack-infra | 23:10 | |
| jeblair | clarkb: i've been unable to replicate the keypair leak on rax with either a successfull or failed image build | 23:11 |
| jeblair | clarkb: hopefully the new logging will help pin it down further | 23:11 |
| clarkb | jeblair: ok, thanks | 23:12 |
| clarkb | zaro: https://review.openstack.org/#/c/58140/1 is that something we ran into with our jenkins? | 23:14 |
| clarkb | curious to know if I approve it before harshar's testing will we have any idication it works? | 23:15 |
| *** Mithrandir has quit IRC | 23:15 | |
| *** oubiwann_ has quit IRC | 23:15 | |
| *** datsun_F40PH has quit IRC | 23:16 | |
| *** mfer has quit IRC | 23:16 | |
| *** ArxCruz has quit IRC | 23:17 | |
| *** rfolco_ has quit IRC | 23:18 | |
| mordred | fungi: well... | 23:18 |
| mordred | funny you say that | 23:18 |
| mordred | fungi: global-requirements is supposed to be capitalization aligned | 23:18 |
| pleia2 | clarkb: updated the infra-pubs-cleanup etherpad with the last commit before delete everything for the new branches in gerrit (no rush, whenever you have some time) | 23:19 |
| mordred | but that doesn't mean that transitives are | 23:19 |
| fungi | oh! very good point | 23:19 |
| fungi | and sad | 23:19 |
| clarkb | pleia2: ok, I am trying to get through code review now. Can poke at that after | 23:19 |
| *** Mithrandir has joined #openstack-infra | 23:19 | |
| *** rfolco_ has joined #openstack-infra | 23:20 | |
| fungi | mordred: so basically, pip's use of local filesystem urls is broken except for very small/carefully constructed dependency chains? | 23:20 |
| jeblair | yay, zuul is doing an 8x merge | 23:20 |
| *** vipul is now known as vipul-away | 23:21 | |
| fungi | mordred: was there a suggestion that pip 1.5 (2.0?) is going to fix that by analyzing local file trees and doing similar case/symbol munging to match package names? | 23:21 |
| * fungi has some vague memory that got mentioned | 23:21 | |
| jeblair | clarkb: i cleaned up rax keypairs | 23:21 |
| fungi | jeblair: clarkb: over the weekend i did the same for the rax openstackci account | 23:22 |
| fungi | since i hit the keypair quota there | 23:22 |
| clarkb | could the bug be in novacleint if we are seeing it in places nodepool doesn't touch? | 23:22 |
| jeblair | fungi: oh fun! | 23:22 |
| fungi | about half were from very old devstack slaves and the other half seemed to be from aborted runs of the launch.py script | 23:23 |
| jeblair | clarkb: launch-node is pretty close to nodepool in behavior here | 23:23 |
| *** gyee has joined #openstack-infra | 23:23 | |
| *** michchap has joined #openstack-infra | 23:24 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Fix pep8 tests https://review.openstack.org/60017 | 23:24 |
| jeblair | with the logging in place, we'll be able to determine whether nodepool is actually requesting the deletion the next time we see a leaked key | 23:24 |
| clarkb | sdague: mtreinish: is https://review.openstack.org/#/c/58605/ something we should WIP until it can pass tests regularly? | 23:25 |
| *** pabelanger_ has quit IRC | 23:26 | |
| *** pabelanger has joined #openstack-infra | 23:26 | |
| *** esker has joined #openstack-infra | 23:27 | |
| *** rfolco_ has quit IRC | 23:27 | |
| zaro | clarkb: about 58140, no we didn't run into that because i don't think we have that setting enabled on any of our slaves. | 23:27 |
| clarkb | zaro: ok thanks. I will wait for hashar to respond then | 23:28 |
| *** dkliban_ has quit IRC | 23:28 | |
| *** rossella_s has quit IRC | 23:29 | |
| *** esker has quit IRC | 23:32 | |
| clarkb | I just approved https://review.openstack.org/#/c/58783/ it had a bunch of +2s but no approval (looked safe to me) | 23:32 |
| *** rfolco_ has joined #openstack-infra | 23:32 | |
| *** weshay has quit IRC | 23:36 | |
| *** Mithrandir has quit IRC | 23:37 | |
| *** Mithrandir has joined #openstack-infra | 23:37 | |
| *** ryanpetrello has joined #openstack-infra | 23:37 | |
| openstackgerrit | A change was merged to openstack-infra/jenkins-job-builder: ModuleRegistry now caches entry points lookup https://review.openstack.org/58783 | 23:38 |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/config: Don't assume substitation for a non-template job https://review.openstack.org/60110 | 23:38 |
| *** rfolco_ has quit IRC | 23:38 | |
| russellb | lifeless: clarkb ^ what we were just discussing in -dev | 23:41 |
| lifeless | russellb: already voted with my puny +1. | 23:41 |
| *** vipul-away is now known as vipul | 23:41 | |
| clarkb | russellb: and it has my +2 too, I will give other cores a chance to review it though that change is simple enough Iwill just approve in a bit | 23:41 |
| russellb | thanks guys | 23:42 |
| *** Ryan_Lane has joined #openstack-infra | 23:42 | |
| *** rfolco_ has joined #openstack-infra | 23:42 | |
| *** paul-- has joined #openstack-infra | 23:42 | |
| clarkb | jeblair: https://review.openstack.org/#/c/59046/ is ready to have your -2 removed | 23:44 |
| * fungi disappears again to head home. bbiab | 23:45 | |
| *** Ryan_Lane has quit IRC | 23:46 | |
| openstackgerrit | A change was merged to openstack-infra/config: Add ironic puppet module to stackforge https://review.openstack.org/59247 | 23:47 |
| openstackgerrit | A change was merged to openstack-infra/config: Don't assume substitation for a non-template job https://review.openstack.org/60110 | 23:48 |
| openstackgerrit | A change was merged to openstack-infra/config: Changing the output directory that gets picked up for the API Quick Start https://review.openstack.org/59556 | 23:48 |
| *** salv-orlando has quit IRC | 23:49 | |
| *** yamahata_ has joined #openstack-infra | 23:50 | |
| openstackgerrit | lifeless proposed a change to openstack-infra/reviewstats: Use six to handle cpickle. https://review.openstack.org/60114 | 23:50 |
| *** mrodden has quit IRC | 23:50 | |
| openstackgerrit | Russell Bryant proposed a change to openstack-infra/reviewstats: Add --csv-rows option https://review.openstack.org/60115 | 23:50 |
| *** ryanpetrello has quit IRC | 23:51 | |
| clarkb | sdague: re https://review.openstack.org/#/c/59289/ maybe tomorrow we can sort out how to land that change and its siblings? I am happy to babysit that | 23:51 |
| *** thedodd has quit IRC | 23:52 | |
| *** mriedem has quit IRC | 23:52 | |
| *** loq_mac has joined #openstack-infra | 23:54 | |
| openstackgerrit | A change was merged to openstack-infra/reviewstats: Remove arata and devananda from TripleO core. https://review.openstack.org/60050 | 23:56 |
| *** rongze has joined #openstack-infra | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!