Thursday, 2015-03-05

« Wednesday, 2015-03-04 Index Friday, 2015-03-06 »
greghaynes:/00:00
jogoclarkb: oh right00:00
greghaynesCan we not just run the devstack bits that install packages durin the dib build00:00
jeblairclarkb: how certain are you that they KeyError still happened after the full removal?  if it only shows up in the log twice, i wonder... i ask because it's not clear how it would end up missing from part of the data structure otherwise00:00
greghaynesor we want to test the dependency logic too I guess00:00
clarkbgreghaynes: then we won't have a cache00:00
mordredwe do00:00
clarkbjeblair: can you grep for the Exception in main loop message?00:01
jeblairclarkb: i did and found the KeyError twice00:01
clarkbjeblair: hrm that is what I see as well00:02
clarkblet me reread irc logs00:02
jeblairclarkb: maybe it was in the process of recovering from your second fix?  it looks like it was restarting the managers around that time, but may have stopped emitting the KeyError00:03
jogogreghaynes: also based on http://logs.openstack.org/28/151128/5/check/check-tempest-dsvm-neutron-full/00627d4//logs/devstacklog.txt.gz00:03
*** ZZelle has quit IRC00:03
jogoinstalling packages is pretty quick00:03
clarkbjeblair: according to my reading of irc logs + nodepool logs first keyerror occured before removing all entries and second occurred after00:03
jogobuilding wheels is ~25% of the time though00:04
*** baoli has quit IRC00:04
clarkb23:31:20     clarkb | ok I have more properly fixed the config, lets see if that shakes some nodes loose <- is when I removed the label and providers too00:04
greghaynesyea00:04
jeblairclarkb: based on the manager restarts though, i think the config reload may not have taken effect until after that00:04
greghaynesjogo: and I think ~80% of that wheel time is numpy + lxml if its anything like tripleo00:04
*** jlvillal_ has joined #openstack-infra00:04
jogogreghaynes: and caching files will mean less trips to the network which will increase reliability (we see rax mirrors break often)00:04
clarkbjeblair: gotcha, I was just being impatient then?00:04
jogogreghaynes: most likely00:04
clarkbthen I restarted ~4 minutes later00:04
jeblairclarkb: understandably so.  :)  but i think that means that the fix for this is the config file syntax check00:05
*** jlvillal_ has quit IRC00:05
clarkbjeblair: +100:05
clarkbjogo: your example ran in rax00:05
clarkbjogo: we are caching packages properly in rax00:05
*** hdd_ has quit IRC00:06
clarkbjeblair: also it was about 4.5 minutes between main loop exceptions so I likely jumped the gun00:06
clarkbjeblair: you have convinced me :)00:06
*** Rockyg has quit IRC00:07
*** markvoelker has quit IRC00:07
jogoclarkb: what about Get:4 http://mirror.rackspace.com/ubuntu/ trusty/main libcroco3 amd64 0.6.8-2ubuntu1 [82.4 kB] ?00:09
clarkbjogo: it will still grab anything that is updated00:09
clarkbjogo: or any new dependencies00:09
clarkbgreghaynes: where is that boot qcow2 script stuff of yours? I finally have a real excuse to use it00:09
greghaynesclarkb: haha00:09
*** hodos|2 has quit IRC00:10
jogoclarkb: that is a lot of new packages since the caching ...00:11
*** ZZelle has joined #openstack-infra00:11
*** ddieterly has quit IRC00:11
*** camunoz has quit IRC00:11
*** bdpayne has quit IRC00:11
*** camunoz has joined #openstack-infra00:12
clarkbjogo: its also possible snapshots are broken too but they don't have chroot issues like dib00:13
greghaynesclarkb: I wonder if https://review.openstack.org/#/c/139064/ is the race issues you were seeing00:13
jeblairclarkb: which image has the initramfs problem?00:14
clarkbjeblair: devstack-trusty00:14
clarkbjeblair: its the qcow2 in my homedir on nodepool.o.o00:14
jeblairclarkb: and did you delete most recent from all 5 hpclouds?00:15
*** prad has quit IRC00:15
clarkbjeblair: I did00:15
*** r-daneel has quit IRC00:15
clarkbjeblair: but not before it started builds in them00:16
openstackgerritSteve Baker proposed openstack-infra/project-config: Create and upload heat test images to tarballs  https://review.openstack.org/15968600:16
clarkbso I think it may need a single build fail pass to use the new image00:16
*** thedodd has quit IRC00:16
*** carl_baldwin_ has joined #openstack-infra00:17
*** bhunter71 has quit IRC00:17
jeblairClientException: Unknown Error (HTTP 500)00:18
jeblairsigh00:18
jeblairseeing a lot of those in hpcloud now00:18
*** carl_baldwin has quit IRC00:19
*** carl_baldwin_ is now known as carl_baldwin00:19
clarkbthat was the floating ip leak before00:19
clarkbits possible that this is really making the leak large00:20
clarkbjeblair: try nova show on a node that does that00:20
clarkbthat was the other thing I was doing to track this stuff down00:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Sorted Package Dependencies.  https://review.openstack.org/16149700:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Removed "Do No Evil" jshint and replaced with eslint.  https://review.openstack.org/16149800:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:no-undefined  https://review.openstack.org/16149900:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:indent  https://review.openstack.org/16150000:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:key-spacing  https://review.openstack.org/16150100:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:no-space-before-semi  https://review.openstack.org/16150200:20
*** wenlock has quit IRC00:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:no-underscore-dangle  https://review.openstack.org/16150300:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:strict  https://review.openstack.org/16150400:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:no-console  https://review.openstack.org/16150500:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:space-unary-ops  https://review.openstack.org/16150600:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:consistent-return  https://review.openstack.org/16150700:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:no-shadow  https://review.openstack.org/16150800:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:space-infix-ops  https://review.openstack.org/16150900:20
openstackgerritMichael Krotscheck proposed openstack-infra/storyboard-webclient: Enabled eslint:no-use-before-define  https://review.openstack.org/16151000:20
jeblairclarkb: they delete instantly00:23
jeblairclarkb: it's the create server call itself that is 500ing00:23
jeblairso there's nothing to see in nova00:23
clarkbjeblair: ok then thats not the boot problems from before00:23
*** annegentle has joined #openstack-infra00:24
clarkbjeblair: those would go ACTIVE and the console would show the unable to mount /00:24
jeblairclarkb: oh yeah, i didn't think it was00:24
jeblairclarkb: those manifested as ssh timeouts in nodepool, i believ00:24
clarkbjeblair: the other thing to check is `nova floating-ip-list | wc -l`00:24
clarkbjeblair: 600 + 4 for the table format is our limit there iirc00:24
jeblair45900:24
clarkbok so not floating ips00:24
openstackgerritSumit Naiksatam proposed openstack-infra/project-config: Create experimental job for group-based-policy  https://review.openstack.org/16151100:24
jeblairclarkb, mordred: i mentioned to the hpcs noc that we are seeing many 500 errors on server create00:25
*** tsg_ has quit IRC00:25
jeblairclarkb: still seeing quite a few "OverLimit: Maximum number of ports exceeded"00:26
jeblairclarkb: what does that mean?00:26
*** bhunter71 has joined #openstack-infra00:26
clarkbjeblair: we also have a limit of 600 neutron ports. Each neutron interface is a port. So in hpcloud thats one port per node. try `neutron port-list -c id -c device_id` to get a listing of port id to host ids00:26
*** hamzy_ has joined #openstack-infra00:27
clarkbjeblair: also wc -l that and limit should be 600 + 400:27
jeblairclarkb: is there a venv with neutronclient installed?00:27
jesusaurusoh, you might be leaking ports rather than leaking floating-ips00:27
*** hamzy_ has quit IRC00:28
clarkbjesusaurus: I have on in my homedir on puppetmaster.o.o /home/clarkb/venv/bin/neutron00:28
clarkber jeblair ^00:29
jesusaurusif you are leaking neutron ports, http://paste.openstack.org/show/188317/ is a little script I wrote to clear them out00:29
jeblairclarkb: 60700:29
jeblairhow is it possible we are leaking neutron ports?00:29
clarkbjeblair: ok, that means we are using our quota there00:29
clarkbwell I am not quite sure we are00:30
clarkbjeblair: I think we may be leaking nodes00:30
jeblairclarkb: as close as i could get running two commands, we have 497 nova instances and 599 neutron ports00:31
clarkbjeblair: huh, ok then I think we are leaking neutron ports.00:31
clarkbjeblair: we probably need to cross check the device_id from port listing with nova instance uuid, filter out the special ones for routers and dhcp then maybe delete the rest?00:31
clarkbjeblair: I also have no idea how we are leaking those00:32
jeblairmordred: ^00:32
jeblairclarkb, mordred: are we using east or west?00:32
jeblairthe noc would like to know00:33
*** ghostpl_ has joined #openstack-infra00:33
jeblairclarkb: fwiw, i think we just got a big batch of working hpcloud nodes00:33
clarkbjeblair: east00:34
clarkbjeblair: well at least tehre is some good news00:34
*** carl_baldwin has quit IRC00:34
*** jlvillal has joined #openstack-infra00:34
jeblairokay, it's past EOD for me.  i'm going to wish nodepool good luck with hpcloud and head out.00:36
*** annegentle has quit IRC00:36
clarkbhave a good one, I think things are mostly happy again00:37
mordredjeblair: east00:37
mordredjeblair: havea  good night!00:37
mordredclarkb: I had a new/different idea on the fix for the ansible/puppet/project-config sequence thing00:38
*** ghostpl_ has quit IRC00:38
ianwclarkb: when you have a second, could you pastebin the full nodepool.yaml config file from host (templating gets in the way of using the .erb file)00:39
*** stevemar has quit IRC00:39
clarkbmordred: ok I am mostly semi listening00:40
clarkbianw: it has sensitive data in it so not really00:40
clarkbianw: just replace the erb stuff with strings00:40
openstackgerritMonty Taylor proposed openstack-infra/system-config: Drive actions around repo creation from ansible  https://review.openstack.org/16151800:41
mordredclarkb: ^^00:41
mordredclarkb: there it is00:41
*** devlaps has quit IRC00:41
*** crc32 has quit IRC00:41
*** mrodden has quit IRC00:41
clarkbmordred: the problem with that is it requires ansible to use our puppet right?00:42
*** david-lyle has quit IRC00:43
openstackgerritMerged openstack-infra/nodepool: Fix exception in dib image cleanup  https://review.openstack.org/16147300:43
mordredclarkb: only if you're using it to drive manage-projects and create-cgitrepos00:43
openstackgerritMerged openstack-infra/system-config: Fix nodepool.yaml whitespace  https://review.openstack.org/16146200:44
ianwclarkb: yeah, doh00:45
mordredclarkb: I'm not wed to it - just seemed like a fairly simple option00:45
*** mrodden has joined #openstack-infra00:47
jogoclarkb: re caching apt packages in rax and HP, I am seeing the same amount of data being downloaded from apt i nboth00:47
jogoclarkb: I think greghaynes has an idea why00:48
clarkbjogo: what was the idea?00:49
greghayneshrm?00:49
greghaynesideas?00:49
greghaynesIm guessing youre referring to the fact that dib clears out the apt cache00:50
clarkbrax images aren't built with dib00:50
greghayneswell, that would be odd if dib was messing that up then ;)00:50
jogoclarkb greghaynes: ahh hrm, so something else is wrong there I guess00:51
*** otter768 has joined #openstack-infra00:51
*** jerryz_ has joined #openstack-infra00:52
clarkbmordred: jeblair I have confirmed that local kvm can boot the image that seemed to give hpcloud trouble00:52
mordredclarkb: awesome00:52
clarkbmordred: any ideas why http://paste.openstack.org/show/188214/ may happen in hpcloud?00:52
greghaynesclarkb: d3.sh worked?00:53
clarkbmordred: also if you want to grab the image that is broken its in my homedir on nodepool.o.o maybe you will have better luck reproducing the fail00:53
clarkbgreghaynes: up00:53
clarkbgreghaynes: I just shoved m qcow2 into the images/ dir and that worked fine00:53
mordredclarkb: uhm .. NO00:53
greghaynesclarkb: nice!00:53
clarkbmordred: boo :P00:53
mordredclarkb: that looks very screwed up00:54
clarkbis there a better way of getting the ip addr for a virsh node than checking arp?00:54
mordredtteggel: ^^ when you wake up ...00:54
clarkbmordred: the only thing I can think of is maybe the file that was uploaded got corrupted? but don't we check md5sum?00:54
clarkbmordred: also it would've had to get corrupted on multiple uploads which seems odd00:54
clarkbmordred: also check out greghaynes' d3.sh00:55
*** tnovacik has quit IRC00:55
*** otter768 has quit IRC00:56
mordredwhere do I find d3.sh?00:56
greghayneshttps://github.com/greghaynes/dib-dev-deploy00:57
clarkbmordred: https://github.com/greghaynes/dib-dev-deploy00:57
clarkbgreghaynes: though it looks like you don't preserve the orinal image00:57
clarkbwhich should be fine for a thing with dev in its name00:57
greghayneshrm?00:57
mordrednice00:58
*** chuckC_ has quit IRC00:58
clarkbgreghaynes: you make the source qcow the backing image file00:58
clarkbgreghaynes: rather than making a copy00:58
greghaynesah, well thats an easy fix00:58
*** Ryan_Lane has quit IRC00:58
clarkbso if I do stuff in the VM the image will change00:58
greghaynesThis was just me getting mad when I had to boot an image by hand, not really planned out ;)00:58
clarkbwhich meh its fine00:58
clarkbtotally works which is cool00:59
clarkbbecause the worst bits of virsh are defining domains00:59
greghaynesyep01:00
*** bodepd has quit IRC01:01
clarkbok I am really going to step away from the computer now01:02
clarkbI tried to reproduce boot failures and failed01:02
clarkblikely not going to make any additional headway today01:02
clarkbmordred: though before I go, any thoughts on how we leak neutron ports?01:03
*** yamamoto has quit IRC01:04
mordredclarkb: no - but I'll try to hammer on that concept in the morning01:05
*** mayurig has quit IRC01:06
*** carl_baldwin has joined #openstack-infra01:07
*** markvoelker has joined #openstack-infra01:08
*** SumitNaiksatam has quit IRC01:11
*** kmartin has quit IRC01:11
*** ZZelle_ has quit IRC01:12
*** markvoelker has quit IRC01:13
*** annegentle has joined #openstack-infra01:15
*** markvoelker has joined #openstack-infra01:15
StevenKpleia2: You have a CentOS machine handy, or I should dust off a chroot/VM?01:15
*** salv-orlando has quit IRC01:16
pleia2StevenK: just brought one up on hpcloud, trying to figure out how to apt-cache search with yum now ;)01:16
*** annegentle has quit IRC01:16
mordredpleia2: "yum search foo"01:16
pleia2how reasonable! thanks01:16
mordredpleia2: it's almost like they wrote one tool that actually does all the things01:16
*** annegentle has joined #openstack-infra01:16
asselingreghaynes, failed again. not sure what the error is. did I accidentally ctrl-c? http://paste.openstack.org/show/188362/01:17
clarkbmordred: because I have a hard time not working, in my homedir on puppetmaster are two files, hpcloud-nodes.txt and hpcloud-ports.txt01:20
mordredclarkb: heh01:20
clarkbmordred: should be able to use that data to figure out the leaked ports01:20
mordredclarkb: well, you could be watching duke whoop up on wake forest on espn3/watchespn01:20
clarkbI just need to clean it up01:20
*** markvoelker has quit IRC01:20
pleia2StevenK: heh, so it's not in stock centos7, so I enabled EPEL01:21
asselingreghaynes, nevermind...sorry I redirected stdout to a log file, but stderr was still going to a terminal. a bit confusing...01:21
pleia2StevenK: zanata-python-client.noarch : Python Client for Zanata Server01:21
pleia2x_x01:21
pleia2doesn't seem to have the java client01:21
*** Daisy_ has joined #openstack-infra01:21
greghaynesasselin: ah. I was very confused ;)01:21
pleia2StevenK: but since there's one for fedora, it may be possible to just bring that rpm over01:22
StevenKpleia2: Possibly.01:22
StevenKThe purist in me wants to keep going to have it on Ubuntu01:22
pleia2StevenK: it's also valuable to have it packaged for users who want it01:23
* mordred watches StevenK implode01:23
openstackgerritMerged openstack-infra/project-config: Fix pkgs-to-install path  https://review.openstack.org/16147201:23
pleia2but yeah, figured I'd give you an out if the deb packaging is kiling you01:23
kevinbentonhi, can I get another set of eyes on this request for a third party plugin repo? https://review.openstack.org/#/c/160339/01:24
kevinbentonit already has one +201:24
StevenKpleia2: I had to learn the ins and outs of mh_make from maven-debian-helper, but that has mostly gotten me a bunch of source packages -- or close to01:25
pleia2fun01:25
StevenKpleia2: I am building on Vivid due to other Java depends, but once I have this lot building on Vivid, backporting to Trusty shouldn't cause too much hair loss01:25
pleia2cool01:26
StevenKAnd please don't say Precise :-P01:26
cineramaaaaaah why is wildfly trying to use weld01:26
pleia2StevenK: haha, no, we're using trusty now :)01:26
mordredcinerama: because pain?01:26
pleia2java is hard01:27
StevenKpleia2: My poor firewall at home is still running *Lucid*01:27
*** Daisy__ has joined #openstack-infra01:27
*** carl_baldwin has quit IRC01:27
mordredStevenK: jeblair and I have a server running lucid01:27
cineramai did it the exact same way01:27
*** annegentle has quit IRC01:27
pleia2I have one running squeeze01:27
mordredStevenK: we're thinking that maybe now that trusty is out and mostly not terrible it might be time to upgrade01:27
cineramado i have to stop this car?01:27
StevenKmordred: I can not upgrade it to Precise, its processor doesn't support CMOV.01:27
pleia2StevenK: haha, nice01:27
mordred:)01:27
StevenKSo I either have to side-grade to Debian, or try and replace it01:28
openstackgerritRamy Asselin proposed openstack-infra/project-config: Use GIT_BASE environment variable  https://review.openstack.org/16153701:28
*** sdake_ has joined #openstack-infra01:28
StevenKIt's a low-power board, no fan or moving parts01:28
StevenKOr I buy another CF card and debootstrap Debian onto it01:28
StevenKpleia2: The other fun with this is because Java, it takes 10-12 minutes to install ~200MiB of Build-Depends.01:29
pleia2StevenK: it takes about 7 minutes for zanata to even *start*01:29
pleia2so that's special01:29
StevenKThe client, or the server? :-P01:29
pleia2server01:30
*** sdake__ has joined #openstack-infra01:30
*** Daisy_ has quit IRC01:30
pleia2java application servers are ridiculous01:30
*** mtanino has quit IRC01:30
mordredpleia2: but they can java things01:31
pleia2mordred: not helping01:31
clarkbmordred: http://paste.openstack.org/show/188367/01:31
clarkbmordred: that needs sanitizing, there are special dhcp ports in there and likely router ports as well01:31
clarkbmordred: but that is the list of ports that do not have device ids that match our nova list01:31
*** Daisy_ has joined #openstack-infra01:32
sigmavirus24mordred++01:32
clarkband with that actually afk now01:32
cineramaBOOM01:33
cineramagot it01:33
*** sdake_ has quit IRC01:33
mordredwoot!01:33
pleia2:D01:33
cineramaso there is this /etc/default/wildfly that the puppet module installs01:33
clarkbmordred: confirmed that router ports are in that list so uh don't delete anything yet :)01:34
cineramait is evil and we need to kill it with fire01:34
clarkbmordred: we will have sad times if we do01:34
sigmavirus24How's zuul doing tonight?01:34
clarkbsigmavirus24: mostly better01:34
pleia2cinerama: that makes me sad, maybe we can convince upstream to make it optional, or just delete it ourselves01:35
*** amotoki has joined #openstack-infra01:35
clarkbwhy is defaults file bad?01:35
*** Daisy__ has quit IRC01:35
*** gyee has quit IRC01:36
pleia2clarkb: just asked for a pastebin of it01:36
cineramadefaults file is here http://paste.openstack.org/show/188368/01:37
ianwhmm, i wonder what happened to https://review.openstack.org/#/c/161055/ ... seems to have disappeared from zuul without a trace01:37
pleia2cinerama: wow, that is a terrible file!01:37
pleia2seems like loading up lots of xml files that are breaking things :\01:38
cineramaoh yeah you betcha01:38
*** liusheng has quit IRC01:38
*** ashleighfarnham has quit IRC01:38
mordredclarkb: timrc has made a patch to add rebuild support to shade, and has also done some benchmarking01:39
mordredclarkb: seems to be roughly 3x faster to rebuild vs. delete/recreate01:39
clarkbin hpcloud ya01:40
timrcmordred, I'm using defaults with create_server (which has auto_ip set to True) so probably not a "real world" comparison in the context of nodepool.01:40
cineramaso the game now is "who's enabling weld"01:40
cineramaah i see01:40
mordredtimrc: actually, I think it is01:40
timrcmordred, Are floating ips not pooled?01:41
cineramaJBOSS_CONFIG=standalone-full.xml <- no bueno01:41
mordredtimrc: because auto_ip is what we'd want in nodepool (although auto_ip needs a few more patches)01:41
mordredtimrc: nope01:41
timrcmordred, Ah.01:41
mordredtimrc: we delete and recreate them every time01:41
pleia2ok, time to get some foods01:41
timrcmordred, Well then.01:41
mordredso honestly, we remove a lot of moving parts when we get rebuild working01:41
timrcYeah, I think the challenge is to work out how nodepool will know what to rebuild an instance as.01:42
clarkband when to rebuild vs delete01:42
mordredyup01:42
mordredit's not a quick patch - but I Think it's going to be a great feature01:42
*** dims has quit IRC01:43
*** dboik has joined #openstack-infra01:44
mordredclarkb, timrc: my napkin thought is to keep a demand counter, and if the demand coutner is > 0, then rebuild, if == 0, then delete01:44
*** sputnik13 has quit IRC01:44
clarkbmordred actually I think we can update the manager queues safely01:45
clarkbthe when you go to add a boot event if delete in queue replace with rebuild01:46
clarkbmaybe01:46
mordredyah01:46
clarkbthe problem is getting lajnch node to know when it can swap delete for rebuild01:46
clarkbbecause you dont want to do this on the delete side of things or code will get nasty01:47
*** adalbas has quit IRC01:47
*** tsg has joined #openstack-infra01:47
mordredclarkb: I'm sure docker would fix it01:48
*** dboik has quit IRC01:48
greghaynesI think we could use a dockerbot01:49
greghaynesthat randomly answers questions with "Maybe you could fix it with docker"01:49
cineramacool, i found the bit in the wildfly puppet module where we can pass the parameter to fix that01:50
*** vilobhmm1 has quit IRC01:50
timrcgreghaynes, ++01:50
cineramasuper obvious...after looking at stuff for two days :)01:50
*** jamielennox is now known as jamielennox|lunc01:51
*** tiswanso has joined #openstack-infra01:51
timrcclarkb, mordred Would you agree a spec for nodepool rebuild would be appropriate? I can start working on that.01:51
*** vilobhmm has joined #openstack-infra01:51
mordredtimrc: yeah. I think it's been requested enough - I know our cloud providers would like us to rebuild01:52
clarkbhonestly I am with jeblair on no features for nodepool until its better tested01:52
mordredwell, yeah - but we can write the spec and get that worked out01:52
*** tiswanso has quit IRC01:53
clarkbsure01:53
*** asettle is now known as asettle-gym01:53
timrcYour production instance is your test bed :)01:53
*** tiswanso has joined #openstack-infra01:53
*** claudiub has quit IRC01:54
timrcclarkb, Testing nodepool is probably something I'd be keen on doing considering how little I know if its inner workings.  So cool.01:55
*** dmsimard_away is now known as dmsimard01:55
timrcof*01:55
mordred\o/01:56
*** mfink__ has quit IRC01:56
cineramaheh...but that's not the only problem with the puppet installation. that would be easy.01:57
*** macjack has quit IRC01:59
*** macjack has joined #openstack-infra02:00
sigmavirus24So someone is telling me that https://review.openstack.org/#/c/161330/ should be dependent on https://review.openstack.org/#/c/161337/ but I didn't see any documentation mentioning that one thing required the other02:01
mordredsigmavirus24: I do't think it's a hard depend ... although I can see the reasoning and maybe we should add a mention to the docs...02:02
mordredsigmavirus24: basically, putting gerrit notifications into a channel that we don't have ACL management on is a little strange02:03
mordredsigmavirus24: of course, just landing both patches is also an easy way to fix that :)02:03
sigmavirus24mordred: ah that makes more sense, thanks for the explanation02:03
*** vilobhmm has quit IRC02:04
*** dmsimard is now known as dmsimard_away02:05
mordredsigmavirus24: that said - there's an error in one of them - I just plopped the -1 on it02:05
mordredsigmavirus24: but I think we can land these without too much trouble02:06
openstackgerritIan Cordasco proposed openstack-infra/project-config: Add gerrit notifications for #openstack-api  https://review.openstack.org/16133702:07
sigmavirus24mordred: fixed02:07
sigmavirus24Thanks for catching that02:07
mordredsigmavirus24: look great - both are +2 from me02:08
sigmavirus24Thanks mordred02:08
*** dannywilson has quit IRC02:10
*** patri455 has quit IRC02:14
*** patri696 has quit IRC02:14
*** patrickeast_ has quit IRC02:14
*** patrickeast has quit IRC02:14
*** patrickeast__ has quit IRC02:14
*** patri582 has quit IRC02:14
*** patrickeast___ is now known as patrickeast02:14
*** enikanorov_ has joined #openstack-infra02:16
*** Longgeek has joined #openstack-infra02:16
*** stevemar has joined #openstack-infra02:16
*** markvoelker has joined #openstack-infra02:16
*** patrickeast__ has joined #openstack-infra02:17
*** patrickeast_ has joined #openstack-infra02:17
*** patri891 has joined #openstack-infra02:17
*** patri598 has joined #openstack-infra02:17
*** patri810 has joined #openstack-infra02:17
*** patri192 has joined #openstack-infra02:17
*** jpeeler1 has joined #openstack-infra02:17
*** tsg_ has joined #openstack-infra02:18
*** aarefiev_ has joined #openstack-infra02:19
*** viglesia1 has joined #openstack-infra02:19
*** jgrimmn has joined #openstack-infra02:20
*** abramley_ has joined #openstack-infra02:20
*** andreaf_ has joined #openstack-infra02:20
*** cinerama` has joined #openstack-infra02:20
*** vponomaryov1 has joined #openstack-infra02:20
*** mrodden_ has joined #openstack-infra02:20
*** Hefeweiz1n has joined #openstack-infra02:20
*** fesp has joined #openstack-infra02:20
*** rwsu_ has joined #openstack-infra02:21
*** mrodden has quit IRC02:21
*** flaper87 has quit IRC02:21
*** abramley has quit IRC02:21
*** zz_jgrimm has quit IRC02:21
*** andreaf has quit IRC02:21
*** jpeeler has quit IRC02:21
*** cinerama has quit IRC02:21
*** vponomaryov has quit IRC02:21
*** tsg has quit IRC02:21
*** reed has quit IRC02:21
*** aarefiev has quit IRC02:21
*** Hefeweizen has quit IRC02:21
*** pcrews has quit IRC02:21
*** rwsu has quit IRC02:21
*** chmouel has quit IRC02:21
*** viglesias has quit IRC02:21
*** enikanorov__ has quit IRC02:21
*** tsg_ has quit IRC02:21
*** tsg_ has joined #openstack-infra02:21
*** jgrimmn has quit IRC02:21
*** jgrimmn has joined #openstack-infra02:21
*** viglesia1 has quit IRC02:21
*** viglesia1 has joined #openstack-infra02:21
*** fesp has quit IRC02:21
*** fesp has joined #openstack-infra02:21
*** mrodden_ is now known as mrodden02:21
*** abramley_ is now known as abramley02:21
*** andreaf_ is now known as andreaf02:21
*** fesp is now known as flaper8702:21
*** markvoelker has quit IRC02:21
*** chmouel has joined #openstack-infra02:22
*** ghostpl_ has joined #openstack-infra02:22
*** coolsvap_ is now known as coolsvap02:22
*** Longgeek has quit IRC02:22
*** Longgeek has joined #openstack-infra02:24
*** Longgeek has quit IRC02:25
*** Longgeek has joined #openstack-infra02:25
*** ChuckC has joined #openstack-infra02:26
*** ghostpl_ has quit IRC02:26
*** pcrews has joined #openstack-infra02:27
*** patrickeast has quit IRC02:29
*** patrickeast__ has quit IRC02:29
*** patri598 has quit IRC02:29
*** patri810 has quit IRC02:29
*** patri192 has quit IRC02:29
*** patri891 has quit IRC02:29
*** patrickeast_ has quit IRC02:29
*** reed has joined #openstack-infra02:32
*** david-lyle has joined #openstack-infra02:33
*** rwsu_ is now known as rwsu-afk02:35
*** reed has quit IRC02:36
*** reed has joined #openstack-infra02:37
*** sdake__ has quit IRC02:38
*** ashleighfarnham has joined #openstack-infra02:39
*** dims has joined #openstack-infra02:39
*** coolsvap is now known as coolsvap_02:41
cinerama`almost there with this02:42
*** cinerama` is now known as cinerama02:42
*** sarob has joined #openstack-infra02:42
*** ashleighfarnham has quit IRC02:44
cineramai think the the end is in sight...tomorrow02:46
*** tsg_ has quit IRC02:46
*** tsg has joined #openstack-infra02:47
*** jamielennox|lunc is now known as jamielennox02:47
*** sarob has quit IRC02:47
mordredStevenK: fyi - if you troll through hours of scrollback, we've got an openstack project that needs a C library that's not in distro02:48
mordredStevenK: so our need to sort out apt repos and building packages has a new use case02:49
StevenKYay?02:49
mordredStevenK: however, in this case, the library _has_ been packaged for debian and is in debian02:49
mordredis just didn't make the ubuntu sync02:49
StevenKmordred: So we can either backport it to trusty-backports and just use that, or some random PPA that trusted people can upload to, or do something else.02:50
*** xchu has joined #openstack-infra02:51
cody-somervilleLet's deploy our own instance of Launchpad.02:51
*** otter768 has joined #openstack-infra02:52
*** xchu has quit IRC02:52
nibalizercody-somerville: plzno02:55
*** achanda has quit IRC02:55
cody-somervillenibalizer: Hey man. You feeling any better?02:56
*** otter768 has quit IRC02:57
nibalizerquite a bit02:58
nibalizer:)02:58
greghaynesI thought that had been atempted03:00
greghaynesalso, I bet StevenK would love to03:00
StevenKThere are a number of private Launchpad's around the place03:00
StevenKMost people don't tend to talk about that.03:00
*** yamahata has quit IRC03:00
greghaynesThe first rule of installing launchpad is...03:00
StevenKHire an ex-Launchpad developer?03:01
mordredStevenK: well, let's say no to PPA03:01
*** ChuckC has quit IRC03:01
mordredStevenK: in general, we should scheme up what having an infra run apt repo looks like that we can use our big fancy build farm to populate03:01
mordredStevenK: we don't need to solve it today though03:02
StevenKmordred: Well, you have some ideas about never removing obsolete packages either, which tends to mean we can't use reprepro or dak03:02
mordredStevenK: I don't know that we have to never remove - but ultimately we don't _need_ to remove obsolete03:02
StevenKapt-mirror could probably be convinced to do it, but oh god03:02
*** rm_work is now known as rm_work|away03:02
dstufftaptly!03:02
mordredor aptly03:03
dstuffthttp://www.aptly.info/03:03
greghaynesI thought apt-mirror didnt clean up unless you run its cleanup.sh thing03:03
dstufftI use that on the PSF Salt stuff03:03
dstufftto host crap for the Python infra03:03
StevenKTBH, I'd just rather have apt-ftparchive cronned :-P03:04
*** ddieterly has joined #openstack-infra03:04
mordredwell, I mean, the thing is - running the repo? not the hard part03:04
StevenKSure03:04
StevenKControlling write acess, how to build, etc, those are the hard parts03:04
mordreddeveloper workflow around getting packages in to it etc? completely undefined03:04
mordredyup03:05
mordredwell, there will be no write access03:05
mordredthere will be zero people with repo upload permissions03:05
mordredthese are things I can say with 100% assurance03:05
StevenKThen the repo is done and it will be empty? :_P03:05
*** ddieterl_ has joined #openstack-infra03:05
mordredStevenK: zuul will be the only person with those permissions03:05
StevenKBut I know what you mean, you want it done by robot03:05
mordredyup03:05
*** ddieterly has quit IRC03:06
mordredrobots are better at things03:06
mordredhumans are very bad squishy piles of error03:06
StevenKYou should watch Terminator 2 to see how that turns out03:06
*** ChuckC has joined #openstack-infra03:06
*** david-lyle has quit IRC03:07
mordredStevenK: :)03:07
mordredStevenK: I mean, there are guns and roses songs - so generally pretty well, right?03:08
clarkbI dont know that I want to visit the jungle though03:08
dstufftit has fun and games03:08
*** camunoz has quit IRC03:09
SlickNikStevenK: Well — the robots in Terminator 2 were built by humans right?03:09
mordredand, apparently, they even know the names of all the things you need03:09
SlickNikCase and point :)03:09
cody-somervilleStevenK: You can tell reprepro to not remove superseded packages.03:09
mordredhoney03:09
StevenKSlickNik: Nope, humans had built the precedessors03:09
mordredthis is why we shy away from letting zuul directly commit code ...03:10
*** jpeeler1 is now known as jpeeler03:10
*** jpeeler has quit IRC03:10
*** jpeeler has joined #openstack-infra03:10
SlickNikStevenK: Okay a generation removed, but still. :)03:11
SlickNikI for one welcome my zuul-gen-2 overlords.03:11
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting  https://review.openstack.org/15887703:11
mordredSlickNik: ++03:11
clarkbthats when I get to move back to island and fish right?03:11
clarkbwhen zuul starts to write code for me03:12
*** camunoz has joined #openstack-infra03:12
mordredclarkb: why, btw, have we not all moved back to island and fish?03:13
greghaynes++03:13
clarkbmordred not enough bandwidth03:13
greghaynesThis is a solvable problem03:13
mordredclarkb: for fishing?03:13
clarkbfor when you arent fishing03:14
mordredclarkb: I think I'm unclear on why that is a concern03:14
clarkbmordred for streaming/torrenting music to clean fish to?03:15
mordredclarkb: I think CDs still physically exist somewhere03:16
clarkbits possible. probably in micronesia03:16
greghaynesCD's exist, but cd drives do not03:16
*** markvoelker has joined #openstack-infra03:17
*** markvoelker has quit IRC03:22
*** asettle-gym is now known as asettle03:26
*** yamamoto has joined #openstack-infra03:28
*** achanda has joined #openstack-infra03:29
*** SumitNaiksatam has joined #openstack-infra03:32
HenryGHey I wonder if anyone can help me to diagnose why check-neutron-dsvm-functional jobs for stable/juno are suddenly failing?03:33
HenryGhttp://logs.openstack.org/25/161325/2/check/check-neutron-dsvm-functional/527457f/console.html#_2015-03-05_01_32_25_38803:33
*** mika has quit IRC03:35
HenryGI see https://review.openstack.org/159900 and https://review.openstack.org/159672 recently merged. Do they not play well together?03:35
*** SumitNaiksatam has quit IRC03:38
*** yamamoto has quit IRC03:41
*** tsg has quit IRC03:42
*** ddieterly has joined #openstack-infra03:43
*** vilobhmm has joined #openstack-infra03:43
*** ddieterl_ has quit IRC03:45
*** ddieterl_ has joined #openstack-infra03:45
*** jamielennox is now known as jamielennox|away03:45
*** ddieterly has quit IRC03:45
*** tsg has joined #openstack-infra03:45
*** sigmavirus24 is now known as sigmavirus24_awa03:47
*** Alexandra_ has joined #openstack-infra03:47
*** teran has quit IRC03:51
*** makkie has joined #openstack-infra03:52
*** spzala has quit IRC03:52
*** yamamoto has joined #openstack-infra03:54
*** teran has joined #openstack-infra03:54
*** Daisy_ has quit IRC04:01
*** weshay has quit IRC04:03
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting  https://review.openstack.org/15887704:04
*** rkukura has quit IRC04:07
*** SumitNaiksatam has joined #openstack-infra04:08
*** ghostpl_ has joined #openstack-infra04:11
*** unicell1 has quit IRC04:13
*** reed has quit IRC04:15
*** ghostpl_ has quit IRC04:16
*** salv-orlando has joined #openstack-infra04:16
*** yamamoto has quit IRC04:17
*** markvoelker has joined #openstack-infra04:19
*** salv-orlando has quit IRC04:21
*** achanda has quit IRC04:22
*** yamahata has joined #openstack-infra04:22
*** dims has quit IRC04:22
*** markvoelker has quit IRC04:23
*** ivar-lazzaro has joined #openstack-infra04:24
*** koolhead17 has joined #openstack-infra04:25
*** ivar-laz_ has quit IRC04:27
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: (Try to) Fix the Neutron functional job for stable branches  https://review.openstack.org/16156904:28
*** ivar-lazzaro has quit IRC04:29
*** zz_dimtruck has quit IRC04:29
*** Daisy_ has joined #openstack-infra04:29
*** yamamoto has joined #openstack-infra04:30
*** makkie has quit IRC04:32
*** vilobhmm has quit IRC04:35
*** woodster_ has quit IRC04:40
*** yamamoto_ has joined #openstack-infra04:43
*** yamamoto has quit IRC04:43
*** sarob has joined #openstack-infra04:44
*** tiswanso has quit IRC04:45
openstackgerritMerged openstack/requirements: Bump oslo.config to latest release  https://review.openstack.org/16135304:47
*** vishy has quit IRC04:47
*** deepakcs has joined #openstack-infra04:48
*** bhunter71 has quit IRC04:48
*** yamamoto_ has quit IRC04:48
*** sarob has quit IRC04:49
*** otter768 has joined #openstack-infra04:53
*** harlowja is now known as harlowja_away04:57
*** otter768 has quit IRC04:58
*** melwitt has quit IRC04:59
*** clarkb has quit IRC05:00
*** ddieterl_ has quit IRC05:01
*** carl_baldwin has joined #openstack-infra05:08
*** carl_baldwin has quit IRC05:08
*** sputnik13 has joined #openstack-infra05:09
*** garyh has quit IRC05:09
*** harlowja_at_home has joined #openstack-infra05:10
*** dannywilson has joined #openstack-infra05:11
*** ayoung has quit IRC05:11
*** stevemar has quit IRC05:13
*** sputnik13 has quit IRC05:13
*** clarkb has joined #openstack-infra05:13
*** Ryan_Lane has joined #openstack-infra05:14
*** dannywilson has quit IRC05:16
*** yamamoto has joined #openstack-infra05:17
*** markvoelker has joined #openstack-infra05:20
*** markmcclain has quit IRC05:20
*** markmcclain has joined #openstack-infra05:22
*** achanda has joined #openstack-infra05:23
*** dims has joined #openstack-infra05:23
*** achanda has quit IRC05:23
*** achanda has joined #openstack-infra05:23
*** mfink_ has joined #openstack-infra05:24
*** markvoelker has quit IRC05:25
*** tsg_ has joined #openstack-infra05:26
*** hamzy_ has joined #openstack-infra05:26
*** ParsectiX has quit IRC05:26
*** ParsectiX has joined #openstack-infra05:27
*** hamzy_ has quit IRC05:27
*** hamzy_ has joined #openstack-infra05:28
*** dims has quit IRC05:28
*** tsg has quit IRC05:29
*** Ryan_Lane has quit IRC05:31
openstackgerritSyed Ismail Faizan Barmawer proposed openstack-infra/devstack-gate: Add ironic-lib to devstack-vm-gate-wrap.sh  https://review.openstack.org/16160005:31
*** Ryan_Lane has joined #openstack-infra05:33
*** Alexandra_ has quit IRC05:34
*** woodster_ has joined #openstack-infra05:34
openstackgerritJoshua Hesketh proposed openstack/requirements: Update pip and pip-missing-reqs  https://review.openstack.org/15929305:35
openstackgerritJoshua Hesketh proposed openstack/requirements: Add new prettytable requirement for nova  https://review.openstack.org/15970605:35
*** harlowja_at_home has quit IRC05:35
*** hamzy has quit IRC05:35
*** hamzy_ is now known as hamzy05:36
*** rkukura has joined #openstack-infra05:37
*** Alexandra_ has joined #openstack-infra05:38
*** sdake_ has joined #openstack-infra05:41
*** Ryan_Lane has quit IRC05:43
openstackgerritSumit Naiksatam proposed openstack-infra/project-config: Create experimental job for group-based-policy  https://review.openstack.org/16151105:44
*** sdake__ has joined #openstack-infra05:44
*** xyang1 has quit IRC05:45
*** sdake_ has quit IRC05:48
*** tsg_ has quit IRC05:54
openstackgerritSyed Ismail Faizan Barmawer proposed openstack/requirements: Add ironic-lib to project.txt and global-requirements.txt  https://review.openstack.org/16160305:57
*** garyk has quit IRC05:59
*** ghostpl_ has joined #openstack-infra05:59
*** ddieterly has joined #openstack-infra06:02
*** ghostpl_ has quit IRC06:04
*** punith has joined #openstack-infra06:06
*** ddieterly has quit IRC06:06
*** Alexandra_ has quit IRC06:07
*** ttx has quit IRC06:07
*** BharatK has joined #openstack-infra06:07
*** ParsectiX has quit IRC06:08
*** unicell has joined #openstack-infra06:14
*** gmann has joined #openstack-infra06:16
*** ttx has joined #openstack-infra06:19
*** unicell has quit IRC06:22
*** markvoelker has joined #openstack-infra06:22
*** unicell has joined #openstack-infra06:23
*** markvoelker has quit IRC06:27
*** ildikov has quit IRC06:32
*** erikmwilson has quit IRC06:34
*** e0ne has joined #openstack-infra06:36
*** e0ne has quit IRC06:38
*** sergsh has joined #openstack-infra06:39
*** vilobhmm has joined #openstack-infra06:40
*** vilobhmm has quit IRC06:41
*** vilobhmm has joined #openstack-infra06:41
*** vilobhmm1 has joined #openstack-infra06:42
*** e0ne has joined #openstack-infra06:42
*** vilobhmm has quit IRC06:43
*** e0ne has quit IRC06:43
*** camunoz has quit IRC06:48
*** Krast has joined #openstack-infra06:51
*** vilobhmm has joined #openstack-infra06:54
*** otter768 has joined #openstack-infra06:54
*** stevemar has joined #openstack-infra06:55
*** koolhead_ has joined #openstack-infra06:56
*** koolhead17 has quit IRC06:57
*** vilobhmm1 has quit IRC06:57
*** koolhead17 has joined #openstack-infra06:58
*** rkukura has quit IRC06:59
*** jyuso310 has quit IRC06:59
*** otter768 has quit IRC06:59
*** koolhead_ has quit IRC07:02
*** fifieldt_ has joined #openstack-infra07:02
*** claudiub has joined #openstack-infra07:03
*** ddieterly has joined #openstack-infra07:03
*** garyk has joined #openstack-infra07:05
*** HeOS has quit IRC07:07
*** ddieterly has quit IRC07:08
*** koolhead_ has joined #openstack-infra07:09
*** koolhead17 has quit IRC07:12
*** ildikov has joined #openstack-infra07:12
*** vigneshvar has joined #openstack-infra07:13
*** salv-orlando has joined #openstack-infra07:14
*** ghostpl_ has joined #openstack-infra07:15
*** salv-orlando has quit IRC07:19
*** ghostpl_ has quit IRC07:20
*** vilobhmm has quit IRC07:21
*** vilobhmm has joined #openstack-infra07:22
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual: Use change instead of patch  https://review.openstack.org/16163207:22
*** markvoelker has joined #openstack-infra07:22
*** jvrbanac has quit IRC07:24
*** salv-orlando has joined #openstack-infra07:27
*** markvoelker has quit IRC07:28
*** vilobhmm has quit IRC07:31
*** liusheng has joined #openstack-infra07:32
*** mrmartin has joined #openstack-infra07:35
openstackgerritMerged openstack-infra/project-config: Add gerrit notifications for #openstack-api  https://review.openstack.org/16133707:35
*** openstackgerrit has quit IRC07:36
*** openstackgerrit has joined #openstack-infra07:36
*** mrmartin has quit IRC07:39
*** woodster_ has quit IRC07:40
*** ashleighfarnham has joined #openstack-infra07:40
*** ashleighfarnham has quit IRC07:45
*** sarob has joined #openstack-infra07:47
*** mpavone has joined #openstack-infra07:50
*** sarob has quit IRC07:53
*** scheuran has joined #openstack-infra07:56
*** garyk has quit IRC07:56
*** markus_z has joined #openstack-infra07:57
*** fifieldt_ has quit IRC07:58
*** garyk has joined #openstack-infra08:02
*** ddieterly has joined #openstack-infra08:04
*** tnovacik has joined #openstack-infra08:06
*** Bsony has joined #openstack-infra08:06
*** ddieterly has quit IRC08:09
*** arxcruz has joined #openstack-infra08:09
*** Ala has joined #openstack-infra08:15
*** achanda has quit IRC08:16
*** ghostpl_ has joined #openstack-infra08:16
*** hashar has joined #openstack-infra08:20
*** ghostpl_ has quit IRC08:22
*** markvoelker has joined #openstack-infra08:25
*** tchaypo is now known as tchorizo08:25
*** tchorizo is now known as tchaypo08:25
*** esheffield_ has joined #openstack-infra08:29
*** markvoelker has quit IRC08:29
*** Krast has quit IRC08:33
GheRiveromorning all!08:33
*** achanda has joined #openstack-infra08:33
ianychoihello : )08:33
*** salv-orlando has quit IRC08:33
*** afazekas has joined #openstack-infra08:34
*** ttx has quit IRC08:34
*** ttx has joined #openstack-infra08:34
*** MaxV has joined #openstack-infra08:35
*** dizquierdo has joined #openstack-infra08:36
*** luqas__ has joined #openstack-infra08:36
*** jlanoux has joined #openstack-infra08:42
*** vivcheri has joined #openstack-infra08:43
*** devvesa has joined #openstack-infra08:44
*** gulic has joined #openstack-infra08:45
*** devvesa has quit IRC08:47
*** devvesa has joined #openstack-infra08:48
*** cyeoh has quit IRC08:49
*** jistr has joined #openstack-infra08:49
*** jcoufal has joined #openstack-infra08:54
*** otter768 has joined #openstack-infra08:55
*** hichihara has quit IRC08:56
*** dtantsur|afk is now known as dtantsur08:58
*** BobBall_AWOL is now known as BobBall08:59
*** otter768 has quit IRC09:00
*** HeOS has joined #openstack-infra09:00
*** dims has joined #openstack-infra09:01
*** pblaho has joined #openstack-infra09:02
*** stevemar has quit IRC09:03
*** stevemar has joined #openstack-infra09:04
*** ddieterly has joined #openstack-infra09:05
*** salv-orlando has joined #openstack-infra09:05
*** dims has quit IRC09:06
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Configure triggers dynamically  https://review.openstack.org/11953409:07
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add support for 'connection' concept  https://review.openstack.org/12152809:07
openstackgerritJoshua Hesketh proposed openstack-infra/zuul: Add base class for triggers  https://review.openstack.org/11953209:07
*** yamahata has quit IRC09:07
*** mika has joined #openstack-infra09:08
*** skolekonov has joined #openstack-infra09:09
*** ddieterly has quit IRC09:09
*** teran has quit IRC09:12
*** achanda has quit IRC09:12
*** BharatK has quit IRC09:17
*** jedimike has joined #openstack-infra09:21
*** ihrachyshka has joined #openstack-infra09:22
*** BharatK has joined #openstack-infra09:22
*** markvoelker has joined #openstack-infra09:26
*** ghostpl_ has joined #openstack-infra09:26
*** markvoelker has quit IRC09:31
*** teran has joined #openstack-infra09:34
*** dizquierdo has quit IRC09:37
*** teran has quit IRC09:39
*** che-arne has joined #openstack-infra09:40
*** mpaolino has joined #openstack-infra09:40
openstackgerritJesse Pretorius proposed openstack-infra/system-config: Add logging to #openstack-ansible  https://review.openstack.org/16141209:41
*** sdake__ has quit IRC09:44
*** Guest40575 has quit IRC09:45
*** Hal has joined #openstack-infra09:45
*** Hal is now known as Guest7703609:45
*** ghostpl_ has quit IRC09:46
*** doude has joined #openstack-infra09:47
*** sarob has joined #openstack-infra09:50
*** sarob has quit IRC09:54
*** ssam2 has joined #openstack-infra09:54
*** salv-orlando has quit IRC09:59
*** penguinRaider has quit IRC10:00
*** liusheng has quit IRC10:00
*** liusheng has joined #openstack-infra10:01
*** jlk has quit IRC10:06
*** wendar has quit IRC10:06
*** ujuc_ has joined #openstack-infra10:07
*** koesik has quit IRC10:10
*** tim_o has quit IRC10:10
*** wendar has joined #openstack-infra10:16
*** kiseok7_ has joined #openstack-infra10:16
*** teran has joined #openstack-infra10:16
*** ghostpl_ has joined #openstack-infra10:18
*** teran_ has joined #openstack-infra10:19
*** teran has quit IRC10:21
*** markmcclain has quit IRC10:22
*** markmcclain has joined #openstack-infra10:26
*** e0ne has joined #openstack-infra10:26
*** e0ne is now known as e0ne_10:26
*** markvoelker has joined #openstack-infra10:27
*** e0ne_ has quit IRC10:27
*** e0ne has joined #openstack-infra10:27
openstackgerrityolanda.robla proposed openstack-infra/puppet-gerrit: Add logging to fetchremotes call  https://review.openstack.org/16166310:30
*** markvoelker has quit IRC10:31
*** dstanek has quit IRC10:33
*** alexpilotti has joined #openstack-infra10:34
*** ujuc_ has quit IRC10:36
*** ujuc has joined #openstack-infra10:37
*** pelix has joined #openstack-infra10:42
*** Daisy_ has quit IRC10:44
*** jp_at_hp has joined #openstack-infra10:51
openstackgerrityolanda.robla proposed openstack-infra/puppet-graphite: Add missing storage subfolders  https://review.openstack.org/16167610:53
*** dims has joined #openstack-infra10:55
*** otter768 has joined #openstack-infra10:56
openstackgerrityolanda.robla proposed openstack-infra/puppet-graphite: Add ability to configure retention policy  https://review.openstack.org/16167910:58
*** salv-orlando has joined #openstack-infra11:00
*** e0ne is now known as e0ne_11:00
*** otter768 has quit IRC11:01
*** rlandy has joined #openstack-infra11:05
*** ddieterly has joined #openstack-infra11:06
*** e0ne_ has quit IRC11:11
*** ddieterly has quit IRC11:11
*** pc_m has joined #openstack-infra11:12
*** e0ne has joined #openstack-infra11:14
*** mfink_ has quit IRC11:15
*** doude_ has joined #openstack-infra11:24
*** mrunge has joined #openstack-infra11:26
*** doude has quit IRC11:28
*** stevemar has quit IRC11:28
*** markvoelker has joined #openstack-infra11:28
pc_mHad a question... if a commit has been approved, but during verify fails Jenkins (passed during check pipeline), how is that handled?11:31
pc_mDo we do a comment with "recheck" or "reverify"?11:31
*** Guest77036 has quit IRC11:31
*** Hal has joined #openstack-infra11:32
*** markvoelker has quit IRC11:32
*** Hal is now known as Guest9589311:33
*** _nadya_ has joined #openstack-infra11:33
yolandahere, i'm having continuos failures on precise for  puppet-gerrit, and don't seem to be related with my changeç11:33
yolandahttp://logs.openstack.org/63/161663/1/check/gate-infra-puppet-apply-precise/8789b86/11:33
yolandais that a known issue?11:34
GheRiveropc_m: I guess is reverify but not sure11:35
yolandaGheRivero, i think it's just recheck, and it pass through check and gate again11:35
yolandareverify was demoted11:35
yolandaat least it was like that some months ago, maybe you know more than me :)11:36
pc_myolanda: I'll try recheck. I just did reverify command.11:36
pc_myolanda: thanks11:36
yolandaanytime11:36
*** ldnunes has joined #openstack-infra11:37
openstackgerritMerged openstack-infra/project-config: Functional tests job for policy enforcement  https://review.openstack.org/15040111:38
*** vigneshvar has quit IRC11:40
*** salv-orlando has quit IRC11:42
*** rfolco has joined #openstack-infra11:52
*** devvesa has quit IRC11:54
*** cdent has joined #openstack-infra11:55
*** _nadya_ has quit IRC12:00
*** EmilienM|afk is now known as EmilienM12:04
openstackgerritThomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project  https://review.openstack.org/16129012:06
*** ddieterly has joined #openstack-infra12:07
*** afazekas has quit IRC12:08
*** gmann_ has joined #openstack-infra12:08
*** ghostpl_ has quit IRC12:11
*** ddieterly has quit IRC12:12
*** dtantsur is now known as dtantsur|brb12:13
*** claudiub has quit IRC12:16
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add the ability to configure jeepyb git repo  https://review.openstack.org/16169112:16
*** e0ne is now known as e0ne_12:20
*** snikitin_ has joined #openstack-infra12:21
*** kirshil has joined #openstack-infra12:22
*** afazekas has joined #openstack-infra12:23
*** ujuc has quit IRC12:25
*** mwagner_lap has quit IRC12:26
*** markvoelker has joined #openstack-infra12:29
openstackgerrityolanda.robla proposed openstack-infra/system-config: Expose jenkins_gitfullname and jenkins_gitemail  https://review.openstack.org/16169512:29
*** aysyd has joined #openstack-infra12:29
*** e0ne_ has quit IRC12:30
*** MaxV has quit IRC12:31
*** afazekas has quit IRC12:31
*** deepakcs has quit IRC12:32
*** weshay has joined #openstack-infra12:32
*** markvoelker has quit IRC12:34
*** ujuc has joined #openstack-infra12:34
*** jcoufal_ has joined #openstack-infra12:34
*** lucasagomes has joined #openstack-infra12:36
lucasagomeshello there, I need to add a new gate job for Ironic. I've never done that before, there's any example someone can point me to? Thank you *goes and keep googling*12:37
*** jcoufal has quit IRC12:37
*** devvesa has joined #openstack-infra12:37
*** markvoelker has joined #openstack-infra12:38
*** coolsvap_ has quit IRC12:38
*** spredzy is now known as spredzy|afk12:38
*** afazekas has joined #openstack-infra12:43
*** coolsvap_ has joined #openstack-infra12:46
*** hashar is now known as hasharLunch12:47
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add the ability to configure jeepyb git repo  https://review.openstack.org/16169112:51
*** dmellado has quit IRC12:52
*** ujuc has quit IRC12:52
*** sarob has joined #openstack-infra12:53
openstackgerrityolanda.robla proposed openstack-infra/system-config: Expose jenkins_gitfullname and jenkins_gitemail  https://review.openstack.org/16169512:54
*** salv-orlando has joined #openstack-infra12:56
*** otter768 has joined #openstack-infra12:57
*** baoli has joined #openstack-infra12:57
*** baoli has quit IRC12:57
*** sarob has quit IRC12:57
*** tsg has joined #openstack-infra12:58
*** dprince has joined #openstack-infra13:01
*** mwagner_lap has joined #openstack-infra13:01
*** otter768 has quit IRC13:01
*** dmellado has joined #openstack-infra13:02
*** baoli has joined #openstack-infra13:06
*** ghostpl_ has joined #openstack-infra13:06
*** ddieterly has joined #openstack-infra13:08
*** ddieterly has quit IRC13:13
*** yamamoto has quit IRC13:13
*** kgiusti has joined #openstack-infra13:16
*** ihrachyshka has quit IRC13:18
*** devvesa has quit IRC13:20
*** devvesa has joined #openstack-infra13:20
*** amotoki has quit IRC13:21
*** ihrachyshka has joined #openstack-infra13:22
*** woodster_ has joined #openstack-infra13:22
*** Bsony_ has joined #openstack-infra13:24
*** Bsony has quit IRC13:26
*** adalbas has joined #openstack-infra13:26
*** lucasagomes is now known as lucas-hungry13:27
*** e0ne has joined #openstack-infra13:29
*** dtantsur|brb is now known as dtantsur13:32
snikitin_Guys, I have a problem. Here is approved patch https://review.openstack.org/#/c/140313/  It passed check tests. But gate tests didn't start. 'reverify' and 'recheck' didn't help. Could someone help me with that?13:33
*** gordc has joined #openstack-infra13:33
*** ddieterly has joined #openstack-infra13:34
*** mbacchi has joined #openstack-infra13:34
*** hamzy has quit IRC13:34
*** hamzy has joined #openstack-infra13:35
snikitin_problem resolved by reapprove13:36
*** yamamoto has joined #openstack-infra13:37
*** tnovacik has quit IRC13:39
*** spredzy|afk is now known as spredzy13:39
*** dmellado_ has joined #openstack-infra13:40
*** dmellado has quit IRC13:40
*** dmellado_ is now known as dmellado13:40
hasharLunchHello, does anyone know whether openstack-pkg-tools packaging helper ends up disabling network/pypi access ?13:44
openstackgerrityolanda.robla proposed openstack-infra/system-config: Expose jenkins_gitfullname and jenkins_gitemail  https://review.openstack.org/16169513:45
*** dstanek has joined #openstack-infra13:45
*** erikmwilson has joined #openstack-infra13:47
*** gmann_ has quit IRC13:48
openstackgerrityolanda.robla proposed openstack-infra/puppet-gerrit: Add logging to fetchremotes call  https://review.openstack.org/16166313:48
*** dims has quit IRC13:51
aarefiev_hi folks! Could somebody help? I have a patch https://review.openstack.org/#/c/161677/  , jenkins fail with merge conflict, but it is up to date. Also it depends on anothe patch from cinder13:51
*** dims has joined #openstack-infra13:52
*** hasharLunch is now known as hashar13:58
*** ayoung has joined #openstack-infra14:02
*** BharatK has quit IRC14:02
*** dboik has joined #openstack-infra14:05
*** sdake_ has joined #openstack-infra14:06
BobBallSomething mus tbe wrong with my zuul config... I keep managing to get changeish's where one of project.name is None which triggers an exception in the status update: http://paste.openstack.org/show/189158/ - Any ideas?14:09
BobBalllayout.yaml is https://github.com/citrix-openstack/os-ext-testing/blob/master/etc/zuul/layout.yaml14:10
*** ddieterly has quit IRC14:10
*** bcourt has joined #openstack-infra14:12
*** dkranz has joined #openstack-infra14:12
*** dustins has joined #openstack-infra14:14
*** sergsh has quit IRC14:14
*** tnovacik has joined #openstack-infra14:16
*** punith has quit IRC14:16
*** peristeri has joined #openstack-infra14:17
*** radez_g0n3 is now known as radez14:18
*** doude_ has quit IRC14:22
*** doude has joined #openstack-infra14:22
*** amotoki has joined #openstack-infra14:23
pc_mANyone here know about testr w.r.t. --coverage target? It is no longer working in the neutron-vpnaas repo (invalid command option).14:23
*** mrmartin has joined #openstack-infra14:24
openstackgerritMonty Taylor proposed openstack-infra/system-config: Drive actions around repo creation from ansible  https://review.openstack.org/16151814:26
*** tiswanso has joined #openstack-infra14:29
*** mriedem-away is now known as mriedem14:29
*** mattfarina has joined #openstack-infra14:35
*** jgrimmn has quit IRC14:38
*** maurosr has quit IRC14:38
*** arxcruz has quit IRC14:39
*** annegentle has joined #openstack-infra14:39
*** jgrimm has joined #openstack-infra14:40
*** achuprin_ has joined #openstack-infra14:41
*** maurosr has joined #openstack-infra14:41
*** ddieterly has joined #openstack-infra14:41
*** _nadya_ has joined #openstack-infra14:41
*** aysyd has quit IRC14:42
*** aysyd has joined #openstack-infra14:43
*** lucas-hungry is now known as lucasagomes14:43
*** jgrimm has quit IRC14:44
*** maurosr has quit IRC14:44
*** ghostpl_ has quit IRC14:44
*** jgrimm has joined #openstack-infra14:45
*** maurosr has joined #openstack-infra14:46
*** r-daneel has joined #openstack-infra14:46
*** chuckC_ has joined #openstack-infra14:46
*** e0ne is now known as e0ne_14:47
*** chuckC_ has quit IRC14:48
*** ghostpl_ has joined #openstack-infra14:48
*** aysyd has quit IRC14:48
*** eharney has joined #openstack-infra14:52
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting  https://review.openstack.org/15887714:52
*** dizquierdo has joined #openstack-infra14:53
*** yamamoto has quit IRC14:53
* mkj_wk hopes patch set 19 makes it through Jenkins so it's worth reviewing... :/14:53
openstackgerritMerged openstack-infra/shade: Add behavior to enable ironic noauth mode  https://review.openstack.org/15960914:54
openstackgerritMerged openstack-infra/shade: Add ironic node deployment support  https://review.openstack.org/16089414:54
openstackgerritMerged openstack-infra/shade: Reorder envlist to avoid the rm -fr .testrepository when running tox -epy34  https://review.openstack.org/15954514:54
*** radez is now known as radez_g0n314:56
*** carl_baldwin has joined #openstack-infra14:57
*** amitgandhinz has joined #openstack-infra14:57
*** otter768 has joined #openstack-infra14:58
*** mpaolino has quit IRC14:58
*** tonytan4ever has joined #openstack-infra14:58
*** sigmavirus24_awa is now known as sigmavirus2414:59
*** yamamoto has joined #openstack-infra14:59
*** krtaylor has quit IRC14:59
*** hashar is now known as hasharOut15:00
timrcSpamapS, Wasn't aware of that patch to add the create_server test to shade, so thanks for pointing it out.  I'm not married to using mock but monkey patching seemed like a super simple and consistent way to test all the success and failure paths in that method.15:01
lucasagomeshello there, I need to add a new gate job for Ironic. I've never done that before, there's any example someone can point me to? Thank you *goes and keep googling*15:02
*** deepakcs has joined #openstack-infra15:02
*** erikmwilson_ has joined #openstack-infra15:02
*** erikmwilson has quit IRC15:02
*** otter768 has quit IRC15:02
*** erikmwilson_ is now known as erikmwilson15:02
TheJuliamordred: Regarding shade, do you have any feelings on the subject of retry and wait logic?  Specifically I'm thinking of adding wait/retry logic into calls that would be made for ironic resources, naturally with the default behavior for them not to engage, but allow a user from ansible playbook to trigger them.15:04
*** hasharOut has quit IRC15:05
mordredTheJulia: hrm. that's an interesting question ...15:05
*** baoli has quit IRC15:05
*** koolhead_ has quit IRC15:05
mordredjeblair: ^^ this might be a conversation for you too15:06
mordredTheJulia: so basically a "keep_retrying_until_the_cloud_doesnt_suck=False" parameter?15:06
jeblairmordred, TheJulia: there's already some optional wait logic in there, right?  seems optional retry wouldn't be a huge expansion...15:07
*** e0ne_ is now known as e0ne15:08
TheJuliamordred: I was thinking retry_count: x15:08
*** dmsimard_away is now known as dmsimard15:08
*** prad has joined #openstack-infra15:09
mordredTheJulia: yours is likely named better than my suggestion15:09
mordredbut yeah - I think I agree with jeblair - seems like a sane and useful idea15:09
timrcSpamapS, Whoa I read the commit https://review.openstack.org/#/c/160155/1 -- in no way does that fill me with confidence :)15:09
mordredTheJulia: does retry_count imply wait/timeout?15:09
*** baoli__ has joined #openstack-infra15:10
jeblair(we would not use it in nodepool because of its own retry and task ordering mechanisms)15:10
timrcSpamapS, At least mock is relatively stable and part of the py34 stdlib...15:10
jeblair(but it seems great for ansible)15:10
mordred++15:10
TheJuliamordred: retry would imply wait, but one may just want to wait.15:10
mordredTheJulia: ++15:11
*** mpaolino has joined #openstack-infra15:11
mordredworks for me!15:11
TheJulia"Oh, it failed... well, okay we're done here" as opposed to "Oh... transient failure... uhh.. lets try this again."15:11
TheJuliaExcellent!15:11
*** nelsnelson has joined #openstack-infra15:13
*** mtanino has joined #openstack-infra15:13
timrcSpamapS, Not sure I like the idea of a *unit test* succeeding or failing because of an external library.  Besides, I'm not testing novaclient, I'm testing the shade rebuild function.  Since mordred +2'd both of changes, I'll let him decide what should be done :P15:16
timrcmordred, Two shade tests, two different styles: https://review.openstack.org/#/c/160155 and https://review.openstack.org/#/c/16106715:17
mordred /me hides15:18
timrc(granted mock is an external library, but like I said, its api seems to be stable and is part of py34 stdlib if I'm not mistaken)15:18
*** tnovacik has quit IRC15:18
jeblairtimrc: there was a conversation in here about it a day or two ago... lemme find it15:18
*** timcline has joined #openstack-infra15:18
timrcexternal library for py27 I meant*15:19
*** ChuckC has quit IRC15:19
jeblairtimrc: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-03-02.log15:19
jeblairtimrc: starting at 2015-03-02T17:40:4015:19
*** tsg has quit IRC15:20
*** yamahata has joined #openstack-infra15:20
timrcjeblair, Thanks.15:21
*** timcline has quit IRC15:21
*** wenlock has joined #openstack-infra15:21
*** timcline has joined #openstack-infra15:22
*** krtaylor has joined #openstack-infra15:25
*** dimtruck has joined #openstack-infra15:26
*** marun has joined #openstack-infra15:26
openstackgerritJames E. Blair proposed openstack-infra/jenkins-job-builder: Fix typo in format string  https://review.openstack.org/16175015:28
*** hockeynut has quit IRC15:31
*** hockeynut has joined #openstack-infra15:32
*** armax has quit IRC15:33
*** armax has joined #openstack-infra15:34
timrcSo mordred SpamapS and whomever else is interested.  I'd prefer the unit tests to test nothing more than the success and failure paths of the shade method.  Since these methods are wrappers around other clients, I think that testing for regressions in those APIs is probably best done with integration tests, e.g. actually using the shade methods to do Real Things.15:35
timrcBut I am low man on this totem pole so I'll defer to the wisdom of my elders :)15:35
*** ghostpl_ has quit IRC15:36
*** esker has joined #openstack-infra15:38
*** garyk has quit IRC15:40
*** ghostpl_ has joined #openstack-infra15:42
TheJuliatimrc: That does seem reasonable.  My one concern would be a transitory issue causing the test to fail when doing Real Things, but really that occuring would just be highlighting a defficency that needs to be addressed15:43
openstackgerritMerged openstack-infra/jenkins-job-builder: Support excludes when recursively processing paths  https://review.openstack.org/6889515:43
anteaya2015-03-05T11:31:16  <pc_m> Had a question... if a commit has been approved, but during verify fails Jenkins (passed during check pipeline), how is that handled?15:44
anteayapc_m: the answer is `recheck` the reverify command has been removed15:44
clarkbI much prefer Zuul's brand of testing with fakes over mocks. Its you cover a larger surface of real world failures without integration test complexity15:44
anteayaa15:45
anteaya2015-03-05T03:33:00  <HenryG> Hey I wonder if anyone can help me to diagnose why check-neutron-dsvm-functional jobs for stable/juno are suddenly failing?15:45
anteaya2015-03-05T03:33:46  <HenryG> http://logs.openstack.org/25/161325/2/check/check-neutron-dsvm-functional/527457f/console.html#_2015-03-05_01_32_25_38815:45
anteayaHenryG: are you still having issues?15:45
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add microsecond columns to the test_runs table  https://review.openstack.org/16005715:45
pc_manteaya: Yeah, found out that I needed to do that vs reverify.15:45
clarkbalso mocks have a way of becoming assert True15:46
HenryGanteaya: Thanks for checking in. We think this is the fix: https://review.openstack.org/16156915:46
*** esker has quit IRC15:46
jeblairclarkb: "test that the python interpreter returns from a function invocation"15:46
* anteaya reviews 16156915:47
openstackgerritMerged openstack-infra/jenkins-job-builder: Check external documentation links are valid  https://review.openstack.org/15887115:48
openstackgerritMerged openstack-infra/jenkins-job-builder: Use sphinx extlinks to shorten links to Jenkins wiki  https://review.openstack.org/15875615:49
openstackgerritMerged openstack-infra/jenkins-job-builder: Change test folder name to 'fixture' for consistency  https://review.openstack.org/15266415:49
*** funzo has quit IRC15:49
clarkbI am going to work on listing the leaked ports in hpcloud now15:50
*** krtaylor has quit IRC15:50
*** rkukura has joined #openstack-infra15:50
openstackgerritMerged openstack-infra/jenkins-job-builder: Move project matrix documentation into a test.  https://review.openstack.org/13797215:50
openstackgerritMerged openstack-infra/jenkins-job-builder: Provide info on number of jobs processed for test, update and delete-all  https://review.openstack.org/15223315:51
openstackgerritMerged openstack-infra/jenkins-job-builder: Provide a more complete yaml reference in the docs for CMake module.  https://review.openstack.org/15224615:51
clarkbjesusaurus taught me about device_owner in neutron identifying the service that uses the port. So you can neutron port-list -c id -c device_id -c device_owner | grep compute to get all the ports used by compute nodes15:51
*** bswartz has quit IRC15:52
anteayaHenryG: were did the BRANCH variable come from? I can't seem to find a BRANCH variable in devstack-gate or devstack15:54
*** skolekonov has quit IRC15:55
openstackgerritJames E. Blair proposed stackforge/gertty: Fix keymap substitution  https://review.openstack.org/16175715:55
jeblairtchaypo: ^15:55
mkj_wkYay, finally Jenkins success on a patch set that AFAICT resolves every review point... ☺15:56
xnoxwhy non-OSI approved license terms are used by openstack upstream published projects on github?15:56
xnoxpleia2: ^15:56
jeblairxnox: can you be more specific?15:57
anteayalucasagomes: this is a good place to start: https://review.openstack.org/#/c/159867/4 note this adds a non-voting job to their repo, the best is to add a brand new job to the experiemental pipeline first and then once the build is stable to move it to non-voting15:57
xnoxjeblair: XStatic-* declaring license "(same as $foo)" in the pypi metadata / setup.py -> instead of using something which is in https://spdx.org/licenses/ e.g. "MIT"15:58
xnoxjeblair: from my point of view, i'm told not to use XStatic-* as it does not have a valid license in the source =(15:58
anteayaaarefiev_: if the patch is in merge conflict, which https://review.openstack.org/#/c/161677/ is you need to rebase the patch15:59
lucasagomesanteaya, thanks much!16:00
anteayaBobBall: the error AttributeError: 'NoneType' object has no attribute 'name' is ringing a bell, we might have encountered that ourselves in the last 24 hours, might a grep of the channel logs yield a result?16:00
jeblairxnox: i would recommend talking to the horizon developers who are the ones maintaining those projects.  i think there is an argument to be made that the license is clear, but if not having the exact value is causing you problems, perhaps they can change it.16:00
anteayalucasagomes: do your best and put something on gerrit, we can go from there16:01
*** garyh has joined #openstack-infra16:01
jeblairxnox: there is an #openstack-horizon channel, or you could use the openstack-dev mailing list16:01
*** baoli__ has quit IRC16:01
lucasagomesanteaya, fair enough! I will put a patch up soonish16:01
lucasagomesthanks16:01
xnoxjeblair: let me do that, on irc.16:01
anteayapc_m: did you get teh vpnaas coverage test working again?16:01
*** MarkAtwood has joined #openstack-infra16:01
aarefiev_anteaya: I've rebased it, it is up to date16:01
*** krtaylor has joined #openstack-infra16:01
*** pc_m has left #openstack-infra16:02
*** dboik_ has joined #openstack-infra16:02
*** baoli has joined #openstack-infra16:02
anteayaaarefiev_: great16:02
*** pc_m has joined #openstack-infra16:02
*** kmartin has joined #openstack-infra16:03
pc_manteaya: Not sure. On one system it works, on the other it doesn't will dig into it later. Have a bigger fish to fry right now :(16:03
anteayapc_m: happy frying16:03
anteayacome find us if you need us16:03
clarkbjeblair: mordred et al http://paste.openstack.org/show/189314/ I think that is a list of leaked ports16:04
clarkboddly some hosts have duplicate ports, not sure why16:04
clarkbalso we have leaked ~89 ports16:05
*** aysyd has joined #openstack-infra16:05
jeblairclarkb: do those hosts still exist?16:05
*** dboik has quit IRC16:05
anteaya2015-03-05T11:33:56  <yolanda> here, i'm having continuos failures on precise for  puppet-gerrit, and don't seem to be related with my change16:05
clarkbjeblair: as far as I can tell no, I cross checked the output of `nova list` with the output of `neutron port-list -c id -c device_id -c device_owner | grep compute`16:05
anteayayolanda: are you still having issues?16:05
*** pcm_ has joined #openstack-infra16:06
clarkbjeblair: I am working to see if those multiple ports might be from routers or similar though16:06
*** dprince has quit IRC16:07
yolandahi anteaya, so looking at the error log, it was throwing lots of errors, but the error that made my test fail was legit16:07
*** funzo has joined #openstack-infra16:07
yolandaalthough that tons of errors and warnings are still present on the puppet log16:07
jeblairtteggel: are you around?  we seem to be leaking neutron ports in hpcloud -- and they aren't attatched to any current nova hosts...16:07
*** pc_m has quit IRC16:07
*** carl_baldwin has quit IRC16:08
openstackgerritThomas Morin proposed openstack-infra/project-config: Add networking-bagpipe-l2 project  https://review.openstack.org/16129016:08
anteayayolanda: okay16:09
anteayayolanda: yes wading through the errors to find the right one is a bit of a chore16:09
*** tiswanso has quit IRC16:09
timrcclarkb, jeblair Sorry I had a 3 year old that had to "show me something" (permanent marker drawings on my wood floors).  Fakes seem more appropriate to me if they originate in the project they're testing as opposed to one that simply wraps other things.16:09
anteayayolanda: good digging!16:09
aarefiev_anteaya: thanks for feedback16:09
timrcclarkb, I would assume novaclient, for example has its own test suite that uses fakes, and that those tests get run whenever there is a change.16:09
yolandaanteaya, yes, i didn't expect to see all these puppet errors so i got confused16:10
anteayaaarefiev_: sure16:10
yolandabut then i could find the real problem between all16:10
anteayayolanda: I can certainly understand that16:10
anteayaway to go!16:10
yolandaanteaya, i've been digging a bit on puppet-* modules to make them more consumable16:10
*** tiswanso has joined #openstack-infra16:10
timrcFor something like shade that wraps other things, I think it's more appropriate to use unit tests to test code paths, and integration tests to sniff out things like api regressions.16:10
anteayayolanda: oh thank you!16:10
*** ashleighfarnham has joined #openstack-infra16:11
clarkbunittests are great, mocks not so much16:11
* anteaya again wonders what the difference is between mocks and fakes16:12
clarkbat least that has been my experience, you end up with mock parroting exactly what you tell it which is "go forth and pass" and it doesnt really expose problems16:12
*** bhunter71 has joined #openstack-infra16:13
yolandaah, clarkb... now that you are around, i'd like your feedback for the enforce nodepool user chane16:13
yolandachange16:13
yolandalooks as there is a bit of polemic there: https://review.openstack.org/16033616:13
sdaguejeblair: is there a storyboard entry for Depends-On causing change stalls like https://review.openstack.org/#/c/160407/16:13
clarkbanteaya fakes are usually simple but otherwise functional replacements of a thing for testing16:13
sdaguethey end up in a state where a recheck won't progress them to the gate16:14
anteayaclarkb: thank you16:14
clarkbanteaya mocks are overrides for methods/obejects that mock behavior by eg being told to return X on this function call16:14
timrcclarkb, Mocking external calls can catch logic errors.16:14
*** _nadya_ has quit IRC16:14
timrcclarkb, In the function you're actually testing.16:14
timrcAnd can ensure code coverage.16:14
jeblairclarkb: i looked up some of those instance ids in the logs, and they were associated with nodes that progressed through the full lifecycle without error16:15
clarkbjeblair huh, is that true for the hosts with multiple ports?16:15
*** ashleighfarnham has quit IRC16:15
anteayaclarkb timrc cool, thanks that helps me get a perspective16:15
clarkbjeblair: those are the ones I am most worried about because they seem special16:15
*** rkukura has quit IRC16:15
jeblairclarkb: oh, sorry, no.  just singletons16:16
jeblairsdague: you filed it: https://storyboard.openstack.org/#!/story/200016916:17
*** ChuckC has joined #openstack-infra16:17
*** imcsk8 has quit IRC16:17
clarkbjeblair: so far I am unable to map any of the multiple port hosts to our router list16:17
*** imcsk8 has joined #openstack-infra16:17
*** MaxV has joined #openstack-infra16:17
jeblairsdague: well, the description doesn't exactly match, but i think it's a very similar, or possibly the same, problem16:17
*** ChuckC has quit IRC16:18
*** ChuckC has joined #openstack-infra16:18
sdagueoh, ok. The fact that storyboard doesn't show you your filed bugs by default makes me lose stuff like this16:18
clarkbtimrc: the problem is that mocks are typically far more forgiving than the mocked code or even fakes16:18
jeblairsdague: workflow +1 will progress it; recheck won't because the jenkins vote doesn't change, which means gerrit doesn't include an approved section in the emitted event16:19
*** mrmartin has quit IRC16:19
sdagueok, lots of people seem to be getting stuck on this one now16:19
clarkbtimrc: so yes you can use them to exercise code paths, but you have to be very careful you don't end up in a trap of everything passes good to go when in reality its passing because you aren't testing anthing16:19
sdagueit's come up a few times in channels the last couple of days16:19
sdaguecontributors keep running recheck because they don't understand why changes are stuck16:19
*** deepakcs has quit IRC16:20
timrcclarkb, Integration the test infrastructure of another project also has its dubious aspects as well though and that should also be considered.16:21
timrcEr integrating*16:21
clarkbsdague: jeblair zaro did write a change to upstream gerrit to change that behavior16:21
clarkbwe should see if upstream has had any feedback yet16:21
*** stevemar has joined #openstack-infra16:21
*** unicell1 has joined #openstack-infra16:22
*** unicell has quit IRC16:22
timrcI think we have to agree on the intent / what we want to accomplish in a shade unit test.  To me, they should simply exercise code paths and flag logic error regressions because they are simply calling out to other things.  Those things should all have their own test suites and we should be doing integration testing with them separately from unit testing to avoid importing their test infrastructure.16:22
jeblairclarkb: none of the multi-port hosts show up in our logs16:23
timrcIf a result of a function in shade is wholly calculated in shade then I think there is value in testing the value of outputs.16:23
clarkbjeblair: sdague https://gerrit-review.googlesource.com/#/c/65530/16:23
timrcvalue in testing the value... hm :)16:24
timrc<-- articulate16:24
*** dkranz has quit IRC16:24
clarkbtimrc: so the issue with full on integration testing is that it is harder to debug, much easier to catch those problems in a simpler environment if you can (notice the recent push towards proper functional testing in openstack). Also, until very recently ~2-3weeks ago novaclient didn't have functional testing, just a few cli tests in the tempest runs and their unittests16:25
clarkbtimrc: so depending on eg novacleitn to do testing is :(16:25
*** ihrachyshka has quit IRC16:25
clarkbjeblair: ya and I don't see any of them in our router list16:26
jeblairclarkb: shall i start talking to the noc?16:26
clarkbjeblair: do you think we should maybe send this list to hpcloud before we delete anything and hope they can debug?16:26
clarkbjeblair: its probably worth at least asking if they know anything more than we do16:26
jeblairmaybe i should open a ticket?16:26
sdagueclarkb/timrc though contributing tests for behavior you want nailed down in novaclient would be ++16:27
sdaguenow that there is infrastructure to do more robust testing16:27
clarkbjeblair: I think so16:27
* mordred is in the middle of moving a test from python-openstackclient to os-client-config16:27
*** garyk has joined #openstack-infra16:28
clarkbjeblair: we aren't hurting for those ports right this second so I don't think we need to rush to delete these, getting answers if we can would be good16:28
mordredbecause testing that os-client-config does the right thing in the middle of the mocking in openstackclient turns out to be crazy strange16:28
mordredbut - we can lock down the bevhavior and contract in os-client-config and then trust in osc that it will actually, you know, work16:28
sdaguemordred: ++16:28
*** salv-orlando has quit IRC16:29
*** chuckC_ has joined #openstack-infra16:29
timrcsdague, Seems like testing is in much demand :)16:29
*** chuckC_ has quit IRC16:29
clarkbjeblair: I don't know if you saw but I was able to locally boot the qcow2 image that was giving hpcloud trouble yesterday16:30
clarkbjeblair: I am using whatever version of kvm and qemu come with ubuntu trusty which is probably newer than what hpcloud is running through16:30
*** ildikov has quit IRC16:30
clarkbI also confirmed md5sums checked out and I wasn't the benefit of some miraculous bit flip16:30
*** david-lyle has joined #openstack-infra16:30
*** dizquierdo has quit IRC16:30
jeblairclarkb: i wonder if it would be worth some more manual testing on hpcloud then....16:30
*** Bsony_ has quit IRC16:30
clarkbjeblair: its possible, at least to confirm its consistent there? maybe a second image upload will work and we can narrow focus to uploads? (mostly just hopeful something will become apparent)16:31
jeblairclarkb: yeah, and maybe we can poke around in the bb shell and find out what actually went wrong?16:32
clarkbmaybe? I don't think hpcloud gives you proper interactive console access16:32
clarkbbut maybe hpcloud support can debug if we point them at our image16:32
*** mfink_ has joined #openstack-infra16:33
*** vivcheri has quit IRC16:33
BobBallanteaya: Unfortunately the NoneType in the logs was for nodepool16:34
*** rkukura has joined #openstack-infra16:34
*** aarefiev_ has quit IRC16:34
*** aarefiev has joined #openstack-infra16:35
jeblairclarkb: the nova console thingy isn't interactive?16:36
clarkbjeblair: nope, it just dumps dmesg16:36
jeblairum.16:36
clarkbrackspace has that vncviewer which I don't think hpcloud supports16:36
*** dkranz has joined #openstack-infra16:36
mordredclarkb: hp totally has interactive console16:36
mordredI used it 2 days ago16:37
clarkboh nevermind then16:37
clarkbmordred: was that added in the 1.1 cloud? 1.0 definitely did not have it16:37
mordredthere are two different tabs - one is the dumped log - one is the interactive console16:37
mordredyeah16:37
mordredit's actually more betterer than the rax one - much more responsive16:37
clarkbcool, I will figure out glanceclient to upload this image16:37
*** salv-orlando has joined #openstack-infra16:39
*** AJaeger has quit IRC16:39
BobBallAnyone know how to fix a Zuul Internal Server Error where one of the jobs has changeish.project == None? http://paste.openstack.org/show/189338/16:39
BobBallOr, failing how to fix it, how it happened so I can avoid it in future? (does enqueue work? or is that causing this?)16:39
*** scheuran has quit IRC16:40
jeblairBobBall: i haven't seen that locally so haven't been able to debug.  we use enqueue all the time16:40
jeblairBobBall: i think asselin has seen it though16:41
BobBallDrat.  Any way to repair it? or is it a case of killing zuul and restarting every couple of days?16:41
*** baoli has quit IRC16:42
jeblairBobBall: i honestly don't know.  any information you can provide would be great.  if you have any theories, i'm happy to work through them with you.16:42
clarkbis enqueue not being given a complete set of data like the project field?16:42
clarkbI thought it was a required arg16:42
BobBallI think zuul requires everything...16:42
BobBallzuul enqueue --trigger gerrit --pipeline silent --project openstack/nova --change 159106,416:42
timrcI've used vnc with hp public cloud...16:42
BobBallThat's all I've ever used to enqueue16:43
*** rwsu-afk is now known as rwsu16:43
anteayaBobBall: ah sorry about that16:43
mordredclarkb: I have an upload script if it's useful16:43
clarkbmordred: no hpcloud is the easy one so should be fine16:43
jeblairBobBall: yeah, that lgtm.16:43
clarkbmordred: if this was rax I would probably ask for it :)16:43
*** achuprin_ has quit IRC16:44
BobBalljeblair: I did see http://paste.openstack.org/show/189346/ as an earlier exception - could that be the cause?16:44
mordredclarkb: ++16:44
BobBallIt happens lots so I assumed it wasn't the cause...16:44
jeblairBobBall: yeah, i think it's unrelated16:45
*** carl_baldwin has joined #openstack-infra16:45
jeblairBobBall: oh! that's the zuul client show running-jobs command16:45
jeblairi thought that was in zuul server16:45
jeblairah, but the error is in the server servicing that command16:46
jeblairi think i'm caught up now.  :)16:46
*** tkelsey has joined #openstack-infra16:46
jeblairBobBall: is that consistent?  ie, if you run it again, do you get the same error?16:46
BobBallThat was the simplest way I thought of to produce it.  debug.log shows it regularly.16:46
BobBallyes16:46
BobBall100%16:46
asselinBobBall, jeblair yeah, we run into that. I've seen it 3 times.16:46
*** tsg has joined #openstack-infra16:46
BobBallHow have you been fixing it asselin ?16:47
asselinBobBall, right now, no fix. just restart zuul16:47
asselinBobBall, patrickeast also ran into it16:47
asselinBobBall, what projects is your zuul monitoring?16:47
*** unicell1 has quit IRC16:47
BobBallkill it rather than service restart, right? service restart seems to hang forever16:47
BobBallasselin: nova + sandbox https://github.com/citrix-openstack/os-ext-testing/blob/master/etc/zuul/layout.yaml16:48
asselinBobBall, sudo service zuul stop, then start. I didn't use restart16:48
*** hamzy has quit IRC16:48
*** hamzy has joined #openstack-infra16:48
asselinBobBall, ok, patrick east and I monitor cinder, so seems less use of cross project dependencies.16:49
clarkbjogo: I have very little interest in running two neutron multinode jobs until someone is working on one fwiw. We can make the current job DVR if that is what people want to debug, but I don't think having two failing jobs with no one looking at them is helpful16:49
*** krtaylor has quit IRC16:49
BobBallWell I wonder if it is a cross project dependency for a project we don't monitor?16:49
*** yamahata has quit IRC16:50
clarkbBobBall: service restart performs a graceful shutdown which stops processing events and only queues them, waits for all jobs to finish, writes out a pickle state, then restarts loading the pick back to start processing the queue backlog16:50
jeblairasselin, BobBall: does "Unable to find project for change" ever show up in your debug logs?16:50
asselinBobBall, maybe...16:50
asselinjeblair, looking16:50
BobBallno16:50
*** baoli has joined #openstack-infra16:51
jeblair(and are you logging at debug level?)16:51
jeblairoh, actually that's a warning16:51
*** mpaolino has quit IRC16:51
jeblairso should show up in normal logs16:51
BobBallyes16:51
BobBalldebug.log is indeed logging DEBUG.16:51
asselinjeblair, also no16:52
*** achanda has joined #openstack-infra16:52
*** doude has quit IRC16:52
BobBallAny way to dump more info manually? to see which job is causing this issue?16:53
*** amotoki has quit IRC16:53
BobBallIf not I can add more debugging so the next time it comes up we have a print of any details we have16:53
*** doude has joined #openstack-infra16:53
asselinBobBall, look at your logs and see when it first starting happening. The change should logged be around that time...16:54
openstackgerritDevananda van der Veen proposed openstack-infra/reviewstats: Overdue update for the ironic project group  https://review.openstack.org/15968016:55
BobBallyes but there are lots of changes and I don't have a start-time for when I know things were good other than a few hours before16:55
BobBalljeblair: We do have Project openstack/horizon not found but I'm assuming that's all expected16:56
asselinBobBall, I never tried that enqueue command. I will try it next time.16:56
*** doude has quit IRC16:57
BobBallok so that's not the culprit then - good to know.16:58
*** tiswanso has quit IRC16:58
*** otter768 has joined #openstack-infra16:59
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add microsecond columns to the test_runs table  https://review.openstack.org/16005716:59
*** Bsony has joined #openstack-infra16:59
*** esker has joined #openstack-infra17:00
*** esker has quit IRC17:00
*** achuprin_ has joined #openstack-infra17:00
*** esker has joined #openstack-infra17:00
asselinBobBall, in my case, it seems a patch starts to run, then I get stack traces that come from the zuul ui filling up the log files. I will find my log paste.17:00
*** krtaylor has joined #openstack-infra17:01
*** dannywilson has joined #openstack-infra17:01
marunclarkb: is BRANCH a valid variable in the context of project-config job definitions? https://review.openstack.org/#/c/161569/1/jenkins/jobs/neutron.yaml17:02
openstackgerritDoug Hellmann proposed openstack-infra/project-config: import aeromancer to stackforge  https://review.openstack.org/16144717:03
*** otter768 has quit IRC17:03
*** jlanoux has quit IRC17:04
clarkbmarun no see line 2317:04
*** yamamoto has quit IRC17:04
*** bdpayne has joined #openstack-infra17:04
*** nfedotov has quit IRC17:05
*** tiswanso has joined #openstack-infra17:05
*** jvrbanac has joined #openstack-infra17:05
openstackgerritMerged openstack/requirements: Add semantic_version to global-requirements  https://review.openstack.org/15146617:05
*** achanda has quit IRC17:05
*** markus_z has quit IRC17:05
*** shivharis has joined #openstack-infra17:06
shivharisfungi: hi17:07
*** koolhead17 has joined #openstack-infra17:07
anteayashivharis: you are going to be waiting a while he is on holidays17:07
openstackgerritPaul Belanger proposed openstack-infra/jeepyb: Add global flag to control replication  https://review.openstack.org/16178617:08
*** jistr has quit IRC17:08
*** baoli has quit IRC17:08
shivharisanteaya: oops, thanks for heads up.17:08
*** luqas__ has quit IRC17:08
shivharisanteaya: I have a question for you...17:08
anteayaclarkb: can you expand, I too am trying to understand if BRANCH is a valid variable and so far am not making the correct decison17:09
anteayashivharis: how about you just ask the question for the channel and someone will speak up if they can help17:09
*** baoli has joined #openstack-infra17:09
anteayait is usually a more effcient use of everyone's time17:09
*** jaypipes has joined #openstack-infra17:09
shivharisanteaya: actually u are the boss on the question i have to ask17:09
asselinBobBall, here's an example of what we run into: http://paste.openstack.org/show/189384/17:10
shivharisanteaya: i need to get voting privileges for Brocade VDX CI, we have been voting for quite some time, with '0'17:11
*** bswartz has joined #openstack-infra17:11
*** dannywilson has quit IRC17:11
*** achuprin_ has quit IRC17:11
clarkbline 23 sets the value of BRANCH_OVERRIDE17:11
anteayashivharis: what repo do you want to verify?17:11
rcarrillocruzmordred: helo helo, if you have some time to review https://review.openstack.org/#/c/158463/ that'd be great, got already several +1s, when it lands i'll add up moar func tests17:11
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Reintroduce "Temporarily disable swift-dsvm-functional for Icehouse"  https://review.openstack.org/15994317:11
shivharisanteaya: neutron17:11
*** dannywilson has joined #openstack-infra17:11
clarkbso without updating that BRANCH doesnt exist17:11
anteayashivharis: you need to talk to mestery17:11
mordredrcarrillocruz: you have a patch up with a dvsm job too, right?17:12
clarkbI need to figure out what the actual issue is to help debug17:12
mtreinishclarkb, mordred: if you get a sec. I want to land: https://review.openstack.org/#/c/160514/ before I push the next subunit2sql release17:12
shivharisanteaya: ah.. ok, will do, thanks for the pointer17:12
anteayashivharis: he is the only one who can make decisons about who verifies on neutron17:12
gordchmm.. is there a bug for grenade tests in juno? specifically  n-api not starting? https://review.openstack.org/#/c/161631/17:12
shivharisanteaya: thanks17:12
anteayashivharis: you're welcome17:12
*** yamahata has joined #openstack-infra17:12
rcarrillocruzmordred: yep, i actually put up the new shiny superuseful Depends-On zuul feature on the dsvm change (thanks to AJaeger for reminding me)...17:13
anteayaclarkb: there is a link to the logs with the failure in the commit message17:13
rcarrillocruzhttps://review.openstack.org/#/c/160190/17:13
rcarrillocruz^17:13
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add non-voting tempest dsvm job for Ironic  https://review.openstack.org/16179017:14
anteayahttp://logs.openstack.org/73/161273/1/check/check-neutron-dsvm-functional/56af914/console.html#_2015-03-05_00_26_21_12917:14
*** ildikov has joined #openstack-infra17:14
jogoclarkb: I agree, lets start with one, as long as someone is working in that neutron multinode job17:15
*** dtantsur is now known as dtantsur|afk17:16
*** erlon has joined #openstack-infra17:16
*** mpavone has quit IRC17:18
clarkbanteaya: marun I think you want to test both BRANCH_OVERRIDE and ZUUL_BRANCH, I will leave a comment17:18
cineramapleia2: BAM, got a working install from puppet17:18
marunclarkb: danke17:18
*** yamahata has quit IRC17:18
*** devvesa has quit IRC17:18
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add non-voting tempest dsvm job for Ironic  https://review.openstack.org/16179017:18
cineramapleia2: we just needed to change one thing in our usage of the wildfly module and ship over the standalone.xml you had from the download17:19
*** yamahata has joined #openstack-infra17:19
anteayaclarkb: thanks17:19
cineramapleia2: i'm going to clean this up and then maybe i should submit it as dependent on your change? need to do some cleanups probably but i'd like to have the progress there17:19
*** patrickeast has joined #openstack-infra17:20
clarkbgordc: not that I am aware of17:21
openstackgerritArmando Migliaccio proposed openstack-infra/project-config: Fix the Neutron functional job for stable branches  https://review.openstack.org/16156917:21
pcm_Question: For functional tests, should testr_results.html show what modules have been processed?17:21
anteayaclarkb: thanks I think that makes more sense to me17:22
gordcclarkb: hmm... will debug after lunch. seems to be only a juno thing.17:22
*** Longgeek has quit IRC17:22
clarkbpcm_: I think it should show all the tests that ran with pass/failure info17:22
pcm_clarkb: As part of https://review.openstack.org/#/c/158389/ my objectives were to have one functional job run tests in tests/functional/ and in tests/functional/openswan, and another job (experimental) run in tests/functional/ and tests/functional/strongswan.17:24
mordredrcarrillocruz: how does that jenkins job call "tox -efunc" ?17:24
pcm_clarkb: https://review.openstack.org/#/c/161714/ was set up to enable gate hook for 2nd job and test it out.17:24
zaromorning17:25
pcm_clarkb: Waiting for experimental run, but on the first job, it doesn't seem like it ran any tests.17:25
pcm_clarkb: Did I make a mistake in specifying the file filtering?17:25
clarkbpcm_: http://logs.openstack.org/14/161714/2/check/check-neutron-vpnaas-dsvm-functional/fea9e69/ that test?17:25
*** gulic has quit IRC17:25
pcm_clarkb: yes17:26
rcarrillocruzmordred: by calling the post-hook script https://review.openstack.org/#/c/158463/14/shade/tests/functional/hooks/post_test_hook.sh17:26
mordredrcarrillocruz: ah. gotcha17:26
pcm_clarkb: Wondering if mistake here... https://review.openstack.org/#/c/158389/7/zuul/layout.yaml17:26
*** Ala has quit IRC17:26
mordredrcarrillocruz: so - I think both look great - and that we should land them in the opposite order17:26
*** dprince has joined #openstack-infra17:26
*** unicell has joined #openstack-infra17:27
mordredrcarrillocruz: that is, I think we should land the job to run functional tests, and then the next patch we will _have_ to land is the patch that adds the functional tests17:27
mordredbut also, it will prove that the functional tests we added actually work - and if they don't, we can't land anything else until they do17:27
pcm_clarkb: I made comment changes in the test modules so the tests would run.17:27
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add gate-tempest-dsvm-full-trove  https://review.openstack.org/16179417:27
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add trove tempest job to trove and devstack  https://review.openstack.org/16179517:27
openstackgerritJames E. Blair proposed openstack-infra/project-config: Make trove tempest job voting  https://review.openstack.org/16179617:27
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate: Remove trove from default config  https://review.openstack.org/16179717:27
clarkbpcm_: the jenkins job ran so the zuul fitlers should be fine17:28
rcarrillocruzok, let me reverse the depends-on then17:28
*** mayurig has joined #openstack-infra17:28
pcm_clarkb: Trying to ensure that it runs test on test_openswan_driver.py17:28
*** hdd_ has joined #openstack-infra17:28
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add non-voting tempest dsvm job for Ironic  https://review.openstack.org/16179017:28
clarkbpcm_: so all those zuul filters do is decide when to run that jenkins job17:29
clarkbpcm_: the job ran so they are working17:29
*** ashleighfarnham has joined #openstack-infra17:29
clarkbthey have no impact on what testr is doing17:29
*** annegentle has quit IRC17:29
jeblairSlickNik, sdague: ^ see those changes to start moving trove out of the default tempest configuration (https://review.openstack.org/161797 is the tip of the stack)17:29
*** achuprin_ has joined #openstack-infra17:29
clarkbpcm_: https://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/neutron-vpnaas.yaml#n28 that is what decides what tests testr runs17:30
pcm_clarkb: Hmmm, so maybe a worse problem... The goal was to have certain tests run, when running one job and different tests for the other (as they require different setups)17:30
pcm_clarkb: So is there a way to specify the sub-directories to run tests on, or do we need to have different roots for each test job?17:31
*** rkukura has joined #openstack-infra17:31
clarkbpcm_: there is but you do that in your job not in zuul17:31
*** krtaylor has quit IRC17:31
*** tonytan4ever has quit IRC17:31
sdaguejeblair: why only on gate? shouldn't it be only on check if picking one?17:32
pcm_clarkb: Are there any pointers on how I do that in the job (select specific dirs to run the test on)?17:32
openstackgerritJames E. Blair proposed openstack-infra/project-config: Make trove tempest job voting  https://review.openstack.org/16179617:32
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add trove tempest job to trove, tempest, devstack  https://review.openstack.org/16179517:32
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/project-config: Add functional job for Shade and enable it on check and gate pipelines  https://review.openstack.org/16019017:33
clarkbpcm_: it looks like your tests are set up to switch based on OS_TEST_PATH so just set that as appropriate17:33
jeblairsdague: because i got that backwards? :)17:33
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Add initial compute functional tests to Shade  https://review.openstack.org/15846317:34
openstackgerritMerged openstack-infra/jenkins-job-builder: Skip duplicate input files  https://review.openstack.org/14654417:34
openstackgerritJames E. Blair proposed openstack-infra/project-config: Make trove tempest job voting  https://review.openstack.org/16179617:34
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add trove tempest job to trove, tempest, devstack  https://review.openstack.org/16179517:34
jeblairsdague: fixed17:34
BobBallasselin: We don't seem to see it processinga  change immediately before the failures start17:34
openstackgerritMerged openstack-infra/jenkins-job-builder: Fix typo in format string  https://review.openstack.org/16175017:35
pcm_clarkb: So set that in tox.ini?17:35
clarkbpcm_: or in whatever calls tox.ini17:35
clarkbpcm_: tox.ini is probably good if you have different targets17:35
clarkbmordred: ok, so I was debating a rant yesterday on the clients not sure I am going to avoid one today17:36
clarkbmordred: 400 Bad Request: Invalid container format 'None' for image. (HTTP 400) <- this after it uploaded the image17:36
clarkbI don't even17:36
BobBall+17:36
EmilienMjeblair: I'm volunteer to be a mentor part of Outreach program for women, and I'm looking at mentoring someone in OpenStack automation (Puppet probably) - I was wondering if OpenStack infra has some Puppet work that could fit with an project (ref: https://wiki.openstack.org/wiki/Outreachy/Ideas )17:36
pcm_So either in post_test_hook or in tox.ini (and use different command)17:36
clarkbpcm_: ya17:36
mordredclarkb: sigh17:36
sdaguejeblair: hmmm you keep changeing this while I'm commenting so gerrit doesn't let me vote at the end17:36
*** ghostpl_ has quit IRC17:37
jeblairsdague: i will take a 5 minute recess :)17:37
* clarkb tries again this time looking into the future for the list of required parmeters17:37
*** krtaylor has joined #openstack-infra17:37
pcm_clarkb: So there's no way to have shared tests, unless the shared test is in another subdirectory, right?17:37
sdagueok, commented, getting out for lunch and errands now, back later17:37
jeblairEmilienM: what's the timeframe?17:38
*** shivharis has quit IRC17:38
pcm_clarkb: tests/functional/openswan, tests/functional/strongswan, tests/functional/common17:38
EmilienMjeblair: a project for the May 2015 to August 2015 - 40h/week17:38
*** carl_baldwin_ has joined #openstack-infra17:38
clarkbpcm_: I don't follow the question17:38
clarkbpcm_: you have the freedom to do what you like in these jobs for the most part17:39
pcm_clarkb: If I have tests that should be run for both jobs... I take it I need to use a separate subdirectory (and not the parent directory)17:39
EmilienMjeblair: in the last round, we had an idea with hogepodge ( https://wiki.openstack.org/wiki/Outreachy/Ideas#Community ) - not sure it's still useful/doable.17:39
clarkbpcm_: not necessarily17:39
clarkbpcm_: the subset of tests that cannot run could be skipped17:40
clarkbpcm_: there are many options here17:40
*** carl_baldwin has quit IRC17:40
pcm_clarkb: My initial thoughts were to have common tests in parent dir (tests/functional). Can you elaborate on how I could select those to run for both jobs?17:40
* pcm_ I'm probably missing something simple17:41
EmilienMjeblair: I should probably use the mailing list - sorry to bother you.17:41
clarkbpcm_: there are skip tests exceptions you could trigger if pieces are missing17:41
clarkbpcm_: or you could do what ou describe and have tests/common tests/special1 tests/special217:41
mordredoh! look - we seem to have gotten some snow17:42
mordred(more snow)17:42
*** carl_baldwin has joined #openstack-infra17:42
anteayamoar snow17:42
pleia2cinerama: probably best to clean up my change (we don't want to land it as-is since it's so broken), you can submit your change on top of mine, or just comment inline and I can fix it up17:42
anteayayou can't have too much17:43
pcm_clarkb: are you taking about skipping tests in the actual test module?17:43
clarkbpcm_: yes17:43
jeblairEmilienM: no problem.  list is probably a good idea actually.  i think we might have some stuff.17:43
cineramapleia2: oh, ok. that was the preferred option for me actually so i'll just do it that way17:43
pcm_clarkb: In this case, some tests can only run, if the environment is set up a certain way17:43
*** carl_baldwin_ has quit IRC17:43
pcm_clarkb: For one job, I must set IPSEC_PACKAGE=strongswan to force devstack to setup differently.17:43
clarkbpcm_: right17:44
pcm_clarkb: Are you saying I can skip a test based on some environment setup?17:44
*** timcline has quit IRC17:44
clarkbpcm_: yes, you can skip based on just about anything17:44
*** timcline has joined #openstack-infra17:44
clarkbfor example db related tests tend to check for an available database and skip otherwise17:45
*** unicell has quit IRC17:46
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add non-voting tempest dsvm job for Ironic  https://review.openstack.org/16179017:46
openstackgerritMerged openstack-infra/storyboard: Fix db Branch and Milestone models  https://review.openstack.org/16121317:46
*** unicell1 has joined #openstack-infra17:46
*** mrunge has quit IRC17:47
pcm_clarkb: Can you point to an example, in case I go that way?17:47
cineramapleia2: though i think what i'm doing now is a bit of a band-aid; the wildfly module would like to control the standalone.xml in the configuration directory so i need to see if i can teach it to create it correctly17:48
*** baoli has quit IRC17:49
openstackgerritMerged openstack/requirements: Bump oslo.db version to 1.5.0  https://review.openstack.org/16073117:49
pleia2cinerama: the patch has evolved heavily :) since it's already non-working, shipping a standalone.xml as a change to this patch is perfectly fine17:49
clarkbok I have image uploaded working on booting a node now17:50
openstackgerritJames E. Blair proposed openstack-infra/project-config: Make trove tempest job voting  https://review.openstack.org/16179617:50
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add trove tempest job to trove and tempest  https://review.openstack.org/16179517:50
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add gate-tempest-dsvm-full-trove  https://review.openstack.org/16179417:50
clarkbmordred: oh you will like this17:50
mordredclarkb: I bet I won't17:50
clarkbmordred: that image upload that failed? it has an image in the glance registry17:50
pleia2cinerama: although, how are you going to modify the wildfly module here? we just ship that from upstream17:50
clarkbmordred: so not only did it explode due to me being unable to see the future I have to clean it up manually17:50
mordredyup17:50
pcm_clarkb: So option A) use three different directories, and have the two jobs specify different test commands, which select the different directories. Option B) use conditional test invocation triggered on the IPSEC_PACKAGE env variable, with no difference in tox commands. Does that sound right?17:51
cineramapleia2: oh not modifying the module itself, just exploring what options we have there since i am currently in "beat with hammer until it works" mode17:51
clarkbpcm_: ya17:51
cineramapleia2: vs "actually do it properly" mode17:51
*** james_li has joined #openstack-infra17:51
pleia2cinerama: ok, we do want to do it properly in the long run, so feel free to upload what works and we can iterate :)17:52
anteayaclarkb: you can't see the future? I'm shocked17:52
pcm_clarkb: Can you point me to examples on how to trigger a test based on an environment variable setting?17:52
clarkbpcm_: look in the nova database migration tests17:53
clarkbthey query the environment to decide whether to run or skip17:53
cineramapleia2: slapping that file in there has the disadvantage that the wildfly module may blart over it due to puppet's unique order of executing things17:53
james_liHi All, I enabled REQUIREMENTS_MODE=soft, but still see 'gate-solum-requirements' is failing:  https://review.openstack.org/#/c/160041/317:54
james_liis that normal?  ^^17:54
*** mpaolino has joined #openstack-infra17:54
pcm_clarkb: It seems like option B is less effort, but requires people to ensure they have the conditional test setup.17:55
pcm_clarkb: Any preference over the two options?17:55
pleia2cinerama: we'll just have to work out dependencies so the ordering goes right17:56
clarkbpcm_: I think triggering two distinct named things is likely simpler in the long run17:56
clarkbpcm_: so option A17:56
cineramapleia2: yup but as a proof of concept we have something that sorta mostly works17:56
clarkbjames_li: I think REQUIREMENTS_MODE=soft is a devstack thing? so won't affect the requirements job, just affects what requirements are synced into your projet17:57
pleia2cinerama: yep, so upload what you have now and we'll iterate until we get it right17:57
pleia2(that's what I've been doing :))17:58
clarkbok first attempt at booting this image on hpcloud results in {"message": "No valid host was found. ", "code": 500, "created": "2015-03-05T17:56:53Z"}17:58
clarkbI guess I should try again17:58
pleia2with 3 of us tracking this patch now, we'll get there17:58
*** harlowja_away is now known as harlowja17:58
*** ivar-lazzaro has joined #openstack-infra17:59
*** ivar-lazzaro has quit IRC17:59
james_liRequirement docker-py>=0.5.1 not in openstack/requirements17:59
james_li2015-03-02 21:36:41.760 | Build step 'Execute shell' marked build as failure17:59
james_liclarkb: but I saw 'gate-solum-requirements' is complaining ^^17:59
*** bswartz has quit IRC17:59
pcm_clarkb: I can't seem to find examples of the skip exception. Looking in nova functional test area.17:59
*** ivar-lazzaro has joined #openstack-infra17:59
james_liclarkb: so no way to avoid that?17:59
pcm_clarkb: Is there some string I can search on? Not sure what I'm looking for exactly.18:00
mordredclarkb: did you build with the right version of qemu?18:00
*** dannywilson has quit IRC18:00
clarkbjames_li: I don't think so, that job enforces the requirements but it appears you want to not enforce, so I think you remove that job then use requirements soft in devstack18:00
mordredclarkb: that's the error I get in hp when the qemu version is off18:00
clarkbmordred: this is the image that nodepool built and was failing to init before18:00
clarkbmordred: so I am pretty sure the version is correct18:00
mordredclarkb: ok18:00
clarkbmordred: though I should find out if I can get the version out of the image18:01
*** unicell1 has quit IRC18:01
*** unicell has joined #openstack-infra18:01
clarkbmordred: maybe this image used a version that some of hpcloud doesn't like in several ways? so depening on hypervisor you get a 500 or you fail to boot?18:01
james_liclarkb: when you say "use in devstack", you mean I enable it in my localrc file?18:02
clarkbjames_li: yes REQUIREMENTS_MODE=soft is a devstack thing iirc18:02
clarkbjames_li: it causes devstack to sync in requirements from global requirements while allowing additional deps18:02
clarkbjames_li: so that you can depend on things not in global requirements but otherwise be in sync18:02
mordredclarkb: wow18:03
james_liclarkb: got it, thanks!18:04
clarkbFYI I am starting the fourth workers on logstash worker nodes. we have been well behind and looking at logs all 4 were actually active previously18:04
clarkbjeblair: ^18:04
jogoclarkb: check-tempest-dsvm-neutron-aiopcpu-full just passed !18:05
clarkbjogo: nice! any idea if changes were made to target that?18:05
*** baoli has joined #openstack-infra18:05
openstackgerritMerged openstack-infra/elastic-recheck: Add query for nova context unit test bug 1428728  https://review.openstack.org/16177818:05
openstackbug 1428728 in OpenStack Compute (nova) "nova.tests.unit.test_context.ContextTestCase.test_store_when_no_overwrite intermittently fails with MismatchError" [Undecided,In progress] https://launchpad.net/bugs/1428728 - Assigned to Johannes Erdfelt (johannes.erdfelt)18:05
*** baoli has quit IRC18:06
jeblairclarkb: oh ok.18:06
*** hashar has joined #openstack-infra18:06
clarkbjeblair: I think that last time we had mass reboots fungi brought them all back up18:06
*** hashar is now known as hasharDinner18:06
clarkbjeblair: and apparently it was working so win I guess :)18:06
*** bswartz has joined #openstack-infra18:06
*** dboik_ has quit IRC18:06
pcm_clarkb: Can OS_TEST_PATH contain multiple paths?18:07
pcm_colon separated?18:07
*** dboik has joined #openstack-infra18:07
clarkbpcm_: I do not know, that appares to be a neutronism?18:07
*** baoli has joined #openstack-infra18:07
*** HeOS has quit IRC18:07
*** achanda has joined #openstack-infra18:08
clarkbI will also be grabbing the 20 node shortly to test my reboot hackaround18:08
clarkbmordred: ok second boot failed the same way so I dunno. Maybe I uploaded the image wrong?18:08
clarkbmordred: disk-format is qcow2 and container-format is bare18:08
pcm_clarkb: Thinking I could specify the shared area and the job specific area, if that is possible.18:08
*** baoli_ has joined #openstack-infra18:08
*** lucasagomes is now known as lucas-dinner18:09
*** radez_g0n3 is now known as radez18:09
*** Guest95893 has quit IRC18:10
pleia2clarkb: don't want to interrupt, but we should look at backup server dns when you have some time (alas, seems it never was created)18:10
openstackgerritJoe Gordon proposed openstack-infra/devstack-gate: If wheelhouse is pre-generated, set WHEELHOUSE in localrc  https://review.openstack.org/16093318:10
clarkbpleia2: did you run the commands?18:10
pleia2clarkb: I did, and I thought it ran fine18:10
clarkbpleia2: that dns.py script only outputs the rax dns client commands to run18:10
clarkbpleia2: it doesn't update dns itself18:10
pleia2clarkb: yep, I ran the rackdns rdns-create command18:11
clarkbkk18:11
pleia2both for ipv4 and v618:11
clarkbhrm18:11
*** baoli has quit IRC18:11
clarkbpleia2: is the reverse ptr in place?18:11
pleia2clarkb: actually, yes18:12
pleia2just not forward18:12
pleia223.253.166.5918:12
clarkbpleia2: so there should've been commands for reverse ptr and a second set for A/AAAA18:13
* pleia2 facepalm18:13
pleia2ok, I'll run the script again and actually finish it this time18:13
clarkbpleia2: but to make things even more confusing there should be a second set of all that for the old node too :)18:13
jogoclarkb: not sure18:13
pleia2clarkb: heh, right, I think that's why I got confused18:13
*** e0ne has quit IRC18:15
*** signed8bit has joined #openstack-infra18:17
pleia2clarkb: all better, thanks :)18:17
clarkbpleia2: cool18:18
*** jcoufal_ has quit IRC18:18
mordredclarkb: that is what it should be18:18
*** vilobhmm has joined #openstack-infra18:19
*** vilobhmm1 has joined #openstack-infra18:19
*** vilobhmm has quit IRC18:23
*** prad has quit IRC18:24
*** SumitNaiksatam has quit IRC18:24
jogonew high water mark for openstack/openstack  commits per day: 197 https://github.com/openstack/openstack/graphs/commit-activity18:24
*** prad has joined #openstack-infra18:24
pleia2nice18:25
clarkbwow we still got 134 changes in after the reboots18:25
*** teran_ has quit IRC18:26
*** cdent has quit IRC18:26
jeblairclarkb: and running at half-capacity for much of that.18:27
jogopleia2: we changed how that repo counts things https://review.openstack.org/#/c/154753/18:28
*** SumitNaiksatam has joined #openstack-infra18:28
pleia2jogo: *nods*18:29
jeblairjogo: i was hoping the results of this search would be, well, bigger: https://review.openstack.org/#/q/topic:big,n,z18:29
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor: Retry on EAI_AGAIN name resolution failures  https://review.openstack.org/16140918:29
openstackgerritGorka Eguileor proposed openstack/requirements: Bump rtslib version to have config persistence  https://review.openstack.org/16129118:29
anteayajeblair: ha ha ha18:30
jeblairclarkb: have you by any chance checked to see if nodepool has built a new devstack-trusty dib image and sucessfully booted it in hp?18:30
clarkbjeblair: I have not18:31
*** annegentle has joined #openstack-infra18:31
*** mpaolino has quit IRC18:31
jeblairi'll do that18:31
*** hdd_ has quit IRC18:31
*** tsg is now known as tsg_away18:32
*** ddieterly has quit IRC18:32
*** ddieterly has joined #openstack-infra18:32
jeblairclarkb: looks like it was ready a bit more than 2 hours ago, so any devstack-trusty hpcloud instances built since then should be using it18:32
clarkbjeblair: cool18:33
jeblairclarkb: yeah, and there are some recently built ones in use, so i think the new version of that image works18:34
*** MarkAtwood has quit IRC18:35
jogojeblair: hehe18:37
*** tonytan4ever has joined #openstack-infra18:37
*** ssam2 has quit IRC18:39
*** claudiub has joined #openstack-infra18:41
*** mrmartin has joined #openstack-infra18:42
*** dizquierdo has joined #openstack-infra18:42
*** ddieterly has quit IRC18:43
*** dtantsur|afk has quit IRC18:43
*** ddieterl_ has joined #openstack-infra18:43
*** SumitNaiksatam has quit IRC18:43
*** dboik_ has joined #openstack-infra18:44
*** dboik_ has quit IRC18:45
openstackgerritSteve Martinelli proposed openstack/requirements: Add oslo.policy to global requirements  https://review.openstack.org/14862918:45
*** dtantsur has joined #openstack-infra18:45
*** dboik_ has joined #openstack-infra18:45
*** krtaylor has quit IRC18:45
*** dboik has quit IRC18:46
*** che-arne has quit IRC18:47
*** SumitNaiksatam has joined #openstack-infra18:48
signed8bitHi infra... Is the requirement of 2 +2s in Gerrit something you can achieve with the normal release or is that something you all patched in?18:48
*** crc32 has joined #openstack-infra18:50
*** dboik_ has quit IRC18:50
*** sambetts has quit IRC18:51
*** dboik has joined #openstack-infra18:51
mordredsigned8bit: we do not enforce hat requirement in code18:52
mordredit's a social requirement18:52
*** carl_baldwin_ has joined #openstack-infra18:52
mordredbut - we do not have any local patches to gerrit at this point18:52
*** sambetts has joined #openstack-infra18:52
mordredsigned8bit: so everything we do with gerrit for openstack is doable via config18:52
clarkbok on the logstash worker front, I have mostly confirmed that my patch to wait for name resolution works, except then it dies because the logstash-indexer service is now not starting18:52
clarkbI haven't touched that service so I have no idea what is going on and upstart logs are not helpful18:52
signed8bitah! thanks mordred, so it is really that a core won't merge unless there 2 +2s18:53
clarkbbut its likely another order issue being tickled by changes to the boot process elsewhere18:53
*** jerryz_ has quit IRC18:53
*** carl_baldwin has quit IRC18:53
*** carl_baldwin_ is now known as carl_baldwin18:53
mordredsigned8bit: yes, that's right18:54
signed8bitgreat thanks mordred... I'll stop trying to puzzle that in the config then :-)18:54
mordredclarkb: aren't you so glad that the boot process is so much better in modern things?18:54
clarkbmordred: I kinda want a meta S99-allmythings script that we just call out to init scripts from in the correct order18:57
clarkbthen the distros/init systems can do whatever they want as long as my stuff happens last in an order I control18:57
jeblairclarkb: rc.local?18:57
clarkbjeblair: does ubuntu/upstart support rc.local? I can't keep track anymore18:58
mordredclarkb: we don't run many services on these boxes that arent' the services we choose to run - we could just replace it all and go sysvinit ourselves ...18:58
openstackgerritMerged stackforge/gertty: Release DB session thread lock earlier in syncs  https://review.openstack.org/14258918:58
openstackgerritMerged stackforge/gertty: Speed up the toggling of reviewed/hidden changes  https://review.openstack.org/14259018:58
openstackgerritMerged stackforge/gertty: Add INFO log level potential with --verbose flag  https://review.openstack.org/14314418:58
*** krtaylor has joined #openstack-infra18:58
openstackgerritMerged stackforge/gertty: Add some INFO level log messages  https://review.openstack.org/14316518:58
mordredclarkb: I believe that upstart does support rc.local18:58
openstackgerritMerged stackforge/gertty: Fix syncing messages attached to draft revisions  https://review.openstack.org/16038818:58
*** jedimike has quit IRC18:58
openstackgerritMerged stackforge/gertty: Fix crashing on files with no changes  https://review.openstack.org/16043218:58
clarkbcool maybe we should use that18:59
openstackgerritMerged stackforge/gertty: Wrap long lines in side-by-side diffs  https://review.openstack.org/16092118:59
clarkbalso I think I have logstash working now on boot18:59
openstackgerritMerged stackforge/gertty: Fix keymap substitution  https://review.openstack.org/16175718:59
*** bhunter71 has quit IRC18:59
clarkbnow to the next boot order problem :) very close to having this working sanely18:59
*** otter768 has joined #openstack-infra18:59
jeblairI would like to release current gertty master tomorrow; so if folks could test it and let me know if there are any last-minute issues, that would be great19:00
*** dizquierdo has quit IRC19:00
openstackgerritClark Boylan proposed openstack-infra/puppet-logstash: Start logstash indexer service after networking  https://review.openstack.org/16184119:01
*** dims_ has joined #openstack-infra19:01
*** gyee has joined #openstack-infra19:01
*** bcourt has quit IRC19:01
mordredclarkb: I was being snarky a second ago - but I just looked at the list of things that upstart is managing on gerrit19:01
mordredclarkb: and it's almost all stuff that is for features we don't use - or it's something we installed19:02
*** otter768 has quit IRC19:04
*** achanda has quit IRC19:04
*** dims has quit IRC19:04
*** ivar-lazzaro has quit IRC19:05
*** AJaeger has joined #openstack-infra19:05
*** SumitNaiksatam has quit IRC19:06
clarkbjeblair: noted, will update before doing reviews today19:07
*** ddieterl_ has quit IRC19:07
*** ddieterly has joined #openstack-infra19:08
*** jp_at_hp has quit IRC19:08
*** SumitNaiksatam has joined #openstack-infra19:08
*** asettle is now known as asettle-boxing19:08
jeblairwow, 17 people contributed patches to gertty since the last release19:08
*** sdake_ has quit IRC19:09
*** sdake_ has joined #openstack-infra19:10
anteayayay19:10
*** bswartz has quit IRC19:10
*** wenlock has quit IRC19:10
*** carl_baldwin_ has joined #openstack-infra19:11
*** baoli_ has quit IRC19:13
*** carl_baldwin has quit IRC19:13
*** carl_baldwin_ is now known as carl_baldwin19:13
*** crc32 has quit IRC19:13
*** bhunter71 has joined #openstack-infra19:14
openstackgerritClark Boylan proposed openstack-infra/puppet-log_processor: Lazily connect to logstash  https://review.openstack.org/16184519:15
clarkbok with ^ that stack and the update to the puppet-logstash module I get both logstash-indexer and logstash gearman workers to come up19:17
*** ghostpl_ has joined #openstack-infra19:18
*** dustins_ has joined #openstack-infra19:22
*** gnuoy has quit IRC19:23
*** pelix has quit IRC19:23
*** jamespage has quit IRC19:23
*** vilobhmm1 has quit IRC19:23
*** e0ne has joined #openstack-infra19:24
*** dustins has quit IRC19:24
*** achanda has joined #openstack-infra19:24
jeblairoh, 16.  one of them was named 'Jenkins'19:25
AJaegerjeblair: Jenkins is a great guy, you should count him double!19:26
anteayaha ha ha19:26
*** tsg_away is now known as tsg19:27
*** vilobhmm has joined #openstack-infra19:27
*** ghostpl__ has joined #openstack-infra19:27
jeblairhttps://etherpad.openstack.org/p/5LxyaNPAWa19:28
jeblairdraft release notes for gertty 1.1.0 ^19:28
*** ivar-lazzaro has joined #openstack-infra19:28
*** david-lyle has quit IRC19:29
*** vilobhmm1 has joined #openstack-infra19:30
*** _nadya_ has joined #openstack-infra19:31
*** ghostpl_ has quit IRC19:31
anteayajeblair: lgtm19:31
*** dannywilson has joined #openstack-infra19:32
jeblairanteaya: thx19:33
*** rm_work|away is now known as rm_work19:34
*** teran has joined #openstack-infra19:34
*** bhunter71 has quit IRC19:34
*** vilobhmm has quit IRC19:34
pleia2jeblair: thumbs up19:34
*** annegentle has quit IRC19:35
*** _nadya_ has quit IRC19:35
*** annegentle has joined #openstack-infra19:35
*** mayurig has quit IRC19:36
*** bhunter71 has joined #openstack-infra19:36
*** hodos|2 has joined #openstack-infra19:37
*** ildikov has quit IRC19:38
mtreinishjeblair: heh saving the best name for last :)19:38
* mtreinish ignores the alphabet19:38
clarkbjeblair: line wrapping seems to work :)19:39
clarkbmordred: I am not a fan of 161518, if we are going to use puppet for config management then we should use it for that19:40
clarkbmordred: we are getting into some real nastyness if we start mixing the two tools19:40
*** mrmartin has quit IRC19:41
mordredclarkb: I don't think that's config management19:41
mordredbut that's fair - I'm not tied to it19:41
clarkbmordred: basically if `puppet agent --test` no longer works then its broken19:41
mordredI think it's management of application data19:41
mordredand puppet sucks at it19:41
clarkbI should not need to use ansible to puppet our nodes19:41
*** HeOS has joined #openstack-infra19:42
mordredshrug. ok. I'm not going to fight hard for it19:42
tsgspamaps: mordred mentioned you were helping escalate the liberasurecode sync into ubuntu - many thanks!19:42
jeblairmordred: puppet is really good at managing application data.  perhaps you meant to say orchestrating application changes, in which case i would agree.19:42
clarkbthe reason I say that is because of the openstackci module and because of local testing19:42
mordredjeblair: I have a change up which uses ansible to run manage-projects directly19:42
clarkbthose are both two major use cases that are not served well by a hard dependency on ansible19:42
mordredrather than using ansible to tell puppet what git change it should use to run manage-projects19:42
mordredclarkb: I think local testing is simple19:43
*** hdd_ has joined #openstack-infra19:43
jeblairmordred: i know, that's what we're talking about, right? :)19:43
mordredclarkb: but the re-use of the puppet module not in an orchestration context is a fair poit19:43
mordredpoint19:43
mordredI personally think the simplification of our case outweighs us helping someone to avoid using ansible19:44
*** dims_ has quit IRC19:44
clarkbmordred: my case is avoiding ansible when doing local testing :)19:44
mordredjeblair: :)19:44
mordredclarkb: why?19:44
clarkbmordred: the fewer layers the easier it is to debug and iterate19:44
*** dims has joined #openstack-infra19:45
clarkbI am not going to run ansible to run puppet to config on a node i will just run puppet19:45
mordredbut you can do local testing with fewer layers this way19:45
mordredright19:45
mordredjust do that19:45
mordredyou don't need to run ansible to run puppet19:45
*** _shaps_ has joined #openstack-infra19:45
clarkbmordred: that won't work with 16151819:45
mordredyou need to run ansible to add projects to a gerrit19:45
clarkbI would have to first update a few files manually then run puppet19:45
mordredpuppet will work fine19:45
jeblairmordred, clarkb: for the moment, let's restrict ourselves to the line clarkb has drawn: we can use ansible to orchestrate and execute the puppet run, including providing necessary inputs to it, but other than that, let's do keep the management of the individual nodes to puppet19:45
mordredwhy?19:45
mordredok19:45
mordredlike I said - I'm not going to die on the hill19:45
jeblairi think that's going to help the greatest number of folks understand and work on these systems19:45
pleia2cinerama: feel free to let me know if you would like some help pushing your patchset up over mine (it really doesn't need to be pretty, I'd like like to see what you've come up with :))19:46
pleia2s/like like/just like19:46
anteayaI like like like19:46
cineramapleia2: oh sure. i'm just doing a couple tweaks to get it working reliably19:46
pleia2cinerama: great19:47
* mordred will work on the other approach19:47
pleia2anteaya: clearly I was in the valley yesterday ;D19:47
clarkbmordred: I think the other good thing about the other approach is it working towards puppet apply instaed of puppet agent19:47
rcarrillocruzmordred: you may want to chime in https://review.openstack.org/#/c/160190/19:47
nibalizerthe role of the project-config repo is sortof strange, even in the puppet codebase19:47
mordrednibalizer: ++19:48
anteayapleia2: like clearly19:48
mordredAJaeger: I suggested that rcarrillocruz swap the order of depends on in that change19:48
rcarrillocruzoh19:48
jeblairmordred: i did like seeing that change though.  it is instructive.  :)19:48
mordredAJaeger: largely beause that way the addition of the functional tests to the repo is self-testing19:48
rcarrillocruzAJaeger: you are here19:48
rcarrillocruzyou weren't earlier19:48
rcarrillocruzsorry19:48
rcarrillocruz:-)19:48
AJaegerrcarrillocruz: I'm here now19:49
nibalizerfor instance, you would sortof expect the puppet-zuul module to manage layout.yaml, but it doesn't, openstack_project::zuul pulls in project-config which sideloads a layout.yaml19:49
mordredjeblair: I though the easiest way to have the conversation was just to put up the cahnge and then talk about it :)19:49
nibalizerits heavily decoupled, which is kinda neat but also kindof unexpect19:49
rcarrillocruzso yeah, if read the scrollback, monty prefers gating the func tests code itself19:49
AJaegerrcarrillocruz: I only saw your comment that meant you wanted to change it and then you didn't...19:49
mordredjeblair: I thnik it took 10 minutes to write that - so it being a no-go is not particulary stressful19:50
SpamapStsg: It's basically not going to go anywhere until it passes Debian NEW19:50
AJaegerrcarrillocruz: but this means you will not be able to merge *anything* unless the other patch is in. Is that ok?19:50
SpamapStsg: I don't know why it has sat for 3 months19:50
clarkbmordred: fwiw I think I would be more comfortable if we decoupled project-config completely rather than piecemeal19:50
AJaegerOr do you want to add the job first as experimental, fix everything and thne make it gating?19:50
rcarrillocruzAJaeger: agreed, i was on your same line of thought, but mordred is cool with it19:50
mordredclarkb: k. I think that's kind of where my brain was eventually going19:50
SpamapStimrc: Hi, I commented back on the review btw. I hope you see that I think we just have different ideas on what a unit test is for, and that I don't think anything you've done is "wrong" or "bad". :)19:50
tsgSpamapS: exactly, I couldn't tell either - I was under the impression that it would trickle down once zigo had created deb packages19:51
mordredclarkb: the side-loading of project-config being somethign that we treat in a different manner, since it's it's own beastie anyway19:51
SpamapStsg: https://ftp-master.debian.org/new.html19:51
clarkbmordred: because then we could say "you need this repo in X location" intead of "here are a list of exceptional files that you need to deal with"19:51
mordredclarkb: ++19:51
SpamapStsg: there are things in that queue that have been there for a year. :(19:51
timrcSpamapS, I'm not offended at all.19:51
tsgwow!19:51
SpamapStimrc: cool, I just don't want to be "that guy" ;)19:51
timrcSpamapS, I appreciate your perspective and where you're coming from.19:51
*** hodos|2 has quit IRC19:51
clarkbok lunch time19:52
tsgSpamapS, notmyname: for now, my suggestion is to package liberasurecode with pyeclib and move forward19:52
clarkbjeblair: any word on the port situation?19:52
* clarkb tries one more nova boot before food19:52
timrcSpamapS, I tend to think of unit testing wrapper functions a little differently due to their nature.  I'm probably closer aligned to you on unit testing most any other type of function.19:52
jeblairclarkb: no response on ticket.  if we want, i could ping noc after lunch19:53
SpamapStimrc: I think mocking is one of those areas where easy becomes complex rather quickly, so I think a better solution is to have real fakes. But I also think novaclient's fakes aren't exactly what I'm talking about either. :-D19:53
clarkbjeblair: we are starting to climb in our node use so may start noticing the leak cause failures19:53
openstackgerritSteve Baker proposed openstack-infra/project-config: Create and upload heat test images to tarballs  https://review.openstack.org/15968619:53
AJaegerrcarrillocruz: I changed my -1 to a +1. If you want to try shooting yourself in the foot in your repo, go for it ;)19:53
jesusaurusnibalizer: i just found a reason why the server and template classes shouldn't be merged together: the single_use_slave class uses template but not server19:53
tsgSpamapS, notmyname: I will keep working on the side to get liberasurecode added to the official deb repos.  thoughts?19:53
rcarrillocruzhah19:53
clarkbjesusaurus: oh right because we don't need all that servery stuff19:53
rcarrillocruzi even suggested non-voting19:53
*** Sukhdev has joined #openstack-infra19:53
SpamapStimrc: Still think we're just diametrically opposed on this, because I think a wrapper is a place where the underlying library's behavior is basically the only thing that matters. :)19:53
jesusaurusclarkb: well, "all the server stuff" is exim and different parameters to the template class19:54
timrcSpamapS, I think that's where integration tests hold value.19:54
rcarrillocruzbut mordred apparently likes to live in the limit :P19:54
jesusaurusclarkb: but yeah19:54
rcarrillocruzbbl19:54
rcarrillocruzthx19:54
clarkbERROR (ClientException): Unknown Error (HTTP 500) <- nice19:54
timrcSpamapS, rcarrillocruz's work there is quite good and should ferret out those underlying issues.19:54
* clarkb tries again19:54
SpamapStimrc: that said, the integration tests will catch more, but I want to be able to poke at things in my development iteration loop.. so the more like integration tests unit tests are, the more I can do that. :-P19:54
nibalizerjesusaurus: yea i called that out in the spec19:54
timrcSpamapS, Now you're just being selfish :P JK19:55
nibalizerwe can add a flag or something to server so that you get only the templatey stuff19:55
SpamapStimrc: also frankly the abuse of mocks in Heat has taught me to be wary, since Heat is basically just a massive wrapper on OpenStack. ;)19:55
SpamapSclarkb: error in cloud19:55
timrcSpamapS, I worked with Mr. Mock himself (Kevin McDermott) so I've been sufficiently frightened with out crazy you can let yourself get with them.19:55
*** melwitt has joined #openstack-infra19:55
timrcwith how*19:56
jesusaurusnibalizer: oh? I don't see it in the spec, but I might be blind19:56
*** bdpayne has quit IRC19:56
timrcI think code coverage is pretty valuable thing and having a set of unit tests to validate that you're hitting all code paths is a boon to any project.19:56
*** sarob has joined #openstack-infra19:56
jesusaurusnibalizer: ah, nevermind, I see that parenthetical now19:56
timrcSpamapS, But mostly I'm happy that people are thinking about testing in shade because it's woefully lacking in any form.19:57
openstackgerritJames E. Blair proposed openstack-infra/devstack-gate: Remove trove from default config  https://review.openstack.org/16179719:57
openstackgerritMerged openstack-infra/project-config: Remove rubick and milk-dev from gerritbot  https://review.openstack.org/16083819:57
mordredtimrc: that's largely because I'm a bad person19:57
*** sarob has quit IRC19:57
timrcmordred, Didn't need to empty test file to draw that conclusion :P JK19:58
timrcneed an*19:58
*** sarob has joined #openstack-infra19:58
*** rkukura has quit IRC19:58
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Bump oslo.context version to 0.2.0  https://review.openstack.org/16186819:58
SpamapStimrc: its true. mordred bathes cats just to throw them.19:58
* mordred hands timrc an unfluffy llama19:58
*** dustins_ has quit IRC19:59
SpamapSLuckily this cat didn't mind that much http://i.kinja-img.com/gawker-media/image/upload/s--ZW_N3NIx--/c_fit,fl_progressive,q_80,w_636/zuubca8qu0tzferh0b79.jpg19:59
dimssdague: ^^ (oslo.context rev)19:59
*** james_li has quit IRC19:59
*** MarkAtwood has joined #openstack-infra19:59
timrcSpamapS, Buhaha.20:00
*** bhunter71 has quit IRC20:00
* jeblair attempts to remember how to write a change to create a new project20:00
mordredjeblair: cargo cult, my friend20:00
mordredjeblair: cargo cult20:00
*** sarob has quit IRC20:00
*** MaxV has quit IRC20:00
*** sarob has joined #openstack-infra20:01
jeblairoh whew, saved by the (lunch) bell20:01
*** vilobhmm1 has quit IRC20:01
nibalizercopy the files20:01
nibalizerchange the names20:01
nibalizerget -1'd by anteaya and AJaeger until you get it right20:01
timrcmordred, There's an inside joke there... once you teach cats how to fly, maybe you can help Raul with his poodles - https://www.youtube.com/watch?v=fpJiyAfXbKI20:02
dimsjeblair: we have reasonable doc :) http://docs.openstack.org/infra/manual/creators.html20:02
mordreddims: ::)20:02
*** rmcall has joined #openstack-infra20:03
*** MarkAtwood has quit IRC20:03
AJaegerjeblair: anteaya and myself will nitpick as usual ;)20:04
anteayathat is our job20:04
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for icehouse eventlet bug 1428821  https://review.openstack.org/16187120:05
openstackbug 1428821 in OpenStack Compute (nova) "n-api fails to start on juno grenade" [Undecided,New] https://launchpad.net/bugs/142882120:05
anteayathis rename patch wouldn't be a bad place to being the cargo cult: https://review.openstack.org/#/c/160963/20:05
anteayabegin20:05
*** hodos has joined #openstack-infra20:06
*** patrickeast_ has joined #openstack-infra20:06
*** patrickeast__ has joined #openstack-infra20:06
*** patri237 has joined #openstack-infra20:06
*** patri262 has joined #openstack-infra20:06
*** patri878 has joined #openstack-infra20:06
*** patri889 has joined #openstack-infra20:06
*** _shaps_ has quit IRC20:09
adam_gany project-config cores around to help push through? these are blocking a critical regression fix to icehouse: https://review.openstack.org/159943 and https://review.openstack.org/16156920:09
AJaegeradam_g: will check20:10
adam_gAJaeger, thanks!20:10
openstackgerritKhai Do proposed openstack-infra/system-config: Update the Gerrit javamelody plugin version on review.o.o  https://review.openstack.org/16187320:11
anteayaadam_g: so 15994320:11
*** hdd_ has quit IRC20:11
anteayaare you adding the job or removing it?20:11
*** fifieldt has quit IRC20:11
anteayaI'm losing track20:11
anteayaand why do you have to keep toggling it?20:11
*** fifieldt_ has joined #openstack-infra20:12
*** timcline has quit IRC20:12
anteayaand what does the todo comment mean?20:12
anteayais this disabling the job or a reminder _to_ disable the job20:12
*** timcline has joined #openstack-infra20:13
*** hasharDinner is now known as hashar20:13
anteayaI've tried looking at this patch a few times and am getting lost20:13
AJaegeranteaya: I think it means: These are temporarily disabled to unblock the gate20:14
AJaegerReenable them again once they are working20:14
adam_ganteaya, its disabling the swift functional job for icehouse.20:15
*** markvan_ has joined #openstack-infra20:15
openstackgerritKhai Do proposed openstack-infra/system-config: Update the Gerrit javamelody plugin version on review-dev.o.o  https://review.openstack.org/16187520:16
*** _nadya_ has joined #openstack-infra20:16
*** MarkAtwood has joined #openstack-infra20:17
*** MarkAtwood has quit IRC20:18
anteayawell this patch has been toggled twice and fungi's comments on the original still haven't been addressed: https://review.openstack.org/#/c/157825/3/zuul/layout.yaml20:19
tsgclarkb, SpamapS: just to make sure I have the right ubuntu rev for the Jenkins slaves - it is 14.04.2 ie "Trusty", correct?20:21
*** BobBall is now known as BobBall_AWOL20:21
*** markvan_ has quit IRC20:21
openstackgerritMerged openstack-infra/elastic-recheck: Add query for icehouse eventlet bug 1428821  https://review.openstack.org/16187120:21
openstackbug 1428821 in OpenStack Compute (nova) "n-api fails to start on juno grenade" [Critical,In progress] https://launchpad.net/bugs/1428821 - Assigned to Matt Riedemann (mriedem)20:21
openstackgerritMerged openstack-infra/project-config: Fix the Neutron functional job for stable branches  https://review.openstack.org/16156920:21
AJaegeradam_g, anteaya: Let me edit the patch...20:22
adam_gAJaeger, i was just about to20:22
AJaegeradam_g: go ahead, then I can better +2 ;)20:22
anteayaI commented20:23
*** Longgeek has joined #openstack-infra20:23
openstackgerritSabari proposed openstack/requirements: Bump oslo.vmware to latest release  https://review.openstack.org/16187820:23
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Reintroduce "Temporarily disable swift-dsvm-functional for Icehouse"  https://review.openstack.org/15994320:23
*** coolsvap_ is now known as coolsvap20:24
AJaegerthanks, adam_g. anteaya this looks better, doesn't it?20:25
*** openstackgerrit has quit IRC20:25
*** openstackgerrit has joined #openstack-infra20:26
anteayaokay fine20:27
*** rkukura has joined #openstack-infra20:27
anteayahaving a clear commit message title "add a job" "remove a job" really helps me20:27
*** Longgeek has quit IRC20:27
anteayareintroduce remove a job not so much20:28
tsgclarkb: I need to add jerasure and lib-gf-complete to the Jenkins slaves for Swift erasure coding stuff - and package seems to be available only on "utopic" and "vivid" - does it need to be available on "trusty" to get it on Jenkins slaves?20:28
*** afazekas has quit IRC20:29
*** vilobhmm has joined #openstack-infra20:29
*** vilobhmm1 has joined #openstack-infra20:29
openstackgerritMerged openstack-infra/infra-specs: Add Gerrit Topic to the spec template  https://review.openstack.org/16094620:30
adam_gAJaeger, anteaya thanks20:31
AJaegeradam_g: welcome20:32
clarkbtsg: yes, trusty, centos7, and fedora 21 are the distro releases running tests against master right now20:32
clarkbtsg: we stopped using ubuntu's intermediate releases because they are only supported for 9 months20:33
*** cnesa has joined #openstack-infra20:33
dprinceAJaeger: hi, is the check queue issue resolved now? or still catching up?20:34
*** vilobhmm has quit IRC20:34
tsgclarkb: ok .. no way to get a utopic package on to the slaves unless it is backported?  (packages in question: jerasure with a single dependency gf-complete)20:34
SpamapStsg: correct20:34
SpamapStsg: so even if we get the package synced into vivid, then we have to get it also uploaded into backports.20:34
AJaegerdprince: should be fine now20:35
AJaegerdprince: the usual queue lenght for this time of the day ;)20:35
AJaegerdprince: any problems?20:35
dprinceAJaeger: I'm perplexed on this one https://review.openstack.org/#/c/151960/. Tried resending it this morning...20:36
openstackgerritMerged openstack-infra/infra-specs: Update gerrit topic to downstream-puppet  https://review.openstack.org/16103820:36
*** carl_baldwin_ has joined #openstack-infra20:36
anteayaadam_g: thanks for taking the time to help me wade through it20:36
tsgSpamapS: this is a bit off-topic from the main liberasurecode subject .. these is one of the erasure code backends (Jerasure) that we were trying to get installed on the slaves for unit tests20:36
tsgs/these/this*20:37
anteayaadam_g: I had tried to talk to someone about it before but john didn't know anything about it20:37
tsghttps://bugs.launchpad.net/trusty-backports/+bug/142883420:37
openstackLaunchpad bug 1428834 in trusty-backports "Please backport jerasure and lib-gf-complete from utopic" [Undecided,New]20:37
tsgSpamapS: this package already exists on utopic and vivid20:37
AJaegerdprince: that patch has a "depends-on" - and the other patch is not merged yet20:37
*** achanda has quit IRC20:38
*** ddieterly has quit IRC20:38
AJaegerand depending on another one...20:38
*** carl_baldwin has quit IRC20:38
*** carl_baldwin_ is now known as carl_baldwin20:38
dprinceAJaeger: sigh, such a simple solution20:38
*** ivar-laz_ has joined #openstack-infra20:38
AJaegerdprince: http://docs.openstack.org/infra/manual/developers.html#cross-project-dependencies20:38
*** MarkAtwood has joined #openstack-infra20:38
AJaegerdprince: depends-on is still newish20:39
dprinceAJaeger: gotcha, saw the email a week or so back. Just hasn't clicked for me yet I guess20:39
pleia2lunchtime20:39
clarkbso I have double checked md5sums and the images all match everywhere20:40
clarkbnext step is to see if I can interrogate the image for version info20:40
*** achanda has joined #openstack-infra20:41
*** ivar-lazzaro has quit IRC20:41
clarkbqemu-img info confirms that compat: 0.10 is set20:42
clarkbit does have a virtual size of 11GB so gonna check the size of / on hpcloud nodes20:42
clarkbqemu-img detects no errors in the image20:43
clarkbhpcloud / appears to be 30GB for the images we are using so that should be fine20:44
*** ddieterly has joined #openstack-infra20:44
* clarkb goes back to trying to boot nodes20:44
*** david-lyle_afk has joined #openstack-infra20:45
clarkbboo, I just hit the port limit20:45
*** ZZelle_ has joined #openstack-infra20:46
*** wenlock has joined #openstack-infra20:46
*** david-lyle_afk has quit IRC20:47
*** david-lyle_afk has joined #openstack-infra20:47
*** bdpayne has joined #openstack-infra20:47
openstackgerritJames E. Blair proposed openstack-infra/project-config: Create devstack-plugin-cookiecutter project  https://review.openstack.org/16188720:49
jeblairmtreinish, sdague, dtroyer: see https://review.openstack.org/161886 and https://review.openstack.org/16188720:49
*** MaxV has joined #openstack-infra20:51
*** coolsvap is now known as coolsvap_20:52
openstackgerritMerged openstack-infra/project-config: Reintroduce "Temporarily disable swift-dsvm-functional for Icehouse"  https://review.openstack.org/15994320:52
openstackgerritMerged openstack-infra/project-config: Use GIT_BASE environment variable  https://review.openstack.org/16153720:52
openstackgerritMerged openstack-infra/project-config: Add non-voting tempest dsvm job for Ironic  https://review.openstack.org/16179020:53
*** rkukura has quit IRC20:53
jeblairstarred changes! i forgot to mention those in the release notes!20:53
*** _nadya_ has quit IRC20:53
*** asettle-boxing is now known as asettle20:54
openstackgerritClark Boylan proposed openstack-infra/system-config: Document new puppet lockfile arg  https://review.openstack.org/16188920:54
clarkbmordred: ^ btw that may play into the ansible work20:54
*** rkukura has joined #openstack-infra20:55
dtroyerjeblair: nice idea20:55
clarkbjeblair: did you see that gertty was mentioned on the "OpenStack options for visually impaired contributors" thread?20:56
AJaegeranteaya: we found the same nits ;)20:56
jeblairclarkb: no i haven't seen that...20:56
anteayaAJaeger: not often I beat you to a review20:56
clarkbok I was able to boot that image in hpcloud az3 and it booted all the way up20:56
anteayaclarkb: is that thread in -dev?20:57
clarkbso I am really stumped now, the only thing I can figure is that glance must've had trouble with the image yesterday and getting it to nodes properly20:57
anteayathat subject isn't ringing a bell20:57
clarkbanteaya: ya20:57
mtreinishjeblair: that's a good idea, I'm in favor of it20:58
anteayaclarkb: thanks20:58
*** tkelsey has quit IRC20:58
*** ivar-laz_ has quit IRC20:58
clarkbjeblair: worth mentioning the password file mode check?20:59
*** ivar-lazzaro has joined #openstack-infra20:59
*** salv-orlando is now known as ihate-salv-orlan20:59
*** ivar-lazzaro has quit IRC21:00
mtreinishjeblair: do we have the cookiecutter repo anywhere? Or are we just making a blank repo to fill in later?21:00
*** ivar-lazzaro has joined #openstack-infra21:00
anteayamtreinish: that patch imports no code21:00
anteayaif you want to make something and host is somewhere we can draw it in21:00
mtreinishanteaya: yes, that's what it looked like, that's why I was asking21:00
*** ihate-salv-orlan is now known as salv-orlando21:00
jeblairclarkb: probably so.21:00
*** otter768 has joined #openstack-infra21:00
jeblairmtreinish: this repo intentionally left blank21:01
mtreinishjeblair: ok21:01
*** tiswanso has quit IRC21:02
*** boris-42 has quit IRC21:02
mordredclarkb: ooh21:02
*** david-lyle_afk is now known as david-lyle21:03
*** crc32 has joined #openstack-infra21:05
AJaegeranteaya: you'll get a chance "beating" me for several hours now ;) Bye for now21:05
*** otter768 has quit IRC21:05
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Add expiremental devstack plugin based ironic functional job  https://review.openstack.org/16148321:05
*** AJaeger has quit IRC21:05
*** aysyd has quit IRC21:07
*** rlandy has quit IRC21:08
*** achanda has quit IRC21:09
jeblairmtreinish, anteaya: i prefer to start with empty projects and develop code within our system.21:10
*** bhunter71 has joined #openstack-infra21:10
anteayajeblair: I can agree with that21:10
*** rfolco has quit IRC21:10
*** hdd_ has joined #openstack-infra21:10
clarkbI am going to try booting off of tis image a few more times21:11
clarkbbut if I can't get it to fail I am going to move on to code reviews21:11
asselinHI, just created my first DIB image. Found this script to test it out, but getting an error. Any tips? http://paste.openstack.org/show/189616/ https://github.com/openstack-infra/project-config/blob/master/tools/mount-image.sh#L2421:13
*** dkranz has quit IRC21:13
*** gnuoy has joined #openstack-infra21:13
clarkbasselin: does lsmod show an nbd module?21:13
clarkbasselin: if not modprobe it in21:14
*** mwagner_lap has quit IRC21:14
anteayaasselin: congratulations, despite teh error21:15
*** ldnunes has quit IRC21:15
asselinclarkb, thanks. next error is: http://paste.openstack.org/show/189617/21:16
asselinclarkb, ok, got past that using the full path21:17
jeblairwhy is there a "devstack-plugins" group in projects.yaml?21:18
asselinclarkb,but  http://paste.openstack.org/show/189618/21:18
asselinanteaya, thanks...been meaning to try it out for a long time now.21:18
anteayame too21:18
anteayaoh so many things, me too21:18
*** boris-42 has joined #openstack-infra21:19
clarkbasselin: I think that may be a bug in the script, nbd1 is the block device but not a partition21:19
clarkbasselin: do you have a /dev/nbd1p0 ?21:19
clarkbor p121:19
openstackgerritMerged openstack-infra/zuul: Update zuul-changes to use the enqueue command  https://review.openstack.org/16001521:20
*** jamespage has joined #openstack-infra21:20
*** jamespage has quit IRC21:20
*** jamespage has joined #openstack-infra21:20
*** kgiusti has left #openstack-infra21:20
asselinclarkb, http://paste.openstack.org/show/189619/21:20
asselinclarkb, fyi, this is on precise21:21
clarkbya you have /dev/nbd1p121:21
clarkbtry mounting that21:21
asselinclarkb, thanks, i'm in!21:22
*** carl_baldwin_ has joined #openstack-infra21:23
clarkbasselin: if you want to update the script to hit the first partition that would be a good bug fix21:24
asselinclarkb, ok will try21:24
*** carl_baldwin has quit IRC21:24
*** carl_baldwin_ is now known as carl_baldwin21:24
openstackgerritMerged openstack-infra/zuul: Fix int conversion in swift  https://review.openstack.org/16002321:25
anteayaI'm just looking at the weather, I may tap on Sunday21:25
*** shivharis has joined #openstack-infra21:26
*** mbacchi has quit IRC21:26
shivharisHi folks, I have a question regarding CI voting21:27
*** Jeba has joined #openstack-infra21:27
anteayajust ask21:27
anteayasingle digit lows and hovering around zero for two weeks: http://www.theweathernetwork.com/14-day-weather-trend/canada/ontario/haliburton21:27
shivharisanteaya: i recently got mestery to give Brocade VDX CI voting privileges, but still do not see Verified+1 when I vote21:28
shivharisI may be doing something wrong, wanted help here21:28
shivharise.g.: https://review.openstack.org/#/c/152383/21:29
anteayashivharis: well you are in the neutron-ci group https://review.openstack.org/#/admin/groups/510,members21:29
anteayado you have multiple Brocade VDX CI accounts?21:29
shivharisBrocade VDX CI is unique, as far as i know, there are other brocade account but only one "Brocade VDX CI"21:30
*** achanda has joined #openstack-infra21:30
anteayagreat21:31
*** adalbas has quit IRC21:31
anteayais your ci account testing the ci-sandbox patches? https://review.openstack.org/#/c/155735/21:32
*** achanda has quit IRC21:32
anteayawould be best if it was and then you could test your message configuration there21:32
*** baoli has joined #openstack-infra21:32
anteayasince you don't haev to have permissions to vote on ci-sandbox patches if you can vote on this patches, you have your verification message syntax formatted correctly21:33
shivharisanteaya: ok i will try the sandbox21:33
*** achanda has joined #openstack-infra21:34
*** dustins has joined #openstack-infra21:34
*** baoli has quit IRC21:36
shivharisanteaya: it works fine on the sandbox, but not on neutron21:36
shivharisanteaya: just verifed21:36
*** baoli has joined #openstack-infra21:36
dhellmannanteaya: I'm having some trouble with a gerrit config file in a patch to import a stackforge project. I don't know what the error message means when it says the file "is not normalized" -- can you help? http://logs.openstack.org/47/161447/2/check/project-config-gerrit/b347e05/console.html#_2015-03-05_17_07_31_31521:36
*** lucas-dinner has left #openstack-infra21:37
*** achanda has quit IRC21:37
anteayashivharis: can you show me how it succeeds on the ci-sandbox please? have you a link?21:39
*** achanda has joined #openstack-infra21:39
anteayadhellmann: looking21:39
openstackgerritMichael Johnson proposed openstack-infra/git-review: Choose correct branch for rebase when submitting  https://review.openstack.org/15887721:39
jeblairit's been so long since i've done a filter branch, i forgot to put it on my ramdisk :/21:40
clarkbjeblair: ha21:40
*** MarkAtwood has quit IRC21:40
*** vilobhmm2 has joined #openstack-infra21:40
anteayadhellmann: yes the config file is not in the expected format21:40
shivharisanteaya:  https://review.openstack.org/#/c/155735/  (at the bottom of the page)21:40
anteayadhellmann: so https://review.openstack.org/#/c/161447/2/gerrit/acls/stackforge/aeromancer.config21:41
mkj_wk...now with command-line-specified branch overriding --track/options.track21:41
*** MarkAtwood has joined #openstack-infra21:41
anteayaline 16 needs to be removed21:41
dhellmannanteaya: yes, but what's wrong with it21:41
* mkj_wk hopes that the 20th time is lucky ☺21:41
anteayawe don't need that in the [submit] stanza21:41
*** cnesa has quit IRC21:41
dhellmannanteaya: ok, I was told to add that by a previous reviewer21:41
anteayaand line 12 can be removed21:41
anteayadhellmann: really?21:41
dhellmannanteaya: yes, Khai: https://review.openstack.org/#/c/161447/21:42
anteayausually if you don't want the cla you just remove any reference to it21:42
*** dprince has quit IRC21:42
dhellmannI'm ok requiring the CLA. So far I'm the only committer and I've signed it.21:42
openstackgerritStephanie Miller proposed openstack-infra/puppet-zanata: Initial commit of Zanata-specific sections  https://review.openstack.org/14794721:42
anteayazaro: https://review.openstack.org/#/c/161447/2 I'm in disagreement with your suggestions to doug for his config file21:43
anteayazaro: do you have some time so we can discuss it?21:43
*** vilobhmm1 has quit IRC21:43
anteayadhellmann: ah okay then true on line 1221:43
dhellmannanteaya: ah21:44
anteayashivharis: show me the next patch that is processed by your ci account please? it might take a bit21:44
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add devstack-plugin-trove  https://review.openstack.org/16191121:46
shivharisthe machine is munging on  https://review.openstack.org/#/c/160290, this will be the next one21:46
shivharisanteaya:  ^^^21:46
zaroanteaya: yeah, am i missing something?21:46
openstackgerritDoug Hellmann proposed openstack-infra/project-config: import aeromancer to stackforge  https://review.openstack.org/16144721:46
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Add detail to gerrit ACL normalization errors  https://review.openstack.org/16191421:46
jeblairSlickNik, dtroyer, sdague: also see https://review.openstack.org/#/c/161911/ and https://review.openstack.org/#/c/161910/21:47
dhellmannanteaya: I think the real problem was actually a missing newline at the end of the file, but I fixed the location of the flag, too21:47
stevemarsdague, dhellmann if we can add oslo.policy to g-r, then i can swap keystone to use it instead of the incubated version (or watch it all blow up in my face) https://review.openstack.org/#/c/148629/21:47
anteayadhellmann: and under refs/heads/* I don't know the motivation for the create line for that21:47
shivharisanteaya: the machine is munging on  https://review.openstack.org/#/c/160290 , this will be the next one (removed , from url )21:47
anteayaa missing newline, really?21:47
morganfainbergstevemar, i like watching things explode21:47
anteayashivharis: okay thanks21:48
morganfainbergstevemar, lets do that one :P21:48
stevemarmorganfainberg, then you'll enjoy this one!21:48
*** SumitNaiksatam has quit IRC21:48
morganfainberg;)21:48
jheskethMorning21:48
anteayazaro: yes, I'm looking at https://review.openstack.org/#/c/161447/1..2/gerrit/acls/stackforge/aeromancer.config21:48
anteayamorning jhesketh21:48
anteayazaro: did you mean for doug to add the cla line in the submit stanza?21:48
cineramapleia2: i've landed what i have now after coming up with a much simpler way of resolving the deps issues i was having. i think now we need to pull the mysql definition into the module itself in accordance with our practice elsewhere and a couple other things21:48
*** baoli has quit IRC21:48
*** mrmartin has joined #openstack-infra21:48
anteayadhellmann: we normalized the acl files a while ago since they were all doing different things21:49
*** MaxV has quit IRC21:49
openstackgerritJames E. Blair proposed openstack-infra/project-config: Create devstack-plugin-cookiecutter project  https://review.openstack.org/16188721:49
dhellmannanteaya: zaro said that was for making branches? My original file was copied from the creator's guide, and I don't really know what all of the options are, so just tell me what I need. :-)21:49
anteayadhellmann: I think we decided the create line under refs/heads/* was not needed21:49
anteayadhellmann: sure21:49
anteayazaro: many acls don't haev a create line in refs/heads/*21:50
*** SumitNaiksatam has joined #openstack-infra21:50
anteayazaro: is that required for creating branches?21:50
openstackgerritDoug Hellmann proposed openstack-infra/project-config: import aeromancer to stackforge  https://review.openstack.org/16144721:50
openstackgerritDoug Hellmann proposed openstack-infra/project-config: Add detail to gerrit ACL normalization errors  https://review.openstack.org/16191421:50
zaroyes, i believe so.21:50
dhellmannanteaya: restored to my original version, thanks21:50
dhellmannI don't care about branches in this repo right now.21:50
openstackgerritJames E. Blair proposed openstack-infra/project-config: Create devstack-plugin-trove project  https://review.openstack.org/16191121:50
anteayadhellmann: let's remove that create line then in refs/heads/*21:50
zaroanteaya, dhellmann : i didn't mean add to submit. that was a mistake by me.  I missed that it's already in receive stanza21:51
zaroanyways PS #3 looks good.21:51
dhellmannanteaya: done21:51
anteayayeah this looks familiar to my eyes: https://review.openstack.org/#/c/161447/4/gerrit/acls/stackforge/aeromancer.config21:51
anteayazaro: ah thought so, thanks21:51
dhellmannzaro: np, I didn't notice there was already a similar line either :-)21:51
*** Jeba has quit IRC21:52
*** bcourt has joined #openstack-infra21:52
anteayathanks for asking about it, sorry for the confusion21:53
*** Sukhdev has quit IRC21:53
*** ivar-lazzaro has quit IRC21:53
shivharisanteaya: when you have time, https://review.openstack.org/#/c/160290 finished and Brocade VDX CI posted21:54
anteayaand thanks for the log fix21:54
zaroanteaya: i guess i assume everyone cares to make branches :)21:54
*** ddieterly has quit IRC21:54
dhellmannanteaya, zaro: thanks for the help!21:54
*** ddieterly has joined #openstack-infra21:54
anteayazaro: ha ha ha21:55
openstackgerritJames E. Blair proposed openstack-infra/project-config: Remove devstack-plugins group  https://review.openstack.org/16191721:55
anteayashivharis: okay see the difference in the two patch comments? Patch Set 101: Verified+1 that is in the ci-sandbox comment21:55
anteayaand is not in your neutron patch comment21:55
anteayanow look at what commands are configured to run on patches to each repo21:56
*** baoli has joined #openstack-infra21:56
shivharisanteaya: yup that is exactly the problem i was trying to figure out why21:56
anteayaI'm guessing you have removed the --verify portion of the one to neutron21:56
anteayasince if you use that and you don't have permission to post verification results your comment is refused21:56
*** jamielennox|away is now known as jamielennox21:56
openstackgerritJames E. Blair proposed openstack-infra/project-config: Create devstack-plugin-trove project  https://review.openstack.org/16191121:57
*** ivar-lazzaro has joined #openstack-infra21:57
shivharisanteaya: not really, the command --verify is exactly the same in both cases, i am guessing there are more than one Brocade VDX CI21:57
*** patri237 has quit IRC21:57
*** patri262 has quit IRC21:57
*** patri878 has quit IRC21:57
*** patrickeast__ has quit IRC21:57
*** patri889 has quit IRC21:57
*** patrickeast_ has quit IRC21:57
anteayaokay21:57
shivharisthe id i needed the permision for is "bci"21:58
shivhariswhich i think may be also "Brocade VDX CI", possible?21:58
anteayayes since the email for the system you are testing with is https://review.openstack.org/#/q/owner:openstack_gerrit%2540brocade.com+status:open,n,z21:58
anteayaand the email taht has permissions to vote on neutron permissions is your gmail address21:59
clarkbjeblair: is my comment on 148926 sane?21:59
anteayahttps://review.openstack.org/#/q/owner:shivharis%2540gmail.com+status:open,n,z21:59
*** patrickeast_ has joined #openstack-infra21:59
*** patrickeast__ has joined #openstack-infra21:59
*** patri439 has joined #openstack-infra21:59
*** patri840 has joined #openstack-infra21:59
*** patri930 has joined #openstack-infra21:59
*** patri995 has joined #openstack-infra21:59
shivharisanteaya: what should i do - i am at a loss21:59
*** bcourt has quit IRC21:59
anteayaso yes there are two accounts floating around21:59
openstackgerritRamy Asselin proposed openstack-infra/project-config: Mounting an image requires full path & partition #  https://review.openstack.org/16192021:59
asselinclarkb, ^^22:00
shivharisaaahhh..22:00
anteayashivharis: is that ringing any bells for you?22:00
*** patrickeast has quit IRC22:00
shivharisanteaya: the permission is for shivharis@gmail.com, appears (my guess) i really need it for openstack_gerrit@brocade.com22:01
*** dkranz has joined #openstack-infra22:01
anteayaclarkb: are you able to check the gerrit db and confirm two different Brocade VDX CI accounts22:01
anteayaone with a gmail address and one with a brocade.com email address?22:01
*** ivar-lazzaro has quit IRC22:01
anteayashivharis: well we really need one account, not two22:02
*** patrickeast has joined #openstack-infra22:02
shivharisanteaya: i agree22:02
anteayaso we need to ensure one is disabled lest we continue to have this problem22:02
clarkbanteaya: confirmed (though didn't need the DB, just typed in the add reviewer box and got gerrit to complete for me)22:02
anteayalet's see what the gerrit db says before we do anything22:02
anteayajust wondering if there are more than two Brocade VDX CI accounts22:02
anteayaand we will need the gmail one disabled please22:03
shivharisanteaya: and enable the other one, or should i go back to mestery?22:03
shivharis(for approval)22:04
anteayalet's disable the gmail one first22:04
shivharisok22:04
openstackgerritRadoslaw Smigielski proposed openstack-infra/devstack-gate: Fix inconsisten Ubuntu version in README.rst  https://review.openstack.org/16192222:04
anteayabecause until that is done, mestery won't be able to find the right one22:04
mesteryandreaf: disabling22:04
lifelesstchaypo: are you finding gertty backlogging? Mine is up to 12K pending syncs22:04
anteayaafter that is done, then yes he will have to add the correct account to the gerrit group22:04
*** Jeba has joined #openstack-infra22:04
anteayamestery: no no22:04
anteayamestery: you don't have to do anything yet22:04
mesteryanteaya: whoops22:04
mesteryanteaya: re-eanvling22:04
lifelesstchaypo: I'm wondering if its a latency thing, since it seemed to be ok from the office22:04
anteayamestery: clarkb needs to disable from the db side22:05
mesteryanteaya: got it22:05
mesteryanteaya: re-enabled22:05
tchaypoyeah, I do see that.22:05
anteayamestery: thanks, sorry to prematurely ping you22:05
clarkbanteaya: can't we just get shivharis to rename one?22:05
anteayawell we should disable a dead account should we not?22:05
clarkbanteaya: or were both actually inteded for CI and now we need to properly disable the other?22:05
clarkbanteaya: I guess that is my question, is there a dead account here?22:05
tchaypowhen I was travelling I found that it was fine in the office or on high-throughput low-latency low-jitter connections22:05
anteayaah22:05
jeblairclarkb: agreed, and narrowed it down to one of your two options22:05
anteayashivharis: is the gmail account a dead account?22:06
tchaypobut in places where the connection is jittery or slow it just builds and builds and builds until I kill it and restart22:06
shivharisanteaya: yes for CI purposes22:06
lifelesstchaypo: yeah. Thats the symptom. The older version with all my tuning was ok, I think I need to repeat the process and see whats up. But for now thats a backburner proposition.22:07
*** Bsony has quit IRC22:07
*** jaypipes has quit IRC22:08
jeblairtchaypo: i merged a patch to fix the keymap thing; can you check master and let me know if that fixes it for you?22:08
tchaypojeblair: thanks22:09
tchaypoi was looking at that last night but people kept putting beer and pizza in front of me22:09
anteayatchaypo: silly people22:09
tchaypoand then I got distracted trying to fix a bug in another tool..22:09
*** Sukhdev has joined #openstack-infra22:13
*** oomichi has joined #openstack-infra22:16
*** camunoz has joined #openstack-infra22:17
openstackgerritAdam Gandelman proposed openstack-infra/project-config: Add expiremental devstack plugin based ironic functional job  https://review.openstack.org/16148322:17
*** hdd_ has quit IRC22:18
asselinIs there any difference between devsack-trusty and devstack-trusty-dib here? https://github.com/openstack-infra/system-config/blob/master/modules/openstack_project/templates/nodepool/nodepool.yaml.erb#L99322:18
*** stevemar has quit IRC22:20
*** stevemar has joined #openstack-infra22:20
*** dkliban is now known as dkliban_afk22:21
*** ayoung has quit IRC22:24
*** ivar-lazzaro has joined #openstack-infra22:24
*** tnovacik has joined #openstack-infra22:24
cineramageneral question - what's the infra convention around site.pp files in puppet modules22:25
anteayaclarkb: should I still be waiting to hear from you on disabling the Brocade account with the gmail address?22:26
clarkbanteaya: sorry was it confirmed to deactivate the gmail one?22:26
anteayayes22:27
anteaya shivharis | anteaya: yes for CI purposes22:27
anteayasorry thought you saw that22:27
nibalizercinerama: so generally a module will never have a site.pp file22:29
nibalizerit will almost always have an init.pp file22:29
*** ivar-lazzaro has quit IRC22:32
jesusauruswell, there are modules with site subclasses, such as lodgeit::site22:33
pleia2cinerama: just got back in, I'll look at your change soon22:33
pleia2cinerama: we've typically done the mysql configuration in the translations.pp or whatever file in http://git.openstack.org/cgit/openstack-infra/system-config/tree/modules/openstack_project/manifests22:34
*** teran_ has joined #openstack-infra22:35
pleia2which is kind of like our local.pp that we have22:35
cineramawe don't setup the database itself for, say, the gerrit module in there though, it's done in the gerrit module itself22:36
*** achanda has quit IRC22:36
pleia2ah, good :)22:36
*** dizquierdo has joined #openstack-infra22:36
*** teran has quit IRC22:36
pleia2so maybe we want to pull from the example in the gerrit module22:36
cineramaalso true for storyboard. i personally think it's best if we define it within our module rather than relying on it to be elsewhere especially since we're going to be relying on stuff like the db username for the xml conf22:37
SpamapSjeblair: I'm thinking we should make a full spec for https://storyboard.openstack.org/#!/story/2000175 .. yes?22:38
*** gordc has quit IRC22:38
marunlifeless: for fixture that persists across tests, is testresources a good option?22:38
*** tonytan4ever has quit IRC22:40
*** koolhead17 has quit IRC22:40
pleia2cinerama: yeah, I agree :)22:40
*** achanda has joined #openstack-infra22:40
*** mrmartin has quit IRC22:40
lifelessmarun: yes22:41
marunlifeless: is nobody actually using it?22:42
tchaypojeblair: yep, that seems to work22:42
marunlifeless: https://github.com/openstack/requirements/blob/master/test-requirements.txt22:42
marunlifeless: :/22:42
*** amitgandhinz has quit IRC22:42
marunlifeless: we're getting ready to start targeting test-managed daemons in neutron functional testing, so long-lived db's and daemons are going to be necessary22:43
marunlifeless: https://review.openstack.org/#/c/12825922:43
*** pcm_ has quit IRC22:43
lifelessmarun: that list is the list for the requirements project itself22:43
lifelessmarun: nothing to do with what openstack as a whole can use22:43
marunlifeless: oops22:43
lifelessmarun: see the comment :)22:43
marunlifeless: ah, global-requirements includes test requirements22:44
*** e0ne has quit IRC22:44
marunlifeless: not sure how I missed that.22:44
marunlifeless: thanks for the clarification :)22:44
*** hashar has quit IRC22:45
jeblairSpamapS: i'm thinking in this case we can save a step and go ahead and write the documentation, treating it like a spec.  then implement the doc.  :)22:45
SpamapSjeblair: +1 !22:46
jeblairSpamapS: would you like to volunteer for that?22:46
clarkbanteaya: shivharis I just need to mark that account as inactive correct?22:46
anteayacorrect22:46
SpamapSjeblair: I think I just did. :)22:46
jeblairSpamapS: i do too :)22:46
anteayaclarkb: that is the best we can do, yes?22:46
clarkbanteaya: yes22:46
anteayaclarkb: since we can't delete22:46
anteayathen yes please22:47
clarkbshivharis: and you don't intend on using that account for personal reviews? that looks like a personal address22:47
* SpamapS assigns and moves forward22:47
jeblairSpamapS: so for clarity, i think we're talking about a patch to system-config that adds a new doc page for infra-cloud (in this collection: http://ci.openstack.org/)22:47
SpamapSjeblair: btw is there a good task or set of tasks where I can start helping with nodepool dib work?22:47
shivharisclarkb: i maintain that account, just in case someday i am not associated with brocade :-(22:47
SpamapSjeblair: I'm having a hard time even figuring out what is todo with that.22:47
shivharishence the CI had the id of "bci"22:47
jeblairSpamapS: and it should include most of the things we talked about at the meetup22:48
clarkbshivharis: thats fine, so I think my question is do you ever intend to potentially use that account with gerrit again?22:48
clarkbshivharis: if so you should change the name instead od deactivating the account.22:48
SpamapSjeblair: yeah I think we just need to distill 2 days of note chaos into a doc. :)22:48
shivharisnot at the moment, i can recreate at a later date, right?22:48
*** mfink_ has quit IRC22:48
clarkbshivharis: not with that email address22:48
*** dboik_ has joined #openstack-infra22:48
jeblairSpamapS: https://review.openstack.org/#/q/status:open+topic:dib-nodepool,n,z  apparently nothing... :/22:49
anteayashivharis clarkb I think that account was created by infra, was it not?22:49
shivharisclarkb: in the worst possible case, do as you feel right, i may endup using some other email address later22:49
jeblairmordred, clarkb: what's next on nodepool-dib?22:49
anteayain which case shivharis doesn't ahve access to change the settings22:49
clarkbjeblair: I don't think anyone has updated fungi's bindep change with the new topic or the shade changes22:49
SpamapSjeblair: storyboard might also be a place that would usually be where to go, but .. trying to be gentle.. it's really hard to find things there.22:49
clarkbanteaya: Iam not sure how it was created22:49
openstackgerritSteve Baker proposed openstack-infra/project-config: Create and upload heat test images to tarballs  https://review.openstack.org/15968622:49
openstackgerritSteve Baker proposed openstack-infra/project-config: Don't enable dib service for heat-dsvm-functional  https://review.openstack.org/16193622:49
clarkbanteaya: but I think I can check that22:50
anteayaclarkb: and my udnerstanding is that the way to reuse an email address for gerrit accounts is to deactivate the account of the old one22:50
*** teran has joined #openstack-infra22:50
jeblairSpamapS: yeah, i'm not ready to recommend that for this sort of thing just yet22:50
jeblairclarkb, SpamapS: the change from mordred and yolanda to use shade in nodepool is one; i just changed the topic for it22:51
anteayaclarkb: I created this wikipage at fungi's behest: https://wiki.openstack.org/wiki/OldtoNewGerritCIAccount22:51
shivharisclarkb: if it would be simpler to just deactivate it, go for it22:51
anteayaclarkb: it tells people who want to reuse email addresses to contact infra to disable the one associated with the old account created by infra22:51
anteayaclarkb: I don't know, I'm going by what fungi told me to do22:52
clarkbanteaya: ok22:52
*** dboik has quit IRC22:52
SpamapSjeblair: ah!22:52
clarkbmostly I am just trying to understand how we got here so that we aren't going to create a third account next week22:52
*** teran_ has quit IRC22:52
SpamapSwell I've been reviewing stuff on shade and nodepool so that should help22:52
anteayaclarkb: agreed22:52
anteayashivharis: how did you end up with two accounts?22:53
*** dboik_ has quit IRC22:53
jeblairSpamapS: nodepool testing could also be really useful.  ianw is working on adding a config file syntax validator.  but we have basically no testing of anything important, so every time we make a change it breaks.22:53
shivharisanteaya, clarkb: i really do not know22:53
ianwjeblair: about to send review on that22:53
clarkbanteaya: and now that I look closer there are already 2 accounts with that email addr so that ship has sailed :)22:53
anteayashivharis: we need to make sure it won't happen again22:53
*** annegentle has quit IRC22:53
anteayaclarkb: really which email addr, the gmail one?22:54
jeblairianw: give it a dib-nodepool topic in gerrit :)22:54
anteayaI was told that wasn't possible22:54
SpamapSjeblair: Oh! so some functional tests that point it at a devstacked nova maybe?22:54
anteayaif both are active22:54
clarkbanteaya: yes, looks like there is a VDX testing account and a BNA account22:54
SpamapSianw: btw, did you see your dib change for tracing _finally_ merged?22:54
jeblairSpamapS: ...actually, i think it might be best to leave that for shade for now... and instead use fakes with nodepool...22:55
anteayashivharis: do you have a BNA account with your gmail address?22:55
*** mriedem is now known as mriedem_away22:55
SpamapSjeblair: cool22:55
jeblairSpamapS: everything nodepool does that is interesting takes hours, so i think fakes are the way to go there22:55
*** hodos has quit IRC22:55
*** yfried has quit IRC22:55
SpamapSjeblair: oh yeah22:55
shivharisno BNA account from me, that is a completely different group in brocade22:55
ianwSpamapS: yes, thanks ... i'll use it when i fix the f21 build in a bit :)22:55
jeblairSpamapS: there's actually quite a bit of fakes in there already, just very few tests that use them22:55
shivharisi could not have created that22:56
SpamapSianw: I found it fairly frustrating that it just sat for so long. :-P22:56
clarkbshivharis: well they are using your email address for the account22:56
SpamapSI do like writing tests. :)22:56
*** peristeri has quit IRC22:56
shivharisclarkb: interesting..22:56
anteayashivharis: they have your email address on their account22:56
* SpamapS cranks up some dubstep and gets-ta-testing22:56
anteayaI confirm it in gerrit ui22:56
* ianw needs to re-instate my self-imposed ban on reading hacker news22:56
*** ghostpl__ has quit IRC22:56
*** jaypipes has joined #openstack-infra22:57
anteayashivharis: so do some investigation with brocade22:57
anteayawe can't have random people creating accounts with your email address22:57
shivharisanteaya, clarkb: i am going to do some calling ...22:57
greghaynesSpamapS: self.useFixture(WubWub())22:57
clarkbshivharis: anteaya its not a huge deal, but like I said I would rather do the account dance once then not need to fix it again later22:57
anteayaget back to us once we can confirm no new accounts will be generated with the same email address22:57
clarkbso if we are sure you don't want the VNX account with the gmail addr I will disable it22:57
anteayaclarkb: agreed22:58
anteayashivharis: please confirm again for clarkb22:58
anteaya^^22:58
shivharisanteaya, clarkb: i need to investigate ...22:58
*** HeOS has quit IRC22:58
clarkbjeblair: 161482 should have its topic updated too :)22:59
shivharisanteaya, clarkb: apologies for the confusion from whereever, and much much thanks for your time...22:59
anteayashivharis: okay get back to us22:59
shivharisanteaya: will do22:59
clarkbjeblair: and 155458 might be considered simply because its adding more tests22:59
*** HeOS has joined #openstack-infra23:00
*** otter768 has joined #openstack-infra23:01
*** dizquierdo has quit IRC23:02
*** mattfarina has quit IRC23:02
jeblairclarkb: thanks23:03
*** prad has quit IRC23:03
jeblairclarkb: got a few more too23:05
*** patri767 has joined #openstack-infra23:05
*** patri394 has joined #openstack-infra23:05
*** patrickeast has quit IRC23:05
*** patri832 has joined #openstack-infra23:05
*** patri460 has joined #openstack-infra23:05
*** patri192 has joined #openstack-infra23:05
*** patri755 has joined #openstack-infra23:05
*** achanda has quit IRC23:05
mordredjeblair: yes - finishing nodepool/shade is is the next thing23:05
*** patrickeast__ has quit IRC23:05
*** patri840 has quit IRC23:05
*** patri995 has quit IRC23:05
*** patri439 has quit IRC23:06
*** patrickeast has joined #openstack-infra23:06
*** otter768 has quit IRC23:06
clarkbjeblair: cool I will swing around and check them out once the current set of zuul reviews are done23:06
*** patrickeast_ has quit IRC23:06
mordredjeblair: I've got a few more image element related things, but Ng will have that sorted before we finish nodepool/shade23:06
*** patri930 has quit IRC23:06
clarkbjeblair: I am going to approve 148121 since that makes zuul cloner act a lot more like g-g-p which I think we want23:06
clarkbjeblair: any reason not to that I am missing?23:06
mordredjeblair, SpamapS: in nodepool/shade, we need to finish the main patch, and also we need to take another pass through shade's floating ip handling23:07
*** hichihara has joined #openstack-infra23:07
mordredSpamapS: we need to make it figure out if it can use neutron and use that or fall back to nova if not, and I think there is logic in current nodepool for failures that we missed porting over to shade23:07
*** achanda has joined #openstack-infra23:08
*** tnovacik has quit IRC23:09
*** _shaps_ has joined #openstack-infra23:10
mordredSpamapS: we should also make a pass through every call that nodepool makes and see if we can catch and re-throw _better_ errors to expose temp vs. perm errors23:10
mordredsince right now we're masking all the goodness23:10
mordredSpamapS: and finally, I believe we should take a pass and make sure that cache coherency and validation has a good story23:10
mordredSpamapS: because for the nodepool case, its a long-running thing that could take good advantage of it - as long as we properly invalidate or inject values every time we create/delete a thign23:11
mordredSpamapS: (and it would allow us to choose to deploy a memcached/redis with nodepool for extra love - but we shoudl assume memory for now)23:11
mordredSpamapS: ALSO - we should make sure that the comment about dogpile.cache memory backend leaking memory is addressed since it'sa  long-lived project23:12
clarkbjeblair: re gertty, it seems that my install on my desktop has to be forcefull refreshed, it does not periodically sync things down over time23:12
clarkbjeblair: but I don't think that is a new bug I seem to haveto hit ^R a lot with it even before updating23:12
mordredSpamapS: and finally- nodepool is multi-threaded, so we want to make sure dogpile doesn't die in a fire23:12
SpamapSmordred: dogpile is good w/ threads IIRC23:12
mordredgreat23:12
SpamapSmordred: like, thats half the reason it's awesome.23:13
openstackgerritMerged openstack-infra/zuul: Reset & clean workspace repo for better hygiene  https://review.openstack.org/14812123:13
clarkbdoes dogpile require a memcached or similar? or can it just cache in its own memory/23:13
mordredSpamapS: the current way we're decorating inside of an inner function may need to change - because we may want to invalidated the decorated function rather than the whole cache23:13
mordredclarkb: can cache to it's own memory23:14
mordredwhich is what we default to23:14
lifelessall the memories23:14
mordredclarkb: deploying an external cache will be _possible_ if we choose23:14
mordredbut not required23:14
clarkbya I don't think we need one which is why I ask23:14
*** dustins has quit IRC23:14
mordredI don't either23:14
clarkbthe amount of data is tiny23:14
mordredyup23:14
SpamapSmordred: with the memory store.. we might want to look at the one that pickles things, as that will at least prevent us from getting into a refcount nightmare where nothing is ever freed.23:14
mordredSpamapS: nod. I think it'sa  good todo list item23:15
*** _shaps_ has quit IRC23:15
*** nelsnelson has quit IRC23:15
lifelessSpamapS: no need for that23:15
SpamapS"There is no size management, and values which are placed into the dictionary will remain until explicitly removed. Note that Dogpile’s expiration of items is based on timestamps and does not remove them from the cache."23:15
lifelessSpamapS: cycles are broken just fine23:15
SpamapSlifeless: oh good. I'm still scarred from my PHP days. ;)23:16
lifelessSpamapS: objects with __del__ can be more sticky, but we shouldn't have any of those in play23:16
shivharisclarkb: hi, there should be no BNA account, not sure where is came from, i specifcally asked the BNA team in brocade23:16
lifelessSpamapS: and more recent python deals with those too23:16
shivharisclarkb: the account should be as in: https://wiki.openstack.org/wiki/ThirdPartySystems23:16
lifelessSpamapS: a cycle is handled when full gc is triggered, which is some N opcodes23:16
clarkbshivharis: should I deactivate both of them then?23:17
SpamapSlifeless: Yeah, so thats good to know, but the comment from the docs I just pasted also means that expiration time won't actually GC inside dogpile.cache, so the point is actually moot. :-/23:17
shivharisok, do it23:17
lifelessSpamapS: yup23:17
lifelessSpamapS: thats why I said all the memories23:17
shivharis(oops, can we live with doing just BNA)23:17
shivharisor you must do both?23:18
openstackgerritMerged openstack/requirements: Add oslo.policy to global requirements  https://review.openstack.org/14862923:18
clarkbshivharis: I don't have to do both23:18
*** ChuckC is now known as ChuckC_23:19
clarkbshivharis: but there are two separate issues at play, first you have two VNX accounts with different email addrs so the thought was to disable one. Second you have two accounts with the same email addr23:19
shivharisclarkb: then deactivate BNA23:19
clarkbshivharis: and what about for the VNX test accounts which should we deactivate?23:19
jeblairclarkb: re gertty, that's really weird.  it should do something every minute.  can you run with verbose or debug and confirm?23:20
shivharisclarkb: I want to retain the gerrit account with gerrit id 'bci' what is the email address on that?23:20
*** dboik_ has joined #openstack-infra23:21
openstackgerritMerged openstack-infra/system-config: Add logging to #openstack-api  https://review.openstack.org/16133023:21
shivharisclarkb: as per https://wiki.openstack.org/wiki/ThirdPartySystems/Brocade_VDX_CI23:22
SpamapSmordred: maybe we should make the DBMBackend default.23:23
*** mjturek1 has quit IRC23:23
SpamapSmordred: that would have the benefit of surviving restarts.23:24
SpamapSI'm a little worried about the memory backend now that I'm thinking more about it.23:24
SpamapSStoring all the flavors is meh, but all the things...23:25
shivharisclarkb: please deactive the VDX account not associated with gerrit id "bci"23:25
*** marun has quit IRC23:25
lifelessclarkb: do you see the syn ccount going up and up and up23:25
mordredSpamapS: I'm good with default of DBMBackend23:25
mordredSpamapS: especially if we get cache invalidation on action correct23:26
*** esker has quit IRC23:26
clarkbjeblair: ya I can restart23:26
mordredSpamapS: I mean, for instance - you know how often I should fetch the list of flavors from the cloud?23:26
clarkbshivharis: I will look23:26
*** ChuckC_ is now known as ChuckC23:26
lifelessevery damn request23:26
lifelessbecause mutable data23:26
clarkblifeless: I do not23:27
lifelesspragmatically, once a day or on errors23:27
clarkblifeless: basically I start it and it does the initial sync, goes to 023:27
clarkbthen it seems to only sync when I tell it, either by writing comments or using ^R23:27
lifelessclarkb: if you run with -d23:27
lifelessclarkb: you might see an error in the log ?23:27
mordredlifeless: yup. I like pragmatically on this one :)23:27
*** jaypipes has quit IRC23:28
clarkbshivharis: its done, both of the gmail accounts set inactive (neither had the bci username)23:32
*** sigmavirus24 is now known as sigmavirus24_awa23:32
*** shaunak has joined #openstack-infra23:33
clarkbok I have gertty running with -d23:33
*** cyeoh has joined #openstack-infra23:34
*** Han_ has joined #openstack-infra23:35
mmedvedeWe were having trouble recently with git.o.o, cloning nova in particular. Getting "error: RPC failed; result=18, HTTP code = 200", as seen here http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/148980/8/check-ibm-tempest-dsvm-full/3a7e1d7/logs/devstack-gate-setup-workspace-new.txt.gz . Is it expected with high load on git.o.o?23:36
mmedvedeThe issue is intermittent23:36
clarkbif anyone reviews infra changes after nowish could you point me at them and I can see if they are synced?23:36
shivharisso what should i get mestery to do for the CI account of "Brocade VDX CI"23:36
clarkbjeblair: mordred lifeless ^23:36
clarkbshivharis: I think mestery can add it again and there will only be one23:36
shivharisclarkb: thanks very much23:36
*** wyh_ has joined #openstack-infra23:37
clarkbmmedvede: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1694&rra_id=all is the most recent spike in git.o.o backend load23:38
clarkbmmedvede: are you cloning from scratch in your jobs?23:38
krtaylorclarkb, yes we are23:38
clarkbmmedvede: looks like it23:38
mmedvedeclarkb: no, but we did not update for a while23:39
clarkbwell in that particular log its cloning from scratch23:39
clarkbanyways, my first suggestion would be to try caching the repo23:39
mmedvedeclarkb: ok, that was my feeling as well23:39
mordredclarkb: yeek. do we need to make more git farm?23:39
clarkbwe have seen load spike on the backends but haven't been able to nail down exactly why it happens but if say a bunch of CIs are cloning from scratch and all hit the same backend we lose23:40
clarkbmordred: its possible23:40
mmedvedeclarkb: I am working on getting cache to work. Would that be a good idea to switch to using github in a meantime?23:40
mordredclarkb: we could block git ports and make everyone mount AFS and clone using file:/// urls ...23:41
clarkbmmedvede: I doubt it, githubs error rate is significant23:41
mordredclarkb: I'm obviously kidding23:41
*** SumitNaiksatam has quit IRC23:41
clarkbmordred: we should just shrink nova23:41
*** Han_ has quit IRC23:41
mordredclarkb: that does seem like an awfully high load spike23:41
mordredclarkb: well, there is also that23:41
*** _shaps_ has joined #openstack-infra23:42
clarkbmordred: well what happens is you get a pileup behind the initial load spike23:42
clarkbbecause we use IP hashing from synchronization23:42
mordredclarkb: kernel.org has a public rsync thing23:42
clarkbrather than round robin or bandwidth or etc23:42
mordredclarkb: for this reason23:42
mordredclarkb: so that people can seed locally by rsyncing - and that doesnt' have to go through git mirror23:42
clarkbthat would likely help23:43
mordredwe could even use it in our image building, tbh23:43
clarkbbut if people are going to clone from scratch anyways I doubt rsync will fix that23:43
*** SumitNaiksatam has joined #openstack-infra23:43
mordredclarkb: well, if we had it, we could tell CI folks when they have a problem to switch to rsync23:43
clarkbwe also aren't sure if it is partially related to bad neighbors23:44
mordred++23:44
clarkbwe could run more data collection around that23:44
*** ddieterly has quit IRC23:44
mmedvedeI'll throw together a quick caching git proxy. Should help on our image updates as well. I did run a couple of updates earlier today, wonder if that is the reason for the spike23:46
jeblairclarkb: i just reviewed 157512.  you may need to wait 60 seconds23:47
clarkbjeblair: thanks23:47
clarkbmmedvede: maybe? we do a fair bit of image update ourselves around 1400-1500UTC which isn't when that spike happened23:47
*** SumitNaiksatam has quit IRC23:47
clarkbmmedvede: which is part of why I have been confused about this since we have our own massive thundering herd that doesn't seem to factor into this23:48
clarkbthough our thundering herd does not sit behind one IP address23:48
clarkbmmedvede: is it possible all your traffic is from a single IP?23:48
*** bcourt has joined #openstack-infra23:49
mmedvedeclarkb: I would not think so23:50
clarkbjeblair: ok it did update that change with your +223:50
*** SumitNaiksatam has joined #openstack-infra23:50
clarkbjeblair: maybe I am just super impatient and making this problem up23:50
mordredMAKE IT ALL GO FASTER NOW KTHXBAI23:50
clarkbjeblair: actually maybe its a special case of a problem. The specific incident I remember is after ansible-puppet was created it did not show up in my global project listing23:50
clarkbmordred: on 157512 how paranoid about novaclient should we be? did they just move the code or is this all new code?23:51
*** openstack has joined #openstack-infra23:54
clarkbmordred: pip should if you depend on a newer version iirc23:54
mordrednope23:54
mordrednot even close23:54
mordredit will not upgrade your non-matching versions at all unless you -U23:54
mordred(we just hit this with zuul)23:54
mordredclarkb: fwiw, nodepool has novaclient 2.20.0 and current is 2.22.023:55
mmedvedeclarkb: have checked, our traffic is not from a single ip, just fyi23:55
clarkbok so a non issue for this particular change23:56
clarkbmordred: I don't know why I thought pip worked in this case but you are absolutely right it doesn't23:56
clarkbmordred: jeblair I am going to approve the novaclient in nodepool change then23:56
mordredclarkb: one second23:56
clarkboh wait23:57
clarkb I misread the versions23:57
*** wyh_ has quit IRC23:57
clarkbwe have 2.20.0 not 2.22.023:57
mordredclarkb: yeah. we need to make sure the v2 codepath works23:57
*** baoli has quit IRC23:57
*** ayoung has joined #openstack-infra23:57
*** ivar-lazzaro has joined #openstack-infra23:57
*** ghostpl_ has joined #openstack-infra23:57
mtreinishmmedvede: from inside ibm, are you sure? The internal ip doesn't map 1:1 outside the firwall.23:57
*** shaunak has quit IRC23:58
mordredclarkb: yup. we'll have to upgrade23:58
mtreinish(despite it being on IBM's class A so it easily could be if they were so inclined)23:58
clarkbmtreinish: for subunit workers did you change to reduce to one worker merge?23:58
mordred>>> import novaclient.v223:58
mordredTraceback (most recent call last):23:58
clarkbmtreinish: we have 4 running after the node reboot23:58
mordred  File "<stdin>", line 1, in <module>23:58
mtreinishclarkb: I believe it did, let me double check23:58
mordredImportError: No module named v223:58
clarkbmtreinish: its possible puppet doesn't sufficiently clean things up, can you link to the change if you find it23:58
clarkbmordred: fun23:59
mmedvedemtreinish: I did check with: curl -s checkip.dyndns.org | sed -e 's/.*Current IP Address: //' -e 's/<.*$//  , and it did return a VM's ip23:59
mtreinishclarkb: yep it did: https://review.openstack.org/#/c/151725/23:59
« Wednesday, 2015-03-04 Index Friday, 2015-03-06 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!