Tuesday, 2016-01-19

fungimordred: oh, i should have caught that. yeah, you have the escaping all wrong for traditional grep regex00:00
*** watanabe_isao has joined #openstack-infra00:02
mordredfungi: too much perl in the brain00:02
*** zz_dimtruck is now known as dimtruck00:02
clarkbI think http://logs.openstack.org/65/267865/2/check/gate-dsvm-nodepool/f9f0316/logs/screen-nodepool-builder.txt.gz#_2016-01-15_19_28_41_020 is actually why these noepool jobs are all timing out00:02
clarkbso I am going to look into that now )00:02
clarkbit is like pulling on a thread until you have no more $article_of_clothing00:03
*** sabeen has joined #openstack-infra00:03
clarkbI thought http://logs.openstack.org/65/267865/2/check/gate-dsvm-nodepool/f9f0316/logs/screen-nodepool-builder.txt.gz#_2016-01-15_19_28_39_254 was supposed to be fixed in dib 1.7.1 ?00:04
clarkbgreghaynes: ^00:04
greghaynesclarkb: yes00:04
clarkboh that ran with 1.7.000:04
clarkbso let me recheck and see if we can get more infos00:04
*** sabeen2 has joined #openstack-infra00:06
*** annegentle has quit IRC00:06
*** lucas-dinner has quit IRC00:06
*** salv-orlando has quit IRC00:07
*** Jeffrey4l has joined #openstack-infra00:07
*** sabeen has quit IRC00:08
*** gildub has quit IRC00:11
openstackgerritIan Wienand proposed openstack-infra/project-config: Cleanup the early devstack clone script  https://review.openstack.org/26888800:11
openstackgerritIan Wienand proposed openstack-infra/project-config: Update remote refs in early devstack clone  https://review.openstack.org/26937400:11
*** lucasagomes has joined #openstack-infra00:11
clarkbif pabelanger's change comes back with logging working and images building I will likely approve that then approve the changes to fix statsd, then approve the change to fix failed builds00:12
ianwclarkb: ^ i think this is finally the solution and explains why we're seeing what we see (remote branches that never die)00:12
clarkbianw: will you be pushing a similar change to dib?00:12
ianwclarkb: yeah, i'm going to think about that now.  it really seems like we should actually create a proper mirror (git clone --mirror -- a bare repo & fetch with "+refs/*:refs/*")00:13
clarkb++ though fetching the changes for that may be overkill00:13
ianwbut, as it stands, the dib mirror has never got the remote refs because it's doing that +master:master fetch00:13
*** salv-orlando has joined #openstack-infra00:13
ianwin any case, we can much more clearly document wtf it is doing00:14
*** pradk has quit IRC00:14
*** amotoki has quit IRC00:14
*** eil397 has joined #openstack-infra00:15
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: module is a variable, not a name prefix  https://review.openstack.org/26937500:17
*** Qiming has quit IRC00:17
*** otter768 has quit IRC00:18
*** salv-orlando has quit IRC00:18
openstackgerritMonty Taylor proposed openstack-infra/system-config: Start copying hieradata and puppet modules  https://review.openstack.org/26937600:20
*** kzaitsev1mb is now known as kzaitsev_mb00:20
mordrednibalizer, clarkb: 269375 please - the other one can happen tomorrow00:21
*** yamamoto has joined #openstack-infra00:22
mordrednibalizer: btw - with that patch applied - it works00:22
mordredthings copy as we expect them to00:22
mordredand they do it pretty cleanly00:22
mordredyah man00:22
clarkbwell we have to do that parent00:22
mordredyah. the parent too00:22
* clarkb reviews00:23
mordredthe parent isn't strictly necessary, but it's pretty simple00:23
clarkbbtw hbo has a new recut of the godfather movies00:23
mordreda new old recut00:23
clarkband if you find yourself with 8 hours of free time00:23
clarkbits new old?00:24
clarkbI only just watched them  all for the first time in the last year00:24
mordredyeah - it was first released in 197900:24
mordredand was aired on NBC one time - except with some things cut out because profanity etc.00:25
mordredit was also released on VHS briefly I think in 198100:25
mordredbut it's basically unseen :)00:25
*** hichihara has joined #openstack-infra00:25
nibalizeris it meaningfully different?00:25
mordred(I have a friend whose mom still has a betamax player so that she can watch the tape of it she has from taping it off of tv)00:25
mordrednibalizer: it's cut chronologically00:26
*** amotoki has joined #openstack-infra00:26
TheloHi all, I quite often see this error message in jenkins jobs logs , is anyone familiar with it? http://paste.openstack.org/show/484222/00:26
mordrednibalizer: parts 1 and 2 - together - but scenes in different order00:26
*** samuelBartel has quit IRC00:26
clarkbit starts much slower00:26
clarkbbut I think it makes more sense00:26
clarkbThelo: are you reusing nodes?00:26
*** mtanino_ has joined #openstack-infra00:27
openstackgerritMerged openstack-infra/ansible-puppet: Rename hieraenvironment to puppet_environment  https://review.openstack.org/26929900:28
openstackgerritMerged openstack-infra/ansible-puppet: module is a variable, not a name prefix  https://review.openstack.org/26937500:28
clarkbwow that was fast00:28
*** tjones has joined #openstack-infra00:28
nibalizerclarkb: i take that as a sign that puppet testing is slow and ansible-puppet doesn't have the best testing right now00:29
Theloclarkb: I dont think so, nodepool creates a new node each time no?00:29
clarkbThelo: it should but you have to configure it to do that00:29
clarkbThelo: but that log looks like the slave is being deleted under the job00:29
*** dimtruck is now known as zz_dimtruck00:29
*** mtanino has quit IRC00:29
clarkbThelo: so I would make sure you arne't running >1 job on each node and if so put the offline when job starts stuff in your zuul ocnfig00:29
Theloclarkb: is there a configuration for this, so I can double check?00:31
clarkbThelo: yes you should have a zuul function that sets up the nodes for single use00:31
*** annegentle has joined #openstack-infra00:32
Theloclark: where is that function?00:32
clarkbThelo: in your zuul config00:32
*** yamamoto has quit IRC00:33
clarkbwe have it in openstack_functions.py00:33
clarkbthen in layout.yaml we associate that function by default with every job00:33
TheloI see, great00:35
clarkbThelo: what that does is offline the slave when the job starts which prevents a second job from running on it before nodepool can delete it00:36
clarkbif you don't do this it is possible that a second job would start before nodepool deletes it then nodepool deletes it rseulting in errors00:36
openstackgerritMorgan Fainberg proposed openstack/requirements: Add "positional" package to g-r  https://review.openstack.org/26727000:38
*** akshai_ has joined #openstack-infra00:40
*** achanda has joined #openstack-infra00:41
*** achanda has quit IRC00:43
*** akshai has quit IRC00:43
*** achanda has joined #openstack-infra00:43
*** achanda has quit IRC00:44
*** davideagnello has quit IRC00:44
*** achanda has joined #openstack-infra00:45
*** sdake has quit IRC00:46
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Define defaults as defaults, not variables  https://review.openstack.org/26937800:46
mordrednibalizer, clarkb: one more if you don't mind ... ^^ we got the precedence rules wrong (it's not urgent, nothing is _broken_ - just the 'turn it all on' patch won't work right without it00:47
*** achanda has quit IRC00:47
*** Sukhdev has quit IRC00:48
clarkbI really don't like that new gerrit doesnt give a +/- for each file00:50
*** achanda has joined #openstack-infra00:50
clarkbI guess you get the color coded bar graph thing00:50
clarkbbut I like concrete numbes00:50
*** ddieterly has quit IRC00:50
nibalizermordred: i am phone unfortunately00:50
*** zhurong has joined #openstack-infra00:51
mordrednibalizer: it's ok00:51
mordrednibalizer: it's honestly not a complex patch00:51
clarkbI +2'd00:51
*** Qiming has joined #openstack-infra00:51
clarkbfeel free to approve if you are babysitting00:51
mordredyah - I am00:51
openstackgerritMerged openstack-infra/ansible-puppet: Define defaults as defaults, not variables  https://review.openstack.org/26937800:52
*** Liuqing has joined #openstack-infra00:53
*** achanda has quit IRC00:55
*** pahuang has quit IRC00:56
*** camunoz has quit IRC00:56
*** aeng has quit IRC00:56
*** fawadkhaliq has joined #openstack-infra01:00
*** achanda has joined #openstack-infra01:00
*** fawadkhaliq has quit IRC01:00
*** achanda has quit IRC01:01
*** yamahata has quit IRC01:01
*** Jeffrey4l has quit IRC01:01
*** dims has quit IRC01:01
*** angdraug has quit IRC01:03
clarkbhuh something is insisting on installing dib 1.7.001:03
*** eil397 has quit IRC01:03
* clarkb is going to rant a bit if it is constraints01:03
anteayathanks for warning me01:04
*** achanda has joined #openstack-infra01:05
clarkbso we need to wait for https://review.openstack.org/#/c/268285/ to merge01:05
clarkbwhich I approved earlier today01:06
greghaynesclarkb: wah01:07
greghaynesclarkb: why does that cause 1.7.001:07
clarkbbecause devstack installs everything using constraints01:07
clarkband constraints says use 1.7.001:07
greghaynesclarkb: yes, that says !=1.6.0 currently01:07
clarkbno thats requirements01:07
clarkbcontraints says ==1.7.001:07
clarkb(the thread is about 20 things long at this point)01:08
clarkbgreghaynes: requirements says what you can install, constraints says what you must install if used01:08
clarkband since we use constraints that means constraints wins01:08
greghaynesclarkb: ya, I know, I am wat'ing at why do we have ==1.7.0 in requirements01:08
*** eil397 has joined #openstack-infra01:08
clarkbbecause it automagically gets proposed to update and I bet none of the gating jobs use dib01:09
clarkbso the 1.7.0 update passed and merged01:09
clarkbI'm not sure its worth adding such a job, but does expose the gaps in our constraints system01:09
*** bmjen has quit IRC01:10
*** Hunner has quit IRC01:11
*** gildub has joined #openstack-infra01:11
*** thorst has joined #openstack-infra01:11
greghaynesclarkb: definitely not worth adding a dib job until they are using local mirrors, way too unreliable01:12
*** ZZelle_ has quit IRC01:12
*** thorst_ has joined #openstack-infra01:13
*** markvoelker has quit IRC01:14
*** Hunner has joined #openstack-infra01:15
*** Hunner has quit IRC01:15
*** Hunner has joined #openstack-infra01:15
*** bmjen has joined #openstack-infra01:16
*** thorst has quit IRC01:16
*** sdake has joined #openstack-infra01:17
*** achanda has quit IRC01:17
*** kzaitsev_mb has quit IRC01:19
*** achanda has joined #openstack-infra01:21
*** shashank_hegde has quit IRC01:22
*** pradk has joined #openstack-infra01:23
*** dims has joined #openstack-infra01:23
*** baoli has joined #openstack-infra01:24
jeblairnodpool is doing a pretty poor job of understanding the demand from zuul right now01:25
*** achanda has quit IRC01:26
jeblairunfortunately, i don't have any suggestions as to why01:26
*** sflanigan has joined #openstack-infra01:26
*** sflanigan has joined #openstack-infra01:26
*** dingyichen has joined #openstack-infra01:27
clarkbjeblair: looks like demand is met? what am I missing?01:27
greghaynesis this a sudden change?01:27
jeblairunless maybe we did something where we changed what nodes some jobs run on recently?01:27
clarkbI guess the post jobs are behind when we do have capacity01:27
jeblairclarkb: i see 841 jobs waiting in the zuul queue, and we're using 50% of our capacity01:28
clarkbif you look at the nodepool status page check and gate are satisfied01:28
clarkband post is about an hour or so behind01:28
*** baoli has quit IRC01:29
*** Daisy has joined #openstack-infra01:29
clarkber zuul status page01:29
jeblairclarkb: hrm, are the out-of-window jobs counted in the queue?01:30
jeblairclarkb: no... i don't think they should be01:30
clarkbthey shouldn't be01:30
jeblairclarkb: look at http://grafana.openstack.org/dashboard/db/zuul-status so you can actually see the zuul graph01:31
clarkbso this is why I was asking about the drop in the graph if we did anything for that01:31
jeblairclarkb: you think there's either a constant or increasing error in that graph?01:32
clarkbI think it is possible that the image build jobs are piling up but no one seemed to know what caused the drop so I focused on other things01:32
clarkbjeblair: I think it may be the nodepool image build jobs piling up01:32
jeblairclarkb: k, let me see if i can sort the 19230 jobs we have by # pending :)01:32
clarkb(progress was made fwiw, there are a stack of nodepool changes that should be good to go once we can confirm they work after the dib constraints update)01:32
clarkbwill fix logging, statsd/paramkiko imports, and the handling of dib build failures01:33
clarkbso thats 1/8th of the number and we have multiple images01:34
jeblairclarkb, greghaynes: ^ there are 7 gearman functions like that (108 builds queued)01:35
jeblairso that accounts for 756 entries in the queue01:35
greghaynesjeblair: eek01:35
greghaynesso it is spinning on trying to delete01:35
*** thorst_ has quit IRC01:35
clarkb| 173711 | fedora-22            | /opt/nodepool_dib/fedora-22-1448720157            | 1448720157 | ready    | 51:10:40:44 |01:35
clarkbis the imgae01:36
*** rfolco has joined #openstack-infra01:36
greghayneshalf of that is probably that we arent handling failure for image-delete jobs on the nodepool side01:36
greghaynesI wonder if theres a tb for the other half...01:36
*** baoli has joined #openstack-infra01:38
*** bardia has quit IRC01:38
jeblairgreghaynes: i see no 'nodepool.builder' log lines01:39
clarkbjeblair: yes, we think we diagnosed that to the late import01:39
greghaynesright - https://review.openstack.org/#/c/267865/01:39
clarkbso there is a handful of stuff that cna hopefully be better tomorrow allowing us to clean up the straggler issues01:40
jeblairah k01:40
greghaynesoh, actually - that patch I linked only applies to separate builder process.01:40
greghaynesclarkb: ah, fun01:40
clarkbgreghaynes: ya the fix for late imports should fix logging on in process01:40
jeblaircool, so mystery solved; zuul is okay, we can fix nodepool tomorrow... i think i'll dinnertime now :)01:41
* greghaynes writes some tests for these failures01:41
clarkbyup I think we are good enjoy dinner01:41
*** baoli has quit IRC01:43
*** watanabe_isao has quit IRC01:44
dimsclarkb : which version of python 3.4 are we using? (random question :)01:45
*** rfolco has quit IRC01:46
*** zhurong has quit IRC01:46
*** eil397 has quit IRC01:47
*** bharathm has joined #openstack-infra01:47
*** Liuqing has quit IRC01:47
*** zhurong has joined #openstack-infra01:47
*** zhurong has quit IRC01:48
*** sc68cal has quit IRC01:49
*** shashank_hegde has joined #openstack-infra01:49
openstackgerritgreghaynes proposed openstack-infra/nodepool: Handle dib build failures  https://review.openstack.org/26886701:49
greghaynesclarkb: ^01:49
greghaynesalthough itll still fail until the requirements deal merges01:49
*** ddieterly has joined #openstack-infra01:50
*** bharathm has quit IRC01:52
clarkbthanks I will try getting all the things in tomkrrow after dib constraint updates01:53
clarkbthen restart the service to pick up the changes01:53
*** zhurong has joined #openstack-infra01:54
*** sc68cal has joined #openstack-infra01:54
*** weshay_xchat has joined #openstack-infra01:54
greghaynescool, I'll get a delete fix up real quick too01:54
*** Jeffrey4l has joined #openstack-infra01:56
*** madorn has joined #openstack-infra01:56
madornit looks like the merge requirements on openstack-dev/sandbox have changed -- does it really require a sandbox core reviewer to +2 to merge??01:57
*** weshay_xchat has quit IRC01:58
jheskethgreghaynes: think you might have a small mistake in that handler (see review)01:59
greghaynesjhesketh: ah, yep02:00
*** apoorvad has quit IRC02:00
ianwgreghaynes: re your comment, i kind of saw that as a feature, in that if you just specify a port, it will default back to localhost02:01
greghaynesianw: oh, I didn't realize that works02:01
*** yamamoto has joined #openstack-infra02:01
greghaynesianw: thats fine by me, then02:01
anteayamadorn: I believe that everyone is a sandbox core reviewer02:01
ianwgreghaynes: yeah, statsd 3.0 specifies defaults, it's a bit different to 2.0 in that respect02:01
openstackgerritgreghaynes proposed openstack-infra/nodepool: Handle dib build failures  https://review.openstack.org/26886702:02
clarkbanteaya: madorn yes any registered user can +2 code review https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access02:02
madornclarkb anteaya: i dont have plus +202:03
clarkbmadorn: are you logged in?02:03
madornit used to work..maybe im doing something wrong.02:03
clarkbmadorn: can you link to the change?02:03
*** unicell has joined #openstack-infra02:04
*** unicell1 has quit IRC02:04
*** mtanino_ has quit IRC02:04
lbragstadI didn't have +2 when I reviewed it02:04
lbragstadcc: madorn clarkb ^02:04
*** sdake_ has joined #openstack-infra02:05
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add inetutils-ping to test-deps  https://review.openstack.org/26939002:05
lbragstadI remember reviewing a change a while back in sandbox (for an onboarding developer) and having +202:05
anteayaI also do not have +202:05
anteayalbragstad: yes it seems like what gerrit is telling us re: sandbox access is not the whole truth02:05
ianwgreghaynes: not sure if upstream will take it but, https://github.com/jsocol/pystatsd/pull/73 would mean we could drop the whole stats.py wrapper, which would be nicer still02:06
lbragstadanteaya ++02:06
lbragstadanteaya I've parsed the mailing list... was there a thread on that?02:06
anteayalbragstad: not that I'm aware02:06
anteayathe sandox?02:06
anteayaor the whole truth?02:07
lbragstadanteaya yeah, the policy around who can merge what on the sandbox02:07
*** bpokorny_ has joined #openstack-infra02:07
*** shashank_hegde has quit IRC02:07
anteayaI don't recall a thread on the merging policy of sandbox02:07
*** sdake has quit IRC02:07
anteayaI believe that we set to to all registered users can merge and expect gerrit to do as told02:08
anteayaseems that didn't happen here02:08
*** dslev has joined #openstack-infra02:08
anteayaperhaps it got toggled somehow in the upgrade in December02:08
lbragstadanteaya gotcha02:08
madornanteaya, yeah looks like that02:08
*** sc68cal has quit IRC02:09
anteayaI'm assuming clark is looking at gerrit back of house things while we chat02:09
anteayaas I don't have access to those things02:09
* lbragstad thanks clarkb 02:09
clarkbuh sorry I just got home inspectio nreport so semi distracted02:09
clarkbbut let me take a look02:09
anteayaclarkb: no no, house first02:09
anteayais it a keeper?02:09
anteayahouse is more inportant than sandbox02:10
clarkbanteaya: well nothin gtoo major showed up but probably going to get a second oponion on at least one thing02:10
anteayayay nothing major showed up!02:10
clarkbso my best guess is that gerrit is silly and doesn't allow registered users to +202:10
clarkbbecause why would you want that02:10
clarkbbut it clearly shows registered users should be able to _202:10
clarkber +202:10
anteayait does indeed tell us that02:11
*** bpokorny has quit IRC02:11
*** Liuqing has joined #openstack-infra02:11
lbragstadwhen I look to "Reply" on a review, I can't seem to +2 or -2 it02:11
lbragstadin sandbox02:11
anteayalbragstad: I see the same02:11
*** bpokorny_ has quit IRC02:11
lbragstadI get the option to -1/+1 and 002:12
anteayareply is the new comment02:12
*** shashank_hegde has joined #openstack-infra02:12
anteayalbragstad: yup, me too02:12
* lbragstad might not be using the new gerrit UI properly... 02:12
anteayaha ha ha02:12
anteayaha ha ha02:12
*** sc68cal has joined #openstack-infra02:12
* anteaya wipes tears from her eyes02:12
anteayalbragstad: thanks for that02:13
lbragstadit's been a learning curve :)02:13
anteayaI don't know as the terms new gerrit UI and properly fit in the same sentence02:13
anteayaha ha ha, you generous soul you02:13
anteayaclarkb: so any suggestions for a way forward, or have we finished when we agreed gerrit is silly?02:14
clarkbI am going with gerrit is silly, if someone has time they can read through code or file a bug02:15
clarkbmaybe zaro has insight02:16
anteayalbragstad madorn guess we are done here02:16
*** kzaitsev_mb has joined #openstack-infra02:16
anteayanice to have the chat02:16
clarkbif we really need to approve a chnage for testing purposes we can probably do that02:16
anteayamadorn: do we need to do that?02:16
*** yamamoto has quit IRC02:17
*** unicell has quit IRC02:17
madornwe run a class where we walk people through submitting their first bug fix02:18
*** otter768 has joined #openstack-infra02:18
madornvia sandbox02:18
madorncurious if there is another sandbox02:19
madornor some way we can show them what its like to get some code merged02:19
lbragstadcurious if there is another sandbox that let's you approve/merge your own changes?02:19
*** unicell has joined #openstack-infra02:20
anteayaah okay hmmm02:20
madornit doesn't even need to happen in real time....if we could just get someone to go through their reviews in the sandbox sometime after class and +2 them02:20
anteayano you are using sandbox for its intended purpose02:21
lbragstadmadorn maybe a bot job?02:21
clarkbya I think we would want this to work02:21
anteayawhen is the class?02:21
clarkbbut newer gerrit may have broken us02:21
clarkbif it wasn't almost 6:30pm local time I would dig into java more :)02:21
anteayamadorn: when are you running the next class?02:21
madornno proble, thanks for looking into it.  the class is tomorrow here in california...8:30am to 4pm.  They will ahve their reviews posted by the end of the day.02:22
*** david-lyle has joined #openstack-infra02:22
anteayamadorn: okay well thanks for letting us know02:23
anteayamadorn: I'll bring it to the attention of folks tomorrow and we will see what we can do02:23
madornsure thanks for looking ito it. let me know how i can help, or if you'd like me to file a bug02:23
*** otter768 has quit IRC02:23
anteayaif you could have everyone in the class set a common topic that would be great02:23
lbragstadanteaya do you know if we could get one approve on the project?02:24
anteayathen if we don't have the issue fixed by class time then someone can go through and +A all the patches using that topic02:24
anteayalbragstad: that is what we will be going to try to figure out02:24
anteayawe want everyone to have +2 and +A02:24
anteayawe just might not have that in place by tomorrow02:25
lbragstadanteaya ah, makes sense - I was going to say that if we can't figure out how to do that by tomorrow, I'd be happy to manually +2 patches for the class02:25
anteayathanks for letting us know ahead of time02:25
anteayathanks, we might take you up on that02:25
anteayaI don't have gerrit backend permissions and everyone else is end of day or holidaying02:26
lbragstadmadorn anteaya just ping me if you need me to do that02:26
anteayabut I'll make a case for this tomorrow02:26
madornalright, thanks clarkb anteaya and lbragstad for your help!02:26
anteayalbragstad: thanks well do02:26
anteayamadorn: thanks for using sandbox for its intended purpose02:26
lbragstadmadorn not a problem - thanks for letting us know02:26
anteayait warms my heart02:26
clarkbworst case we can make a group that gets +2 there that includes registered users02:26
anteayaclarkb: cool02:26
clarkbI am hoping that is sufficiently confusing to work around any gerrit weirdness02:26
anteayaha ha ha02:27
anteayaI have to get up soon to chair my next meeting02:27
anteayag'night all02:27
clarkbgood night02:27
*** keedya has quit IRC02:27
*** MarkAtwood has joined #openstack-infra02:28
*** tjones has quit IRC02:29
greghaynesclarkb: so if a builder fails to delete, should nodepool still delete the record from the db?02:31
*** dslev has quit IRC02:31
clarkbgreghaynes: probably not02:32
*** shashank_hegde has quit IRC02:33
greghaynesclarkb: hrm, do we automatically delete dib images?02:34
greghaynesclarkb: I am now confused why nodepool would spin on a delete02:34
greghaynesclarkb: my tests shows nodepool trying the delte once then hanging forever...02:34
clarkbgreghaynes: we delete them when we delete the image in glance02:35
clarkbgreghaynes: so its delete glance then delete on disk02:35
greghaynesclarkb: ok, but that is manual?02:35
greghayneslooks like periodicCleanup does deletes02:35
*** thorst has joined #openstack-infra02:36
*** gfreeman has quit IRC02:36
clarkbits periodic02:37
greghaynesok, so I wonder if the periodic cleanup is what is spinning02:37
greghaynessince the row is not being removed02:37
greghaynesif so, we need a way to signal "we tried to delete, it fails, sorry02:38
greghayneshrm, actually, the issue isn't that nodepool is spinning on trying to delete, the builder must not be consuming those jobs...02:40
*** thorst has quit IRC02:44
*** davideagnello has joined #openstack-infra02:45
*** burgerk has joined #openstack-infra02:45
*** kzaitsev_mb has quit IRC02:50
*** davideagnello has quit IRC02:51
*** markvoelker has joined #openstack-infra02:52
openstackgerritMerged openstack-infra/subunit2sql: Omit subunit2sql/tests from coverage  https://review.openstack.org/26883102:53
*** burgerk has quit IRC02:54
*** markvoelker_ has joined #openstack-infra02:56
*** e0ne has quit IRC02:59
*** otter768 has joined #openstack-infra03:00
*** e0ne has joined #openstack-infra03:00
*** markvoelker has quit IRC03:00
*** markvoelker_ has quit IRC03:01
*** e0ne has quit IRC03:01
*** lezbar has quit IRC03:04
*** markvoelker has joined #openstack-infra03:06
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output  https://review.openstack.org/23444703:06
*** yamamoto has joined #openstack-infra03:09
*** shashank_hegde has joined #openstack-infra03:09
*** yuanying has quit IRC03:15
*** rguillebert has quit IRC03:17
*** dims has quit IRC03:20
TheloHi all, recently a new version of os-brick was release but the test result are quite unstable ( sometimes it just does not find the os-brick 0.8.0 library : http://logs.openstack.org/25/267725/2/check/gate-tempest-dsvm-postgres-full/7eea4ff//logs/devstacklog.txt.gz#_2016-01-19_02_39_50_853), it looks like some of the pypi mirrors maybe missing the os-brick release (http://pypi.region-b.geo-1.openstack.org/packages/2.7/o/), does anyone know how to fix th03:20
*** achanda has joined #openstack-infra03:21
*** Sukhdev has joined #openstack-infra03:22
*** baoli has joined #openstack-infra03:24
*** shashank_hegde has quit IRC03:25
clarkbThelo: you should not use our pypi mirrors03:25
clarkbto fix it an infra-root will need to log in to the machine and diagnose the problem03:26
*** links has joined #openstack-infra03:27
clarkbflock: cannot open lock file /var/run/bandersnatch/mirror.lock: No such file or directory03:27
clarkbbut there is no running bandersnatch03:27
clarkbthere is no /var/run/bandersnatch dir03:27
*** ybathia has joined #openstack-infra03:30
*** tjones has joined #openstack-infra03:31
*** ybathia_ has joined #openstack-infra03:32
*** sflanigan has quit IRC03:33
Theloclarkb: is there any infra-root in this timezone?03:34
clarkbyes I just posted the issue03:35
clarkbI don't have time to dig further tonight though03:35
*** ybathia has quit IRC03:36
*** ybathia_ is now known as ybathia03:36
clarkbI think the lock file didn't clean up properly so futher runs are failing03:36
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941103:36
*** woodster_ has quit IRC03:36
Theloclarkb: so you think its not a mirror issue?03:36
clarkbit is a mirror issue03:36
clarkbit isn't updating because a lock can't be obtained03:37
clarkbthat said you should not use our mirrors. we prpvode no warranty for their usefulness outside of our tests03:37
Theloclarkb: this was tested by your test.03:37
clarkboh for whatever reason  I thought it was third party testing03:38
clarkbin that case maybe someone else has time to debug further. I have a kd on my lap and dinner is soon03:39
*** ashleighfarnham has joined #openstack-infra03:39
Theloclarkb: Sure, enjoy your evening. Just quickly where did you post this bug?03:39
clarkb03:27:27     clarkb | flock: cannot open lock file /var/run/bandersnatch/mirror.lock: No such file or directory03:39
clarkb03:27:59     clarkb | there is no /var/run/bandersnatch dir03:40
*** ddieterly has quit IRC03:42
*** thorst has joined #openstack-infra03:42
*** tlian has quit IRC03:45
clarkbmordred: https://github.com/dear-github/dear-github03:46
*** kzaitsev_mb has joined #openstack-infra03:46
jheskethclarkb: I'm here, just in a meeting... will be done in a second03:48
*** davideagnello has joined #openstack-infra03:48
clarkbjhesketh: I looked in puppet-bandersnatch and that should be a dir so maybe puppet is broken03:49
*** thorst has quit IRC03:50
*** mudassirlatif has joined #openstack-infra03:50
*** sflanigan has joined #openstack-infra03:50
*** sflanigan has joined #openstack-infra03:50
*** yamahata has joined #openstack-infra03:52
*** wei-wang has joined #openstack-infra03:52
*** kzaitsev_mb has quit IRC03:52
*** davideagnello has quit IRC03:52
*** zz_dimtruck is now known as dimtruck03:52
*** tjones has quit IRC03:54
jheskethclarkb: hmm okay03:55
*** flwang has quit IRC03:57
*** achanda has quit IRC04:00
jheskethclarkb: that host is in ansibles static disable:children section04:01
jheskethso not really sure why it's been disabled04:01
*** dimtruck is now known as zz_dimtruck04:03
*** fedexo has joined #openstack-infra04:05
*** yuanying has joined #openstack-infra04:07
*** Daisy has quit IRC04:10
*** ybathia has quit IRC04:12
*** cfarquhar has quit IRC04:13
*** cfarquhar has joined #openstack-infra04:14
*** cfarquhar has quit IRC04:14
*** cfarquhar has joined #openstack-infra04:14
*** yamahata has quit IRC04:15
*** asselin has quit IRC04:15
*** Ramanjaneya has joined #openstack-infra04:17
*** sdake_ has quit IRC04:22
*** moravec has joined #openstack-infra04:22
*** Hal has joined #openstack-infra04:23
*** Hal is now known as Guest6818804:24
*** Ramanjaneya has quit IRC04:28
*** bpokorny has joined #openstack-infra04:28
openstackgerritShamail Tahir proposed openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee  https://review.openstack.org/26942704:29
*** [1]Thelo has joined #openstack-infra04:29
*** shashank_hegde has joined #openstack-infra04:30
*** achanda has joined #openstack-infra04:30
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941104:31
*** ashishb has joined #openstack-infra04:31
*** baoli has quit IRC04:31
*** zz_dimtruck is now known as dimtruck04:32
*** Thelo has quit IRC04:32
*** [1]Thelo is now known as Thelo04:32
*** ashleighfarnham has quit IRC04:33
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941104:34
*** ashishb has quit IRC04:37
*** achanda has quit IRC04:38
*** watanabe_isao has joined #openstack-infra04:38
*** achanda has joined #openstack-infra04:39
*** markvoelker has quit IRC04:39
*** sflanigan has quit IRC04:39
*** markvoelker has joined #openstack-infra04:40
*** dimtruck is now known as zz_dimtruck04:41
*** moravec has quit IRC04:41
*** ddieterly has joined #openstack-infra04:43
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Run the ofagent ci job in the periodic-qa queue  https://review.openstack.org/26732304:44
*** annegentle has quit IRC04:45
*** ddieterly has quit IRC04:47
*** thorst has joined #openstack-infra04:49
*** sabeen2 has quit IRC04:49
*** kzaitsev_mb has joined #openstack-infra04:49
*** mudassirlatif has quit IRC04:51
*** akshai_ has quit IRC04:51
*** shashank_hegde has quit IRC04:52
*** sflanigan has joined #openstack-infra04:55
*** sflanigan has joined #openstack-infra04:55
*** thorst has quit IRC04:55
*** asselin has joined #openstack-infra04:56
*** pradk has quit IRC04:58
*** Daisy has joined #openstack-infra05:05
*** fedexo has quit IRC05:07
*** nikil89_ has joined #openstack-infra05:11
*** nikil89_ is now known as naggappan05:11
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941105:12
*** mrmartin has joined #openstack-infra05:14
*** ashishb has joined #openstack-infra05:15
*** kzaitsev_mb has quit IRC05:22
*** shashank_hegde has joined #openstack-infra05:23
*** bharathm has joined #openstack-infra05:24
*** Ramanjaneya has joined #openstack-infra05:25
*** bharathm has quit IRC05:29
*** Jeffrey4l has quit IRC05:31
*** Ramanjaneya has quit IRC05:32
*** baoli has joined #openstack-infra05:32
*** otter768 has quit IRC05:38
*** amotoki has quit IRC05:39
*** Ramanjaneya has joined #openstack-infra05:40
*** mrmartin has quit IRC05:41
*** shashank_hegde has quit IRC05:42
*** shashank_hegde has joined #openstack-infra05:42
*** salv-orlando has joined #openstack-infra05:42
*** abhishekk has joined #openstack-infra05:43
abhishekkhi all, I am facing problem while regstering for ICLA on review.openstack.org, its raising internal server error05:44
*** ddieterly has joined #openstack-infra05:44
abhishekkdoes any one has any idea about this?05:44
*** amotoki has joined #openstack-infra05:44
*** diltram has quit IRC05:44
*** sdake has joined #openstack-infra05:44
*** jaosorior has joined #openstack-infra05:45
watanabe_isaoasselin, ping.:)05:46
*** diltram has joined #openstack-infra05:47
*** ddieterly has quit IRC05:48
*** vivekd has joined #openstack-infra05:49
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add extlinux pkg-map for redhat to bootloader  https://review.openstack.org/26943905:50
*** jtomasek has joined #openstack-infra05:50
*** ashleighfarnham has joined #openstack-infra05:50
*** elo has quit IRC05:51
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463905:51
*** abregman has joined #openstack-infra05:52
*** Jeffrey4l has joined #openstack-infra05:52
*** thorst has joined #openstack-infra05:53
AJaegerstevemar: we need to keep the bandit job unless we backport the changes to liberty.05:53
openstackgerritEli Qiao proposed openstack-infra/project-config: Magnum: enable mesos voting  https://review.openstack.org/26944005:54
AJaegerianw: could you review https://review.openstack.org/#/c/261994 , please?05:54
AJaegerstevemar: I commented also on https://review.openstack.org/#/c/269268 why we need these stepy by step05:57
*** ildikov has quit IRC05:58
openstackgerritMerged openstack/diskimage-builder: Run package-intalls with py3k if we must  https://review.openstack.org/25438605:58
*** jamielennox is now known as jamielennox|away05:59
*** thorst has quit IRC06:00
*** elo has joined #openstack-infra06:02
*** ashleighfarnham has quit IRC06:04
*** HeOS has joined #openstack-infra06:05
*** aukhan has joined #openstack-infra06:06
*** Ramanjaneya has quit IRC06:06
*** bharathm has joined #openstack-infra06:08
*** gildub has quit IRC06:13
*** vivekd has quit IRC06:15
*** salv-orlando has quit IRC06:15
*** Ramanjaneya has joined #openstack-infra06:16
*** kzaitsev_mb has joined #openstack-infra06:17
*** vgridnev has joined #openstack-infra06:22
*** Ramanjaneya has quit IRC06:23
*** kzaitsev_mb has quit IRC06:23
*** salv-orlando has joined #openstack-infra06:24
*** achanda_ has joined #openstack-infra06:25
*** yfried has quit IRC06:25
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/26796206:27
*** elo has quit IRC06:27
*** elo has joined #openstack-infra06:28
*** achanda has quit IRC06:28
*** amotoki_ has joined #openstack-infra06:32
*** amotoki has quit IRC06:34
*** amotoki_ has quit IRC06:35
*** bharathm has quit IRC06:36
*** davideagnello has joined #openstack-infra06:37
*** elo has quit IRC06:37
*** davideagnello has quit IRC06:38
*** elo has joined #openstack-infra06:38
*** vivekd has joined #openstack-infra06:39
stevemarAJaeger: yeah, i saw the comments, i think i lost in all the depends-on06:43
*** achanda_ has quit IRC06:43
AJaegerstevemar: sorry for losing you06:44
*** amotoki has joined #openstack-infra06:44
stevemarAJaeger: don't sweat it06:44
stevemarAJaeger: i shouldn't have rushed it06:44
*** shashank_hegde has quit IRC06:44
*** bharathm_ has joined #openstack-infra06:44
*** ddieterly has joined #openstack-infra06:45
*** bharathm_ has quit IRC06:45
AJaegerstevemar: no worries, thanks for the reviews06:45
*** vgridnev has quit IRC06:45
*** annegentle has joined #openstack-infra06:46
stevemarAJaeger: always a pleasure sir06:46
*** ddieterly has quit IRC06:49
*** nihilifer has quit IRC06:49
*** nihilifer has joined #openstack-infra06:50
*** annegentle has quit IRC06:50
TheloHI, is there any infra-root that could help on this : recently a new version of os-brick was release but the test result are quite unstable ( sometimes it just does not find the os-brick 0.8.0 library : http://logs.openstack.org/25/267725/2/check/gate-tempest-dsvm-postgres-full/7eea4ff//logs/devstacklog.txt.gz#_2016-01-19_02_39_50_853), it looks like some of the pypi mirrors maybe missing the os-brick release (http://pypi.region-b.geo-1.openstack.org/06:51
Thelopackages/2.7/o/), does anyone know how to fix this06:51
*** Jeffrey4l has quit IRC06:52
jheskethThelo: yeah clarkb looked at that earlier and it looks like that mirror isn't syncing with puppet06:52
jheskethfor some reason it's been disabled in the ansible inventory, but I'm not sure why06:52
jheskethas such, I don't really want to take it out in case there was a good reason06:52
Thelojhesketh: Do you know who could help with that?06:54
jheskethThelo: not sure sorry... I'06:56
*** thorst has joined #openstack-infra06:57
*** amotoki has quit IRC07:00
*** shashank_hegde has joined #openstack-infra07:00
* jhesketh is looking to see if there's anything obvious07:00
*** jcoufal has joined #openstack-infra07:03
*** yfried has joined #openstack-infra07:04
*** thorst has quit IRC07:04
*** salv-orlando has quit IRC07:05
*** _nadya_ has joined #openstack-infra07:09
*** markvoelker_ has joined #openstack-infra07:11
*** markvoelker has quit IRC07:15
*** vgridnev has joined #openstack-infra07:16
*** tdurakov_ has joined #openstack-infra07:16
*** amotoki has joined #openstack-infra07:17
openstackgerritZhi Chang proposed openstack-infra/project-config: Add py27 to steth  https://review.openstack.org/26946007:18
*** tdurakov_ has quit IRC07:19
*** mudassirlatif has joined #openstack-infra07:19
*** flepied1 has quit IRC07:20
*** kzaitsev_mb has joined #openstack-infra07:20
*** shashank_hegde has quit IRC07:21
*** elo has quit IRC07:23
*** ccrouch has quit IRC07:26
*** Ramanjaneya has joined #openstack-infra07:27
*** nmagnezi has joined #openstack-infra07:27
*** mpavone has joined #openstack-infra07:28
*** scheuran has joined #openstack-infra07:34
*** hdd has quit IRC07:35
*** changzhi has joined #openstack-infra07:35
*** placko has joined #openstack-infra07:36
*** otter768 has joined #openstack-infra07:39
openstackgerritZhi Chang proposed openstack-infra/project-config: Add py27 and docs to steth  https://review.openstack.org/26946007:40
*** Jeffrey4l has joined #openstack-infra07:42
*** tdurakov_ has joined #openstack-infra07:43
*** hdd has joined #openstack-infra07:43
*** otter768 has quit IRC07:44
*** kzaitsev_mb has quit IRC07:45
*** ddieterly has joined #openstack-infra07:46
*** jlanoux has joined #openstack-infra07:46
*** vgridnev has quit IRC07:49
*** binbincong has quit IRC07:49
*** jpr1 has joined #openstack-infra07:50
*** jpr2 has joined #openstack-infra07:50
*** amotoki has quit IRC07:50
*** aopincar has joined #openstack-infra07:51
*** samuelBartel has joined #openstack-infra07:51
*** ddieterly has quit IRC07:52
*** eliqiao_ is now known as eliqiao07:52
*** wei-wang has quit IRC07:53
*** mudassirlatif has quit IRC07:54
*** jpr1 has quit IRC07:54
*** Jeffrey4l has quit IRC07:56
*** wei-wang has joined #openstack-infra07:56
*** markvoelker_ has quit IRC07:57
*** wei-wang has quit IRC07:58
*** _nadya_ has quit IRC07:59
anteayaabhishekk: try reading this: https://ask.openstack.org/en/question/56720/cannot-store-contact-information-when-updating-info-in-openstack-gerrit/08:00
*** rcernin has joined #openstack-infra08:01
*** binbincong has joined #openstack-infra08:02
openstackgerritZhi Chang proposed openstack-infra/project-config: Add py27 and docs to steth  https://review.openstack.org/26946008:03
*** tdurakov_ is now known as tdurakov08:04
*** thorst has joined #openstack-infra08:05
*** salv-orlando has joined #openstack-infra08:06
openstackgerritMerged openstack-infra/project-config: Set Ceilometer backend to Gnocchi in integration test  https://review.openstack.org/26332508:06
*** rossella_s has quit IRC08:08
*** ildikov has joined #openstack-infra08:09
*** flepied has joined #openstack-infra08:10
*** thorst has quit IRC08:10
*** markus_z has joined #openstack-infra08:10
*** salv-orlando has quit IRC08:11
*** HeOS has quit IRC08:12
*** Jeffrey4l has joined #openstack-infra08:14
*** flepied has quit IRC08:17
*** vgridnev has joined #openstack-infra08:19
*** isaacb has joined #openstack-infra08:22
*** arxcruz has joined #openstack-infra08:23
*** dingyichen has quit IRC08:25
*** mrmartin has joined #openstack-infra08:26
*** chlong has quit IRC08:27
*** matrohon has joined #openstack-infra08:28
*** sdake_ has joined #openstack-infra08:28
*** ifarkas has joined #openstack-infra08:29
*** sdake has quit IRC08:31
*** vivekd has quit IRC08:32
*** acruz has joined #openstack-infra08:35
*** yfried_ has joined #openstack-infra08:35
*** yfried has quit IRC08:35
*** binbincong has quit IRC08:35
*** arxcruz has quit IRC08:36
*** markvoelker has joined #openstack-infra08:38
*** davideagnello has joined #openstack-infra08:38
*** kzaitsev_mb has joined #openstack-infra08:39
*** daemontool has joined #openstack-infra08:41
*** placko has quit IRC08:41
*** vivekd has joined #openstack-infra08:42
*** Sukhdev has quit IRC08:43
*** markvoelker has quit IRC08:43
*** davideagnello has quit IRC08:44
openstackgerritMerged openstack-infra/project-config: Add neutron -plus job template for advanced features  https://review.openstack.org/24769708:45
*** markvoelker has joined #openstack-infra08:46
*** jordanP has joined #openstack-infra08:48
*** ddieterly has joined #openstack-infra08:49
*** ifarkas has quit IRC08:49
*** hdd has quit IRC08:50
*** tdurakov has quit IRC08:50
abhishekkanteya: thank you08:51
*** tdurakov has joined #openstack-infra08:52
*** flepied has joined #openstack-infra08:53
*** ddieterly has quit IRC08:53
*** fhubik has joined #openstack-infra08:54
*** fhubik is now known as fhubik_brb08:54
*** lezbar has joined #openstack-infra08:56
*** ifarkas has joined #openstack-infra08:58
*** ifarkas has quit IRC08:59
*** ifarkas has joined #openstack-infra08:59
*** vivekd has quit IRC08:59
*** binbincong has joined #openstack-infra09:00
*** fhubik_brb is now known as fhubik09:01
*** markvoelker has quit IRC09:02
*** daemontool has quit IRC09:02
*** watanabe_isao has quit IRC09:02
*** e0ne has joined #openstack-infra09:03
AJaegeranteaya: FYI: https://review.openstack.org/#/c/269440 -1 for voting job only in check09:03
*** daemontool has joined #openstack-infra09:03
*** vivekd has joined #openstack-infra09:04
*** fhubik is now known as fhubik_brb09:04
*** sdake_ has quit IRC09:07
*** thorst has joined #openstack-infra09:07
*** MarkAtwood has quit IRC09:08
*** jordanP has quit IRC09:08
openstackgerritMerged openstack-infra/project-config: Update remote refs in early devstack clone  https://review.openstack.org/26937409:09
*** sputnik13 has joined #openstack-infra09:11
*** amotoki has joined #openstack-infra09:11
*** dizquierdo has joined #openstack-infra09:11
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: Skip one test to unlock saharaclient gate  https://review.openstack.org/26756109:11
*** HeOS has joined #openstack-infra09:12
*** thorst has quit IRC09:15
*** kzaitsev_mb has quit IRC09:16
daemontoolclarkb, ping09:17
daemontoolor anyone available, have issues with http://logs.openstack.org/06/269206/3/check/gate-freezer-devstack-dsvm/405ec04/09:17
daemontoolrelated to: http://logs.openstack.org/06/269206/3/check/gate-freezer-devstack-dsvm/405ec04/09:17
daemontoolbut I'm not sure how to solve it09:17
*** markvoelker has joined #openstack-infra09:19
*** jistr has joined #openstack-infra09:20
*** matrohon has quit IRC09:20
*** hichihara has quit IRC09:22
*** hichihara has joined #openstack-infra09:22
*** fhubik_brb is now known as fhubik09:23
*** markvoelker has quit IRC09:24
*** hashar has joined #openstack-infra09:33
*** e0ne has quit IRC09:34
*** matrohon has joined #openstack-infra09:36
*** Daisy has quit IRC09:37
*** electrofelix has joined #openstack-infra09:38
*** daemontool has quit IRC09:39
*** daemontool has joined #openstack-infra09:39
*** markvoelker has joined #openstack-infra09:40
*** otter768 has joined #openstack-infra09:40
*** fhubik is now known as fhubik_brb09:40
*** derekh has joined #openstack-infra09:43
*** boris-42 has quit IRC09:43
*** markvoelker has quit IRC09:45
*** otter768 has quit IRC09:45
*** fhubik_brb is now known as fhubik09:46
*** fhubik is now known as fhubik_brb09:47
*** amotoki has quit IRC09:48
*** e0ne has joined #openstack-infra09:48
*** ddieterly has joined #openstack-infra09:50
*** sdake has joined #openstack-infra09:51
*** dtardivel has joined #openstack-infra09:51
*** abhishekk has left #openstack-infra09:53
*** aukhan has quit IRC09:53
*** aukhan has joined #openstack-infra09:54
*** ddieterly has quit IRC09:54
*** markus_z is now known as markus_z_meeting09:56
*** e0ne has quit IRC09:57
*** sshnaidm has quit IRC09:57
*** dizquierdo has quit IRC09:59
*** Jeffrey4l has quit IRC10:00
*** vgridnev has quit IRC10:06
*** vgridnev has joined #openstack-infra10:07
*** salv-orlando has joined #openstack-infra10:07
*** daemontool has quit IRC10:08
*** timothyb89 has quit IRC10:08
*** daemontool has joined #openstack-infra10:08
*** timothyb89 has joined #openstack-infra10:10
*** sshnaidm has joined #openstack-infra10:10
*** fhubik_brb is now known as fhubik10:10
*** Ramanjaneya has quit IRC10:10
*** daemontool has quit IRC10:11
*** wei-wang has joined #openstack-infra10:11
*** daemontool has joined #openstack-infra10:11
*** Qiming has quit IRC10:12
*** jordanP has joined #openstack-infra10:12
*** thorst has joined #openstack-infra10:12
*** salv-orlando has quit IRC10:13
*** vgridnev has quit IRC10:13
*** _nadya_ has joined #openstack-infra10:14
*** pigmej has joined #openstack-infra10:15
*** sdake has quit IRC10:15
*** markvoelker has joined #openstack-infra10:16
*** daemontool has quit IRC10:17
*** daemontool has joined #openstack-infra10:17
*** rguillebert has joined #openstack-infra10:17
*** dtantsur|afk is now known as dtantsur10:18
*** placko_ has joined #openstack-infra10:18
*** amotoki has joined #openstack-infra10:18
*** Jeffrey4l has joined #openstack-infra10:18
*** daemontool has quit IRC10:19
*** daemontool has joined #openstack-infra10:19
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing  https://review.openstack.org/25281410:20
*** thorst has quit IRC10:20
*** ociuhandu has quit IRC10:21
*** markvoelker has quit IRC10:21
*** yfried has joined #openstack-infra10:22
*** vgridnev has joined #openstack-infra10:26
*** yfried_ has quit IRC10:27
*** aspiers has quit IRC10:27
TheloHi, any infra-root here?10:29
*** ldnunes has joined #openstack-infra10:30
*** AzherKhan has joined #openstack-infra10:30
*** Jeffrey4l has quit IRC10:31
*** aukhan has quit IRC10:31
*** sputnik13 has quit IRC10:32
*** zhurong has quit IRC10:33
*** zeih has joined #openstack-infra10:37
*** hichihara has quit IRC10:38
*** daemontool has quit IRC10:38
*** davideagnello has joined #openstack-infra10:40
*** daemontool has joined #openstack-infra10:42
*** ildikov has quit IRC10:44
*** markvoelker has joined #openstack-infra10:44
*** davideagnello has quit IRC10:46
*** dizquierdo has joined #openstack-infra10:48
*** dims has joined #openstack-infra10:48
*** rfolco has joined #openstack-infra10:48
*** vgridnev has quit IRC10:49
*** markvoelker has quit IRC10:49
*** ddieterly has joined #openstack-infra10:50
*** AzherKhan has quit IRC10:54
*** ihrachys has joined #openstack-infra10:54
*** ddieterly has quit IRC10:55
*** yamamoto has quit IRC10:55
*** tdurakov has quit IRC10:56
openstackgerritDmitry Tantsur proposed openstack/requirements: Pin more broken DIB versions: 1.6.0-1.7.1  https://review.openstack.org/26955310:57
*** erlon has joined #openstack-infra11:00
*** Qiming has joined #openstack-infra11:05
*** Roamer` has quit IRC11:06
*** aysyd has joined #openstack-infra11:06
*** rossella_s has joined #openstack-infra11:07
*** ildikov has joined #openstack-infra11:09
*** wei-wang has quit IRC11:11
*** pigmej has left #openstack-infra11:12
andreafhello, I'm looking for a puppet review in https://review.openstack.org/#/c/269136/ - it's just adding "if ! defined" statements to imports in log_processor puppet module11:13
andreafand a tiny infra doc update as well: https://review.openstack.org/#/c/254725/ (periodic pipeline test results are collected into the DB as well now)11:14
*** markvoelker has joined #openstack-infra11:14
*** changzhi has quit IRC11:15
*** vgridnev has joined #openstack-infra11:15
*** tdurakov has joined #openstack-infra11:16
*** zeih has quit IRC11:16
openstackgerritMerged openstack/requirements: Blacklist DIB 1.7.0  https://review.openstack.org/26828511:17
*** thorst has joined #openstack-infra11:17
*** markvoelker has quit IRC11:20
*** sfinucan has joined #openstack-infra11:20
*** jaosorior has quit IRC11:22
*** jaosorior has joined #openstack-infra11:22
openstackgerritDmitry Tantsur proposed openstack/requirements: Pin more broken DIB versions: 1.6.0-1.7.1  https://review.openstack.org/26955311:23
dtantsurttx, had to rebase ^^11:23
*** thorst has quit IRC11:25
*** fhubik has quit IRC11:27
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins  https://review.openstack.org/26956711:28
*** aopincar has quit IRC11:30
*** aspiers has joined #openstack-infra11:30
*** markvoelker has joined #openstack-infra11:32
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/26796211:32
*** zeih has joined #openstack-infra11:35
*** ericksonsantos has joined #openstack-infra11:36
*** markvoelker has quit IRC11:36
*** ociuhandu has joined #openstack-infra11:37
*** e0ne has joined #openstack-infra11:38
*** zeih has quit IRC11:40
*** binbincong has quit IRC11:40
*** otter768 has joined #openstack-infra11:41
*** yamamoto has joined #openstack-infra11:45
*** otter768 has quit IRC11:46
*** Ramanjaneya has joined #openstack-infra11:47
*** markus_z_meeting is now known as markus_z11:49
openstackgerritTimur Sufiev proposed openstack-infra/project-config: Source exports from horizon repo for horizon dsvm-job  https://review.openstack.org/26957811:49
*** treaki has joined #openstack-infra11:50
*** Ramanjaneya has quit IRC11:50
*** Ramanjaneya has joined #openstack-infra11:51
*** ddieterly has joined #openstack-infra11:51
*** zeih has joined #openstack-infra11:54
*** jlanoux has quit IRC11:55
*** ddieterly has quit IRC11:55
*** tdurakov has quit IRC12:00
*** binbincong has joined #openstack-infra12:01
openstackgerritQuan Zhou proposed openstack-infra/activity-board: Update JS library 15.06-38-gcd52ae4  https://review.openstack.org/26958212:02
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/26843812:02
*** yamamoto has quit IRC12:03
*** weshay_xchat has joined #openstack-infra12:04
*** daemontool has quit IRC12:05
*** Thelo_ has joined #openstack-infra12:05
*** daemontool has joined #openstack-infra12:06
*** Liuqing has quit IRC12:06
*** yamamoto has joined #openstack-infra12:07
*** baoli has quit IRC12:07
*** salv-orlando has joined #openstack-infra12:08
*** Ramanjaneya_ has joined #openstack-infra12:08
*** ashishb has quit IRC12:10
*** yamamoto_ has joined #openstack-infra12:11
*** vivekd_ has joined #openstack-infra12:11
*** markvoelker has joined #openstack-infra12:11
*** vivekd has quit IRC12:11
*** vivekd_ is now known as vivekd12:11
*** vivekd_ has joined #openstack-infra12:12
*** salv-orlando has quit IRC12:13
*** yamamoto has quit IRC12:14
*** hashar has quit IRC12:15
openstackgerritMerged openstack-infra/activity-board: Update JS library 15.06-38-gcd52ae4  https://review.openstack.org/26958212:15
*** vivekd has quit IRC12:16
*** vivekd_ is now known as vivekd12:16
*** markvoelker has quit IRC12:16
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Upload metadata for pseudo folder  https://review.openstack.org/26324012:16
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Send instructions to workers to upload metadata  https://review.openstack.org/26321812:16
*** _nadya_ has quit IRC12:16
openstackgerritAngus Lees proposed openstack/requirements: Add oslo.privsep to global-requirements.txt  https://review.openstack.org/26618912:16
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_libvirt job  https://review.openstack.org/26959212:17
*** julim has joined #openstack-infra12:19
*** yamamoto_ has quit IRC12:23
*** _nadya_ has joined #openstack-infra12:26
*** tdurakov has joined #openstack-infra12:27
ttxdtantsur: re +2ed12:30
*** jtomasek__ has joined #openstack-infra12:30
*** jtomasek__ has quit IRC12:30
*** jtomasek_ has joined #openstack-infra12:30
*** dizquierdo has quit IRC12:33
*** ildikov has quit IRC12:35
*** dizquierdo has joined #openstack-infra12:36
*** davideagnello has joined #openstack-infra12:42
*** links has quit IRC12:46
*** davideagnello has quit IRC12:47
*** thorst has joined #openstack-infra12:48
*** markvoelker has joined #openstack-infra12:49
*** amrith is now known as _amrith_12:50
*** moravec has joined #openstack-infra12:50
*** binbincong has quit IRC12:51
*** ddieterly has joined #openstack-infra12:52
*** Guest68188 has quit IRC12:52
*** mpavone has quit IRC12:52
*** markvoelker has quit IRC12:54
*** binbincong has joined #openstack-infra12:56
*** ddieterly has quit IRC12:56
*** ildikov has joined #openstack-infra12:57
*** yamamoto has joined #openstack-infra12:59
*** Hal has joined #openstack-infra13:01
*** mpavone has joined #openstack-infra13:01
*** mpavone has quit IRC13:01
*** Hal is now known as Guest1717313:02
*** jlanoux has joined #openstack-infra13:02
*** sdake has joined #openstack-infra13:06
*** aopincar has joined #openstack-infra13:07
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper  https://review.openstack.org/26928413:07
*** dprince has joined #openstack-infra13:07
*** ddieterly has joined #openstack-infra13:08
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Properly suppress undefined warning  https://review.openstack.org/26962013:09
*** moravec has quit IRC13:09
*** markvoelker has joined #openstack-infra13:10
*** daemontool_ has joined #openstack-infra13:10
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use the puppet role to copy the puppet code  https://review.openstack.org/25158113:10
mordredmorning fungi - when you awaken, could you bless those two? ^^13:11
*** daemontool has quit IRC13:11
*** gordc has joined #openstack-infra13:11
*** vgridnev has quit IRC13:12
*** wei-wang has joined #openstack-infra13:13
*** vgridnev has joined #openstack-infra13:14
*** markvoelker has quit IRC13:15
daemontool_hi, if anyone is available, we need some help with http://logs.openstack.org/06/269206/3/check/gate-freezer-devstack-dsvm/405ec04/console.html13:16
daemontool_after branch creation the error we get at the gate is "ERROR: branch not allowed by features matrix: 1.2.0"13:17
daemontool_branch create is stable/1.2.013:17
*** arxcruz has joined #openstack-infra13:18
*** ekarlso has quit IRC13:18
*** ekarlso has joined #openstack-infra13:18
*** acruz has quit IRC13:19
*** dizquierdo has quit IRC13:23
*** isaacb has quit IRC13:24
*** rhallisey has quit IRC13:26
*** rhallisey has joined #openstack-infra13:27
*** dslev has joined #openstack-infra13:30
*** edmondsw has joined #openstack-infra13:30
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper  https://review.openstack.org/26928413:31
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix some minor project group issues  https://review.openstack.org/26962713:31
EmilienMAJaeger: it that better now? https://review.openstack.org/#/c/269284/2..3/zuul/layout.yaml13:31
EmilienMAJaeger: I was wondering if noop-jobs would also in check, or only gate in this case13:32
openstackgerritMerged openstack/requirements: bump versions for each keystone library  https://review.openstack.org/26662913:37
*** kgiusti has joined #openstack-infra13:37
*** placko_ has quit IRC13:39
*** otter768 has joined #openstack-infra13:42
*** vivekd_ has joined #openstack-infra13:42
*** tlian has joined #openstack-infra13:43
*** yfried has quit IRC13:44
*** vivekd has quit IRC13:45
*** vivekd_ is now known as vivekd13:45
*** Ramanjaneya has quit IRC13:46
*** Ramanjaneya_ has quit IRC13:46
*** otter768 has quit IRC13:47
mordredclarkb: whence you awake, the nodepool-src jobs in shade have gone red, but for the life of me I cannot find any actual errors13:47
mordredclarkb: http://logs.openstack.org/05/267405/5/check/gate-dsvm-nodepool-src-shade/c3e9585/ for instance13:47
mordredclarkb: it seems to be timing out - but all of the things seem to be successing13:48
*** aopincar has quit IRC13:48
*** jaosorior has quit IRC13:48
*** jaosorior has joined #openstack-infra13:48
clarkbmordred it is due to dib 1.7.0 breaking image builds. New dib constraint that went in overnight should fix13:49
mordredok. cool13:49
clarkbalso I am not actually awake right now13:49
*** daemontool__ has joined #openstack-infra13:49
mordredclarkb: sweet. don't be awake then13:49
krotscheckAny non-mordred cores around to review the wheel-mirror patches? https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open13:49
krotscheckclarkb: I know how you feel.13:50
* mordred feels left out13:50
*** daemontool_ has quit IRC13:50
krotscheckmordred: Feel like a pioneer! You were there first!13:50
mordredShrews: ^^ see above13:50
mordredkrotscheck: \o/13:50
mordredkrotscheck: did I receive dissentary for my troubles?13:50
krotscheckDysentery? No, but you broke your leg near chimney rock.13:50
*** ccrouch has joined #openstack-infra13:50
*** markvoelker has joined #openstack-infra13:52
*** dkranz has joined #openstack-infra13:53
*** Roamer` has joined #openstack-infra13:53
*** daemontool__ is now known as daemontool13:54
*** burgerk has joined #openstack-infra13:55
*** markvoelker has quit IRC13:57
*** nelsnelson has joined #openstack-infra13:58
*** amotoki has quit IRC13:58
*** ccrouch has quit IRC13:58
*** markvoelker has joined #openstack-infra14:00
openstackgerritMerged openstack-infra/system-config: Use the puppet role to copy the puppet code  https://review.openstack.org/25158114:00
AJaegersdague: while you might be fine with the direction of https://review.openstack.org/#/c/267669/1 I still think the +2 is wrong, the change does *nothing*14:00
*** nelsnels_ has quit IRC14:00
*** aopincar has joined #openstack-infra14:01
*** markvoelker_ has joined #openstack-infra14:01
clarkbmordred though my rechecks have aboht a 50% pass rate so there may be another issue14:01
AJaegerEmilienM: just add noop to the gate, see openstack/tripleo-heat-templates for an example14:01
openstackgerritZara proposed openstack-infra/storyboard-webclient: Add Create Story Button in Worklists  https://review.openstack.org/26965614:02
sdagueAJaeger: why ... ?14:02
EmilienMAJaeger: ok, I see thanks14:02
*** ari has joined #openstack-infra14:02
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper  https://review.openstack.org/26928414:02
AJaegersdague: see what I posted in the comment, the change has no effect14:02
AJaegersdague: sorry not time to discuss now further, have to go offline for a bit now.14:03
clarkbmordred we may just need a longer timeout14:05
*** tiswanso has joined #openstack-infra14:05
*** markvoelker has quit IRC14:05
*** daemontool has quit IRC14:06
*** daemontool has joined #openstack-infra14:07
*** hashar has joined #openstack-infra14:07
*** sabeen1 has joined #openstack-infra14:08
*** annegentle has joined #openstack-infra14:09
*** doug-fish has joined #openstack-infra14:10
*** akshai has joined #openstack-infra14:10
*** sabeen3 has joined #openstack-infra14:11
openstackgerritafazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys  https://review.openstack.org/26966014:11
*** sabeen3 has left #openstack-infra14:11
*** sabeen1 has quit IRC14:13
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949214:13
mordredclarkb: excellent14:13
mordredclarkb: btw, I know you're not awake, but ^^ I think I'm going to be able to land that today :)14:13
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix some minor project group issues  https://review.openstack.org/26962714:14
*** ddieterly has quit IRC14:15
*** peristeri has joined #openstack-infra14:15
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Fix width of "no result" and "loading" rows in list tables  https://review.openstack.org/26966314:15
*** vivekd has quit IRC14:16
*** vivekd has joined #openstack-infra14:16
*** lucasagomes is now known as lucas-afk14:16
*** ddieterly has joined #openstack-infra14:17
sdagueAJaeger: I'm looking at before and after match criteria, they are different, the important bit I highlighted in a follow on comment14:17
*** ddieterly has quit IRC14:18
*** amitgandhinz has joined #openstack-infra14:19
*** Daisy has joined #openstack-infra14:20
openstackgerritMerged openstack-infra/shade: allow for updating passwords in keystone v2  https://review.openstack.org/26836114:22
mrmartinmordred: with this local puppet apply puppetboard.o.o will consume reports of puppet runs?14:22
*** jsavak has joined #openstack-infra14:23
*** baoli has joined #openstack-infra14:23
mordredmrmartin: it will indeed14:23
openstackgerritMerged openstack-infra/shade: Support neutron subnets without gateway IPs  https://review.openstack.org/26740514:24
mordredmrmartin: we have the code to grab the report files from the hosts after we run puppet on them and upload to the puppetdb server14:24
mrmartinoh ok.14:24
*** annegentle has quit IRC14:27
*** baoli_ has joined #openstack-infra14:28
*** daemontool_ has joined #openstack-infra14:30
*** rossella_s has quit IRC14:30
*** [1]Thelo has joined #openstack-infra14:30
*** baoli has quit IRC14:31
*** mriedem_away is now known as mriedem14:31
*** daemontool has quit IRC14:31
*** rossella_s has joined #openstack-infra14:31
*** sabeen1 has joined #openstack-infra14:31
*** tiswanso has quit IRC14:31
*** Thelo has quit IRC14:33
*** [1]Thelo is now known as Thelo14:33
*** tiswanso has joined #openstack-infra14:33
*** acruz has joined #openstack-infra14:33
*** daemontool_ has quit IRC14:34
*** ccrouch has joined #openstack-infra14:34
*** daemontool_ has joined #openstack-infra14:34
*** naggappan has quit IRC14:35
*** arxcruz has quit IRC14:35
*** gordc has quit IRC14:36
*** ccrouch has quit IRC14:36
*** ccrouch has joined #openstack-infra14:36
*** yamamoto has quit IRC14:36
openstackgerritJulien Danjou proposed openstack/requirements: Fix voluptuous version  https://review.openstack.org/26967714:36
*** daemontool_ has quit IRC14:37
*** daemontool_ has joined #openstack-infra14:38
*** vivekd_ has joined #openstack-infra14:39
*** baoli_ has quit IRC14:39
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc  https://review.openstack.org/26202814:39
smcginnisclarkb: When you wake up... :)14:39
smcginnisclarkb: Any resolution of the mirrors not getting updated from last night?14:39
smcginnisclarkb: Trying to get a requirements update through, but tests keep failing due to that.14:40
*** fhubik has joined #openstack-infra14:41
*** vivekd has quit IRC14:42
*** vivekd_ is now known as vivekd14:42
*** aopincar has quit IRC14:42
*** davideagnello has joined #openstack-infra14:43
clarkbsmcginnis: I havent done anything mkre than what I posted here14:45
clarkbjhesketh looked and found puppet was disabled on those machines14:45
clarkbI do not know why14:45
*** _amrith_ is now known as amrith14:46
*** ddieterly has joined #openstack-infra14:47
*** davideagnello has quit IRC14:48
*** gordc has joined #openstack-infra14:48
openstackgerritSam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic  https://review.openstack.org/26968414:51
*** notmorgan has quit IRC14:51
*** flepied has quit IRC14:51
*** dizquierdo has joined #openstack-infra14:54
tdurakovAJaeger, sdague hi, talking about https://review.openstack.org/#/c/267669/ could we merge it now, or changes required?14:54
*** eharney has quit IRC14:54
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins  https://review.openstack.org/26900714:54
*** knikolla has joined #openstack-infra14:55
*** regXboi has joined #openstack-infra14:55
knikollaHi guys. I’m currently setting up on Gerrit, but I think I’ve previously done the setup with a different launchpad account which I no longer have (merged it) and reserved my preferred username. So I can no longer select that username now. Would it be possible to free that username up so I can reregister it with a different launchpad account?14:57
*** placko_ has joined #openstack-infra14:58
*** akshai has quit IRC14:59
*** watanabe_isao has joined #openstack-infra14:59
*** bryan_att has joined #openstack-infra15:00
*** akshai has joined #openstack-infra15:00
*** notmorgan has joined #openstack-infra15:00
*** notmorgan has quit IRC15:00
*** notmorgan has joined #openstack-infra15:00
*** notmorgan has quit IRC15:01
*** notmorgan has joined #openstack-infra15:01
*** notmorgan has quit IRC15:01
*** notmorgan has joined #openstack-infra15:01
*** baoli has joined #openstack-infra15:03
*** tdurakov has quit IRC15:03
*** Daisy has quit IRC15:03
*** baoli has quit IRC15:03
*** annegentle has joined #openstack-infra15:04
*** watanabe_isao has quit IRC15:04
*** baoli has joined #openstack-infra15:04
*** annegent_ has joined #openstack-infra15:04
*** moravec has joined #openstack-infra15:05
*** tdurakov has joined #openstack-infra15:06
*** jcoufal_ has joined #openstack-infra15:06
*** baoli has quit IRC15:07
clarkbknikolla: generally no, but it may be possible yo associate new openid to old account. Would that work?15:07
*** baoli has joined #openstack-infra15:08
*** annegentle has quit IRC15:08
knikollaclarkb: Can the new openid assigned be the one I’m trying to set up now with gerrit?15:09
*** peristeri has quit IRC15:09
*** jcoufal has quit IRC15:09
knikollaclarkb: That would work. What other information would be needed?15:10
clarkbwe need to know what the two accounts are and make sure they belong together15:10
*** dslev has quit IRC15:11
fungiThelo: jhesketh: clarkb: it looks like when i rebooted the hpcloud pypi server a few days ago to more thoroughly clean up its full rootfs, /var/run/bandersnatch never got created and so the cronjob couldn't create the lockfile. as a result it's not been updating for ~3 days. i just made that directory by hand and so it should start syncing again (at least until we finally take down hplcoud in a week15:12
fungior so)15:12
*** daemontool_ has quit IRC15:12
clarkbfungi becaude puppet isnt running15:12
knikollaclarkb: The account with the username I want to get is “knikolla”. The account I’m currently on is tied to the launchpad account with email “knikolla@bu.edu” and id gerrit id 2016315:12
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc  https://review.openstack.org/26202815:12
*** eharney has joined #openstack-infra15:13
fungidaemontool_: it sounds like you're trying to use devstack-gate with a nonstandard stable branch name. as far as i know that's not a solved problem unless you want it tested with master of everything else15:13
*** jcoufal has joined #openstack-infra15:13
*** tdurakov has quit IRC15:13
*** vivekd has quit IRC15:13
*** tonytan4ever has joined #openstack-infra15:14
fungiclarkb: yeah, though that also seems sort of chicken-and-eggish to puppet an ephemeral directory in a tmpfs15:14
*** zz_dimtruck is now known as dimtruck15:14
*** tdurakov has joined #openstack-infra15:14
fungibasically leaves things broken until the first puppet run after a reboot. probably not a great pattern for us to be following15:14
*** tdurakov has quit IRC15:16
*** jcoufal_ has quit IRC15:16
*** annegent_ is now known as annegentle15:16
*** tdurakov has joined #openstack-infra15:17
*** yfried has joined #openstack-infra15:17
*** peristeri has joined #openstack-infra15:18
* mordred suggests that bandersnatch should create its own directory in /var/run given the nature of such things15:19
anteayaAJaeger: thank you15:19
*** lucas-afk is now known as lucasagomes15:19
clarkbknikolla: and the old ope id doesnt exist because it was merged into the current lp account?15:20
*** jpr2 has quit IRC15:20
fungimordred: well, the cron job should anyway, since it's the cron job that wants that directory there (bandersnatch doesn't use it, our flock wrapper does)15:21
mordredI agree that the cron job should ensure the directory, not puppet15:22
knikollaclarkb: I believe so, I’m sure I’ve gone through the gerrit setup before about half-year ago. But since that I’ve done some kung fu with my launchpad accounts merging them because I realized I had 3 different ones.15:22
*** MarkAtwood has joined #openstack-infra15:23
openstackgerritafazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys  https://review.openstack.org/26966015:23
fungiknikolla: clarkb: keep in mind that merging launchpad accounts doesn't actually merge openids. login.launchpad.net is a separate service from launchpad.net. login.l.n provides a different openid for each e-mail address, while launchpad.net is simply accepting all of them as being for the same lp account15:24
*** daemontool_ has joined #openstack-infra15:24
*** rbrndt has joined #openstack-infra15:24
fungithat is the actual reason for the constant confusion over duplicate gerrit accounts15:24
clarkbyup thatd the issue here15:25
fungiwe mistakenly think that launchpad openids are a 1:1 mapping to launchpad accounts, but they're completely separate things15:25
openstackgerritMerged openstack-infra/ansible-puppet: Properly suppress undefined warning  https://review.openstack.org/26962015:25
fungiwhich just happen to share a domain name15:25
clarkbso old account is no longer accessible amd new one is created15:25
*** annegentle has quit IRC15:26
fungiwell, it's not so much that, if the user has all those e-mail addresses associated with the same lp account, then they'll still authenticate with a different openid to gerrit depending on which e-mail address they provide when logging into login.l.n15:26
*** annegentle has joined #openstack-infra15:26
*** salv-orlando has joined #openstack-infra15:27
clarkbfungi: thats not what we saw with amrith iirc15:27
fungiunless the account merge has been improved to update login.launchpad.net now as well (it didn't used to)15:27
*** pradk has joined #openstack-infra15:27
clarkbthe old openid is actaually a 40415:27
fungithat could be a more recent behavior change then15:27
clarkbso its still brokem just in new and better eays15:27
*** flepied has joined #openstack-infra15:28
*** burgerk has quit IRC15:28
knikollaclarkb: I see. I’m gonna see if I’m able to get access to the old openid, but still I’d like the IDs to be switched, so that I can use the new launchpad account with the knikolla username.15:28
*** woodster_ has joined #openstack-infra15:29
fungiknikolla: what he's going to do is make it so that your new launchpad account authenticates you to your original gerrit account rather than causing creation of a new gerrit account15:29
*** achanda has joined #openstack-infra15:29
*** flepied has quit IRC15:29
*** flepied has joined #openstack-infra15:30
knikollafungi: Oh, I see. That makes sense.15:30
*** angdraug has joined #openstack-infra15:30
*** sdake has quit IRC15:31
knikollakungi: clarkb: I need to run now to a meeting. Thanks for all the help.15:31
*** knikolla has quit IRC15:31
*** sigmavirus24_awa is now known as sigmavirus2415:31
openstackgerritZara proposed openstack-infra/storyboard-webclient: Display Assignees on Cards and Worklists  https://review.openstack.org/26970715:32
*** akshai has quit IRC15:32
smcginnisclarkb: Thanks for the update on the mirroring.15:32
smcginnisclarkb: Is there a plan to resolve that, or is there a way we can get Jenkins to avoid the non-mirrored repos so tests don't fail?15:32
*** ajmiller has joined #openstack-infra15:33
clarkbsmcginnis: fungi has resolved it15:33
fungismcginnis: clarkb: what repos?15:33
smcginnisclarkb: Oh, excellent. Thanks! And thanks fungi!15:34
clarkbfungi: it was the pypi mirror being out of sync15:34
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: We don't have tripleo.sh --flavors anymore  https://review.openstack.org/26971015:34
clarkbfungi: so was missing some newer releases15:34
fungioh, smcginnis said something about mirroring repos and avoiding broken repo mirrors15:34
smcginnisclarkb: So no real reason that particular mirror was disabled?15:34
fungiwasn't sure what "avoid the non-mirrored repos" meant15:34
clarkbsmcginnis: it was not disabled, it was just behind15:34
smcginnisclarkb: Oh, I misunderstood the earlier statements then. Good to hear.15:35
fungismcginnis: it hadn't pulled the latest release of os-brick since it was running a three-day-old snapshot of pypi basically15:35
Thelofungi: Great thanks, so it should be fine now? ( the repo?)15:35
clarkbfungi: https://login.launchpad.net/+id/McTKxdP is the old openid and does 40415:35
fungiclarkb: that's an improvement in that case. makes things a little simpler for us now15:35
smcginnisfungi: There was a comment earlier that that particular mirror was disabled in some ansible scripts.15:36
fungiThelo: probably? i haven't checked to see if it's serving the latest os-brick release there yet15:36
smcginnisfungi: It didn't sound like (at the time) there was a desire to reenable, so I was just trying to understand how to proceed.15:36
smcginnisfungi: Thanks for your help. I guess we should be good now.15:36
anteayaoh I'll just toss this out there but it would be great if sandbox could allow all registered users to really +2 +A patches rather than just telling us they have those permissions15:37
*** hashar has quit IRC15:37
fungismcginnis: well, at this point the server and its public cloud provider are going away entirely in a little over a week, so there's not a lot of incentive to remember who disabled puppet for that server and why15:37
smcginnisfungi: True! ;)15:37
anteayasomeone is teaching a class tonight and actually using sandbox as intended (woooo) would be great if they can +2 and approve patches15:37
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add networking-generic-switch project  https://review.openstack.org/26766415:37
fungianteaya: is there an acl update proposed for that? i take it our documentation contradicts the current acl?15:38
anteayawell the permissions are there15:38
clarkbnow I just need to find the launchpad/ubuntu one account to see if I can check the email addresses match up there but maybe that is overly paranoid15:38
anteayait is just in a actual patch they aren't15:38
anteayalet me get links15:39
openstackgerritMerged openstack-infra/project-config: Magnum: enable mesos voting  https://review.openstack.org/26944015:39
anteayaso right now it seems to me that gerrit is contradicting itself15:40
*** achanda has quit IRC15:40
clarkbright my theory was gerrit does something to prevent registered users from working in that manner15:40
clarkbwith a potential workaround of making a new group and adding registered users to that group15:40
*** tdurakov has quit IRC15:40
anteayaI see +1, 0, -1 when I try to review that patch15:40
*** tdurakov has joined #openstack-infra15:41
anteayawhen signed in15:41
anteayaclarkb: yup, so far that approach makes sense to me15:41
*** tdurakov has quit IRC15:43
*** _nadya_ has quit IRC15:43
*** otter768 has joined #openstack-infra15:43
*** tdurakov has joined #openstack-infra15:43
*** yamahata has joined #openstack-infra15:44
*** akshai has joined #openstack-infra15:45
*** mtanino has joined #openstack-infra15:46
clarkbfungi: want to sanity check http://paste.openstack.org/show/484297/ I am still half asleep but I think that is what is necessary15:47
*** otter768 has quit IRC15:47
clarkbanteaya: let me set up a test on review-dev really quickly15:50
anteayaclarkb: thank you15:50
anteayaclarkb: I appreciate your help even though you really should be asleep15:50
*** eharney has quit IRC15:50
*** ddieterly has quit IRC15:50
anteayaI just didn't want this item to be lost15:50
*** markvoelker_ has quit IRC15:51
*** angdraug has quit IRC15:52
clarkbanteaya: what can you vote verified on https://review-dev.openstack.org/5399 ?15:53
clarkbregistered users should be able to -2 to +215:53
clarkbthen if that fails as expected we can try the new group thing15:53
*** hdd has joined #openstack-infra15:54
*** salv-orlando has quit IRC15:54
fungiclarkb: i would just do `update account_external_ids set account_id=16465 where external_id="https://login.launchpad.net/+id/McTKxdP";` and then deactivate the new account15:57
anteayaclarkb: code review is still only -1, 0, +2 but I just voted verified +2 on that patch15:57
*** burgerk has joined #openstack-infra15:57
anteayaclarkb: but I still don't have workflow +1 available to me15:58
clarkbanteaya: ok then I think that means registered users can +215:58
clarkbso something else must be happening15:58
clarkbfungi: oh derp that is easier15:58
*** bradjones has joined #openstack-infra15:58
*** bradjones has quit IRC15:58
*** bradjones has joined #openstack-infra15:58
anteayawell registered users can vote +2 verified on sandbox in review-dev with that setup15:59
krotscheckfungi: I've got +2 on the system-config whele patches. Any chance you can roll through and take a look at them today-ish? https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open+project:openstack-infra/system-config15:59
*** achanda has joined #openstack-infra15:59
clarkbanteaya: yes which refutes the theory that registered users can't +2 things due to a gerrit bug15:59
anteayabut I wonder if we offer that as a workaround we actually are creating a different problem by confusing our users15:59
anteayaclarkb: I still can't +2 code review15:59
anteayaclarkb: nor can I +1 workflow15:59
clarkbanteaya: I used verified instead of code review because there is no code review +2 on review-dev15:59
anteayaclarkb: ah I see now, sorry I was confused16:00
mrmartinclarkb, fungi: ask-staging.openstack.org is alive again. I've enabled the openstackid auth provider, but I need a little help to roll this out into production during the next week if everything goes well.16:00
*** Ramanjaneya has joined #openstack-infra16:00
*** sdake has joined #openstack-infra16:01
*** baoli has quit IRC16:01
*** lucasagomes is now known as lucas-brb16:01
*** eharney has joined #openstack-infra16:02
openstackgerritMerged openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper  https://review.openstack.org/26928416:02
*** boris-42 has joined #openstack-infra16:03
*** achanda has quit IRC16:03
*** jsavak has quit IRC16:03
openstackgerritDevananda van der Veen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin  https://review.openstack.org/26531116:04
devanandajroll: ^16:04
*** zeih has quit IRC16:04
*** ddieterly has joined #openstack-infra16:06
fungikrotscheck: my tuesdays are afflicted with meetingitis, so what little space there is on my dance card is filling up very quickly. i'll see if i can skim those after the infra team meeting while i'm lurking the tc meeting16:06
*** hdd has quit IRC16:06
krotscheckfungi: Thanks. I'll see if I can solicit an infra-core to help me land them and provision the instances before moving on to adding the gate jobs.16:07
*** Guest17173 has quit IRC16:07
fungimrmartin: exciting! unfortunately i'm not around much at all next week since that's when we're having our quarterly foundation staff meetings/get-together16:07
krotscheckfungi: I just want to make sure you're aware of what we're doing in case the world goes splodey16:07
fungikrotscheck: much appreciated!16:08
jrolldevananda: just one comment - also, we'll need to get the +A on that before we land the plugin things, so they land together16:08
*** nmagnezi has quit IRC16:09
sdagueclarkb: I think something is wrong with the common flags code and internap16:09
openstackgerritAngus Lees proposed openstack-infra/zuul: Explicitly set basepython=python2.7  https://review.openstack.org/26973116:09
devanandajroll: I'd love to test the plugin thing before landing it, honestly16:09
*** Qiming has quit IRC16:09
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use /opt not /etc as the system-config location  https://review.openstack.org/26973216:09
mrmartinfungi: ok, I no problem, as other infra cores have access to ask.o.o we can do it without you. the askbot upstream repo changes really broke the staging, I hope it won't happen with production, I'll prepare every documentation required for a seamless upgrade.16:09
*** bpokorny has quit IRC16:10
clarkbsdague: we may have to trim the flag list then (I thought it was fairly minimal though)16:10
sdagueclarkb: it's not dynamically generated?16:10
jrolldevananda: yeah, I agree, but it can be tested locally you know :)16:11
clarkbsdague: no I think its basically a standard libvirt cpu type less a couple flags but I would have to go reread the code16:11
*** _nadya_ has joined #openstack-infra16:11
jrolldevananda: the downside to testing it before landing is we'll be running each test twice (once from tempest, once from ironic tree), until we re-update the regex16:11
*** markus_z has quit IRC16:11
*** moravec has quit IRC16:11
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use /opt not /etc as the system-config location  https://review.openstack.org/26973216:12
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949216:12
mordredfungi, clarkb: real quick - had an oops ^^16:12
clarkbsdague: # Create a gate64 cpu model that is core2duo less monitor and pse3616:12
fungimordred: which one--both?16:12
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941116:12
clarkbwhich includes vme and ssse316:12
devanandajroll: indeed. anyway, I'm around all week and will keep an eye on moving this (and hte neutron work) forward16:13
mwhahahais there a way to tell which DC a job ran in? We're running into some issues where some jobs are taking excessively long to download source which is causing failures on some nodes but not others16:13
*** wei-wang has quit IRC16:14
clarkbmwhahaha: it is encoded in the test node hostname16:14
*** moravec has joined #openstack-infra16:14
mwhahahaah so it looks like the hpcloud nodes16:14
jrolldevananda: awesome, ty16:14
fungialso try to avoid downloading things in jobs. we cache a lot of source on the workers already--what source are you downloading?16:14
* mgagne feels better now16:14
sdagueclarkb: ok, you up for proposing that? Because it's in devstack, we're going to have to update every branch16:14
mwhahahathe puppet modules for unit tests16:14
clarkbsdague: sure16:14
mordredfungi: no - just 26973216:15
sdaguethat's the cause of some chunk of the gate backlog16:15
fungimwhahaha: oh, so not openstack puppet modules, just other random dependencies?16:15
*** fhubik is now known as fhubik_brb16:15
*** fhubik_brb is now known as fhubik16:15
sdaguemriedem: any idea if we have a bug for this yet?16:15
mwhahahafor these jobs yea16:15
*** Liuqing has joined #openstack-infra16:16
*** zhurong has joined #openstack-infra16:16
mwhahahait looks like there might be some overloaded nodes in the hpcloud because some are OK and some are not, https://review.openstack.org/#/c/262587/ the gate-fuel-library-puppet-unit-3.{7,8}-dsvm-centos7 took an hour and timed out while the others were OK in < 10 mins16:16
*** gongysh has joined #openstack-infra16:16
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949216:16
openstackgerritMonty Taylor proposed openstack-infra/system-config: Don't rsync puppet on localhost  https://review.openstack.org/26973616:16
*** jsavak has joined #openstack-infra16:16
mordredfungi, clarkb: ALSO - 269736 if you get a chance16:16
*** jpr has joined #openstack-infra16:18
clarkbjust reading up on cpu flags we may actually want vme, but we can figure that out later16:18
openstackgerritSean Dague proposed openstack-infra/elastic-recheck: add signature for internap cpu model issue  https://review.openstack.org/26973816:18
clarkbits all emulated anyways so any performance benefit is likely to be tiny16:19
sdagueclarkb: yeh, but internap doesn't have vme16:19
sdagueso if that's in our cloud, we can't have it16:19
sdagueclarkb: ideally it would be nice for the gate64 model to be defined in devstack-gate, passed to devstack some how, and let devstack change this. But we can do that once things are fixed.16:20
*** moravec has quit IRC16:20
clarkbsdague: I was thinking maybe we provide feedback to them that they should have this16:21
clarkbbut I need to do more research before I care that strongly16:21
sdagueclarkb: sure, that's fine16:22
clarkb(it apperas that it does allow for more performant interrupts)16:22
mriedemsdague: for what?16:22
sdaguewhen did internap get added to the mix?16:22
*** jordanP has quit IRC16:22
clarkbsdague: first week of janaury16:22
mriedemoh http://logs.openstack.org/34/254434/10/gate/gate-grenade-dsvm-multinode/fda9fbe//logs/old/screen-n-cpu.txt.gz#_2016-01-19_15_50_50_26916:22
sdaguemriedem: yeh that16:23
mriedemsdague: i haven't seen that one16:23
mriedemi can report one quick16:23
sdaguemriedem: I already did16:23
clarkboh was there a bug I should've used?16:23
clarkbI can update commit messages16:23
sdagueclarkb: sure16:23
*** fawadkhaliq has joined #openstack-infra16:24
*** egonzalez has joined #openstack-infra16:25
mtreinishsdague: fwiw: http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj0116:25
anteayafor me newton will be named in honour of fig newtons16:26
anteayaand I need to greive the loss of Om as a release name for a bit first16:26
sdaguemtreinish: I guess internap can't pass on multinode tests16:27
sdaguethat's where we specify the kvm arch16:27
*** Jeffrey4l has joined #openstack-infra16:27
mordredanteaya: I know - I was hoping for Om myself - but apparently it's a popular enough name that  many other non-openstack people got to it before we did16:27
clarkbfungi: http://paste.openstack.org/show/484307/ that look better?16:28
mtreinishsdague: yeah, that's what it looks like16:28
*** lucas-brb is now known as lucasagomes16:28
anteayamordred: ah I can see that16:28
anteayaI do like fig newtons though16:28
sdaguemtreinish: how does your os health handle multinode jobs16:28
fungiclarkb: yep16:28
sdaguedoes it register against both providers?16:28
*** Jeffrey4l has quit IRC16:28
sdagueor only the master16:28
*** Jeffrey4l has joined #openstack-infra16:29
clarkbsdague: multinode is always within a single region16:30
openstackgerritMerged openstack-infra/elastic-recheck: add signature for internap cpu model issue  https://review.openstack.org/26973816:30
sdagueclarkb: oh, when did that change?16:30
clarkbsdague: it has always been the case16:30
fungiclarkb: is that the same account as http://lists.openstack.org/pipermail/openstack-dev/2016-January/084382.html is about?16:30
sdaguehmmmm... really?16:30
clarkbsdague: yes16:30
mordredzaro, pleia2: when you get in - I updated https://review.openstack.org/249492 based on your reviews and would love follow up reviews16:30
fungisdague: yeah, that was an original design constraint of the feature16:31
clarkbfungi: no I don't think so, different email address at least16:31
mordredzaro, pleia2: also, I believe we can roll that live today16:31
fungisdague: same provider, region and az16:31
clarkbfungi: it may not be az I think due to how we have azs in hpcloud16:31
clarkbbut once hpcloud goes away that will be true16:31
clarkbI can't remember if we allow the az to float or not16:31
*** kzaitsev_mb has joined #openstack-infra16:32
sdagueok, I thought I remember being told that wasn't going to be constrained, which is why we built all the cpu flags stuff16:32
fungii thought multinode in hpcloud was tied to all subnodes being in the same neutron network (because we defined them as providers in nodepool) and then each of those neutron networks went to one and only one az?16:32
sdaguebecause live migrating from rax -> hp triggered weird flag cross overs16:33
anteayamtreinish: so this is the job failure rate: http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj01 so this is tracking the host's ability to build the node and provision for the job?16:33
fungisdague: i thought it was that rackspace had inconsistent cpu between instances that we ran into that issue16:34
anteayamtreinish: it doesn't have anything to do with if the actual tests failed or succeeded?16:34
SamYaple/w/win 116:34
mordredfungi: yes - I believe that is true- it is tied to az as a happenstance of how we neutron in hp16:34
mtreinishanteaya: right now it's only counting if the tests failed16:34
sdaguefungi: the overlap game I remember playing was hp -> rax16:34
mordredfungi: not because the code itself is az aware16:34
mtreinishif devstack-gate or devstack fails, it's not counted16:34
anteayamtreinish: oh okay16:34
*** sridhar_ram has joined #openstack-infra16:35
sdagueif this is all in one region now, and the same flavor, my guess is we could probably get rid of this entirely16:35
fungisdague: interesting. i don't recall us ever having the ability to try that (other than setting it up manually)16:35
clarkbsdague: the cpu flag stuff is necessary because within a region we can have vastly different cpu flags16:35
anteayaokay yes, I was aware that was how openstack-health operates I just wondered if it was different for the part where you are tracking providers, thanks for the clarification16:35
sdagueclarkb: for the same flavor?16:35
clarkbsdague: yes16:35
clarkbsdague: so the issue wasn't rax vs hp so much as hp vs hp and rax vs rax16:35
*** baoli has joined #openstack-infra16:35
sdagueon just rax?16:35
*** markvoelker has joined #openstack-infra16:36
sdagueon hp the definitely smoothed it all out16:36
clarkbsdague: yes I think hp may have consistent flags but rax didn't16:36
mtreinishanteaya: see: http://blog.kortar.org/?p=279 the "Failures Before Test Runs aren’t counted" section (I really need to add heading links)16:36
mordredcloud providers not so much with using exactly the same hardware on their hypervisors16:36
*** moravec has joined #openstack-infra16:36
*** flepied1 has joined #openstack-infra16:36
mtreinishanteaya: we do have a way forward on that, so it'll be fixed before too long16:36
sdaguemordred: that surprises me at the flavor level16:36
*** flepied has quit IRC16:36
mordredsdague: many things surprise me at all the levels16:36
clarkbsdague: so the idea was find subset that worked for rax and hp not because we mixed rax and hp but because we run jobs on all those cpu flags16:36
mordredsdague: but I've grown to expect the most unexpected and worst things from clouds16:37
sdagueright, now that we are mixing all the world, I think we've got to reset expectations. Possibly with a dedicated arch per provider.16:37
clarkbfungi: ok that account is all munged inthe db now16:37
anteayamtreinish: now I understand, thank you16:37
*** salv-orlando has joined #openstack-infra16:37
sdaguebecause slicing out all the flags for lowest common denominator across all clouds is going to potentially sacrifice performance we want in places where we can have it16:37
mordredwould it be useful to do a cpu flag detection at node creation and storing that somewhere that we can read it when setting up the flags for devstack?16:38
*** bpokorny has joined #openstack-infra16:38
clarkbmordred: well its already done for us by the kernel16:38
clarkbthe tricky bit is you have to install the flags during the devstack run because libvirt16:38
mordredthat's why I was saying at node creation16:38
clarkbso we could update devstack to accept a list and do the right thing rather than just being hard coded16:38
clarkbmordred: node creation is too early16:38
mordredor, multi-node association time16:39
nibalizergood morning16:39
*** bpokorny has quit IRC16:39
mordredthe first time we, as nodepool, know that the two nodes are going to be associated with each other16:39
clarkbmordred: ya then16:39
*** abregman has quit IRC16:39
mordredwe cna communicate to both of them what the cpu flag set on each is16:39
anteayamtreinish: you only need one more devstack core reviewer +2 for this to go in, well done: https://review.openstack.org/#/c/234232/16:39
*** egonzalez has quit IRC16:39
anteayagood morning nibalizer16:39
mordredbut then the logic for finding the right flags16:39
mordredshould likely live in devstack16:39
*** bpokorny has joined #openstack-infra16:39
mordred"hey, devstack, I've got these lists of cpu flag lists"16:40
mtreinishanteaya: I actually think I caught a bug in that last night, so I might need to respin it16:40
clarkbI mean this would be a trivial fix if nova flavors enforced this >_>16:40
mordredcause I agree with sdague - doing this as the LCD for all of our cloud is going to be continually unfun16:40
mordredbut mapping discreet combinations is likely less suck16:40
clarkbmordred: yup, that said the cloud should solve this16:40
mordredclarkb: totes16:41
clarkbso this would still be a bandaid16:41
sdagueanyway, clarkb's fixes should be promoted16:41
fungialso this introduces yet another possible corner case for differentiation that can impact jobs. it might be a detail we want to add to logstash for the job info too?16:41
sdague2 current gate resets are that issue16:41
anteayamtreinish: nice catch16:41
mordrednibalizer: when you wake up ... today is the day  - all of the pre-reqs are in and working16:42
openstackgerritDevananda van der Veen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin  https://review.openstack.org/26531116:42
sdagueor internap turned off until this settles out and those get in16:42
sdagueor blacklist it from multinode jobs16:44
openstackgerritSam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic  https://review.openstack.org/26968416:45
mtreinishanteaya: well when I looked at: https://review.openstack.org/234447 the subunit stream had the devstack stream had the devstack "test" listed twice16:45
jeblairmordred, clarkb: yeah, probably not at node association time though -- i think d-g should grab the info and either do the intersection calculation or pass it to devstack to do it.16:45
openstackgerritDevananda van der Veen proposed openstack-infra/project-config: Make the ironic-parallel job experimental  https://review.openstack.org/26975716:45
*** annegentle has quit IRC16:46
anteayamtreinish: ah16:46
*** Liuqing has quit IRC16:46
clarkbjeblair: I think passing the intersection to each devstack run is probably sanest16:46
*** moravec has quit IRC16:46
mordredclarkb, jeblair: ++16:47
anteayamtreinish: quite the prompt you have there16:47
openstackgerritJames E. Blair proposed openstack-infra/project-config: Don't use internap for 2-node labels  https://review.openstack.org/26975916:47
jeblairsdague: ^16:47
mtreinishanteaya: ?16:47
*** samuelBartel has quit IRC16:48
anteayamtreinish: http://paste.openstack.org/show/476207/16:48
jeblairclarkb: where are your in-flight fixes?16:48
anteayajeblair: https://review.openstack.org/#/q/topic:bug/153579916:48
clarkbjeblair: 269739 269742 26974316:48
jeblairoh, i didn't add query support for the gertty link thing16:49
sdaguejeblair: +A16:49
mtreinishanteaya: hah, I didn't remember that I posted a paste with the same issue back in Oct16:49
*** _nadya_ has quit IRC16:49
anteayamtreinish: :) well you are consistent in that you found it again16:50
*** annegentle has joined #openstack-infra16:50
*** moravec has joined #openstack-infra16:50
mtreinishanteaya: and the prompt is even better on my terminal, it's got colors16:50
anteayaha ha ha16:50
anteayatoo bad paste can't share that16:50
clarkbthe things to keep in mind for this is you need to make an intersection with a complete view of the nodes you have running right now. You then have to map those flags onto the flags libvirt understands (this was mostly 1:1 when I checked a year ago). You have to write the mapping file out after libvirt is installed, then do a service stop start in order for them to be properly loaded16:50
*** Liuqing has joined #openstack-infra16:50
*** placko_ has quit IRC16:51
mtreinishanteaya: http://i.imgur.com/YvGLkSC.png16:51
clarkbfor these reasons it was simplest at the time to just make a hardcoded sane cpu type and have devstack install it itself16:51
anteayamtreinish: aha ha haha16:51
*** nelsnelson has quit IRC16:51
*** salv-orlando has quit IRC16:51
*** Liuqing has quit IRC16:51
openstackgerritJames E. Blair proposed openstack-infra/project-config: Revert "Don't use internap for 2-node labels"  https://review.openstack.org/26976316:51
*** yamahata has quit IRC16:51
jeblairclarkb, sdague: ^ revert depends on clarkb's internap fixes16:52
*** nelsnelson has joined #openstack-infra16:52
*** gyee has joined #openstack-infra16:53
*** Liuqing has joined #openstack-infra16:53
*** Liuqing has quit IRC16:53
*** rcernin has quit IRC16:53
sdaguejeblair: thanks16:54
openstackgerritZara proposed openstack-infra/storyboard-webclient: Display Assignees on Cards and Worklists  https://review.openstack.org/26970716:54
*** apoorvad has joined #openstack-infra16:54
*** matrohon has quit IRC16:55
*** annegentle has quit IRC16:55
clarkbfungi: it would be nice (though maybe it breaks the protocol) if openids could be redirected from one to another in the case of an account merge16:56
clarkbfungi: and that permanent redirect would indicate to the consumer side to use the new url16:56
*** ashleighfarnham has joined #openstack-infra16:57
anteayazaro: when you are about, review.o.o sandbox is not allowing all registered users to +2 and +A patches even though it reports all registered users have permission to do so16:57
*** scheuran has quit IRC16:57
fungiclarkb: well, this has got me thinking about how to go about our transition to openstackid.org for gerrit authentication16:58
anteayazaro: it would be great if we can figure out how to give +2 +A to all registered users on sandbox16:58
*** ajmiller has quit IRC16:58
*** david-lyle has quit IRC16:58
*** ajmiller has joined #openstack-infra16:58
*** dslevin has joined #openstack-infra16:59
*** gongysh has quit IRC16:59
*** sshnaidm has quit IRC16:59
fungiclarkb: if we get to the point where we can make it optional to log in with either login.launchpad.net or openstackid.org, and if we can get openstackid to support id lookups by e-mail address, then i can batch add all the openstackid.org ids to gerrit's database for any existing accounts based on a search for all their known e-mail addresses17:00
*** sshnaidm has joined #openstack-infra17:00
*** jsavak has quit IRC17:00
jeblairinfra-root: https://threatpost.com/serious-linux-kernel-vulnerability-patched/115923/ is a kernel local privilege escalation that it scheduled to get patched today: CVE-2016-072817:00
*** jsavak has joined #openstack-infra17:00
*** claudiub has joined #openstack-infra17:00
fungijeblair: oh, yep, i saw that come through oss-security and hadn't looked closely enough at it yet. thanks!17:00
jeblairthere's apparently poc code for it out there... which is lovely.17:01
*** e0ne has quit IRC17:01
fungijeblair: on that note, care to weigh back in on 267778 now that it works?17:01
fungiand i guess we'll want to move over to the incident channel when it comes time to do some reboots17:02
*** apoorvad has quit IRC17:02
*** salv-orlando has joined #openstack-infra17:03
jeblairfungi: any reason not to put the class apt purge into the conditional?17:03
*** MarkAtwood1 has joined #openstack-infra17:03
fungijeblair: because it's replacing include apt17:04
jeblairfungi: ah, and you want apt in there regardless17:04
fungii mean, i could do an else with the old include apt if we really want17:04
*** sdake has quit IRC17:04
*** sdake_ has joined #openstack-infra17:04
jeblairfungi: no prob, approved.  just felt like i was missing something (and i was)17:04
jeblairclarkb: what should i look at in nodepool-land?17:05
jeblair(before the storm hits)17:05
jeblairi found https://review.openstack.org/26886717:05
*** david-lyle has joined #openstack-infra17:06
*** mrmartin has quit IRC17:06
*** ddieterly has quit IRC17:06
*** jlanoux has quit IRC17:07
*** flwang has joined #openstack-infra17:08
*** mrmartin has joined #openstack-infra17:08
*** ccrouch has quit IRC17:08
*** ddieterly has joined #openstack-infra17:08
clarkbjeblair: there is that one, ianw's stack at https://review.openstack.org/269266 and https://review.openstack.org/267865 fixes logging17:08
*** ddieterly has quit IRC17:09
clarkbjeblair: I got tests to rerun against them way too early this morning and I think they look good and can start approving them17:09
*** rcernin has joined #openstack-infra17:09
*** yamahata has joined #openstack-infra17:09
*** dtantsur is now known as dtantsur|afk17:09
openstackgerritMerged openstack-infra/project-config: Don't use internap for 2-node labels  https://review.openstack.org/26975917:10
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Also handle work exceptions in builders  https://review.openstack.org/26977417:10
jeblairclarkb, greghaynes: ^ greghaynes change to add the failure handler lgtm with that addition ^17:10
clarkboh there is also ianw's fixes for the dib elements17:11
clarkblet me find that one17:11
*** hdd has joined #openstack-infra17:11
clarkboh sweet it merged already17:11
openstackgerritMerged openstack-infra/system-config: Puppet the Ubuntu sources.list files on servers  https://review.openstack.org/26777817:11
pleia2mordred: yay docs, looking now17:12
*** moravec has quit IRC17:12
mordredpleia2: I apparently have an unhappy docs job17:12
jeblairclarkb: does "from nodepool import nodepool; from nodepool import builder;"  read weird to you?  like where is builder coming from?  nodepool or nodepool.nodepool? :)17:13
pleia2mordred: I have a comment inline too, posting now17:13
jeblairclarkb: i guess it's doing the right thing.... but i think i might use the slightly more clear but verbose forms17:14
clarkbjeblair: I hadn't thought of it that way :) though when I worked around the statsd thing I did use nodepool.builder to be explicit17:14
mordredpleia2: woot17:14
*** asselin has quit IRC17:14
pleia2mordred: nit-ish but easier on the eyes link-wise17:14
mordredpleia2: oh - really?17:14
jeblairwell, i better check the dsvm test to make sure it is doing the right thing17:14
mordredpleia2: do I need to do any path things?17:14
*** prometheanfire has joined #openstack-infra17:14
pleia2mordred: should work with magic17:14
prometheanfiregate stuck or backed up?17:15
*** Daisy has joined #openstack-infra17:15
jeblairprometheanfire: what makes you ask?17:15
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949217:15
nibalizermordred: today is a pooey day for me but I can try to help17:15
prometheanfire265662 has been waiting over 5 hours17:15
prometheanfirejust curious17:15
nibalizerI'll be around more in the afternoon17:15
mordredpleia2: neat. I've learned something today17:16
*** ifarkas has quit IRC17:16
mordrednibalizer: cool. I'm thinking of rolling it live after the meeting17:16
*** ddieterly has joined #openstack-infra17:16
nibalizermordred: ya I have an internal meeting at 11 then a tour thing at 12 and then I'm available17:17
nibalizerthis means my participation in the infra meeting will be limited to nonexistant :(17:17
pleia2mordred: the rendering here is an example of how it works: http://docs.openstack.org/infra/system-config/sysadmin.html#adding-a-new-server17:17
mordrednibalizer: well, luckily, so far it all seems to be working, so it should be very little needed17:17
jeblairprometheanfire: according to http://status.openstack.org/zuul/ the top of the gate queue is at 17 hours, and the test node graph at the bottom looks like we're at capacity for our test node use17:18
openstackgerritZara proposed openstack-infra/storyboard-webclient: Display Assignees on Cards and Worklists  https://review.openstack.org/26970717:18
pleia2:file:`manifests/site.pp turns into "system-config: manifests/site.pp" with the link to git.o.o17:18
mordredpleia2: that's so super fancy17:18
pleia2it's one of my favorite doc-things17:18
jeblairprometheanfire: the jobs launched graph seems to indicate we're launching 1,500 jobs per hour17:18
jeblairprometheanfire: so i think things are moving17:18
*** Daisy has quit IRC17:19
jeblairprometheanfire: the failure rate of some of the common gate jobs has been high recently: http://grafana.openstack.org/dashboard/db/tempest-failure-rate17:20
*** Jeffrey4l has quit IRC17:22
jeblairclarkb: it looks like the nodepool dsvm job may be flaky... have you looked into potential causes?  (at least to determine whether it's internal to nodepool or external?)17:22
jeblairclarkb: one i'm looking at says: 2016-01-18 23:03:56,769 ERROR nodepool.SnapshotImageUpdater: Exception updating image trusty-server-cloudimg-amd64-disk1 in devstack:17:23
jeblairException: Timeout waiting for ssh access17:23
*** HeOS has quit IRC17:23
*** derekh has quit IRC17:23
jeblairclarkb: it ran on ovh; i wonder if we need to update our ssh timeout in the dsvm job?17:24
clarkbjeblair: it used to be external until last night. global requirements was constraining dib to a broken version but that was updated to version 1.7.1 last night17:26
clarkbjeblair: now the fail rate seems to be closer to 50% instead of 100% and I think its due to timeouts17:26
clarkbqemu performance is not so great17:27
*** _et_ has quit IRC17:28
*** mrmartin has quit IRC17:28
*** shashank_hegde has joined #openstack-infra17:29
*** _et_ has joined #openstack-infra17:29
openstackgerritArmando Migliaccio proposed openstack-infra/irc-meetings: Move the Neutron drivers meeting  https://review.openstack.org/26978017:30
clarkbit is possible that if we swap in dropbear we could make that go quicker17:30
*** mudassirlatif has joined #openstack-infra17:30
* clarkb makes that change for the lols17:30
AJaegerany project-config core to +2A https://review.openstack.org/#/c/261994/ and https://review.openstack.org/#/c/268276/ ,please?17:31
jeblairokay, i think we need to promote some debug log messages to info...17:31
*** zeih has joined #openstack-infra17:31
AJaegerAlso, armax has a patch for jeepyb that I think will help bug triage and it needs another +2:  https://review.openstack.org/253229, please17:32
*** arxcruz has joined #openstack-infra17:32
*** acruz has quit IRC17:32
clarkboh wait no can't do that bceause it doesn't support scp/sftp17:34
clarkbwhich is why we don't cirros17:34
pleia2wow, it never occurred to me that dropbear didn't do scp17:36
pabelangerjeblair: fungi: are in good shape for zuul release, or still waiting of a bug fix?17:36
*** davideagnello has joined #openstack-infra17:37
clarkbpleia2: ya I remmeber having a hard time debugging this the first time aroudn, then my memory returned to me17:37
*** dslevin has quit IRC17:37
*** apoorvad has joined #openstack-infra17:37
*** ccrouch has joined #openstack-infra17:37
anteayaAJaeger: you are changing the template running on keystone from python-jobs to python-jobs-linters, thereby removing the python-jobs template from keystone, and you are creating settings for a new job gate-keystone-linters which I don't see being used on keystone17:37
anteayaAJaeger: so I'm confused17:38
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Normalize some log lines in builder  https://review.openstack.org/26978417:38
*** sfinucan has quit IRC17:39
AJaegeranteaya: python-jobs-linters template includes gate-keystone-linters17:39
jeblairclarkb, greghaynes; ^ that will help me from going cross-eyed trying to read the dsvm logs17:39
AJaegeranteaya: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n32117:39
*** shashank_hegde has quit IRC17:41
AJaegerthanks, anteaya17:41
anteayaand I'm already +2 on 26827617:42
jeblairclarkb: i'm happy that https://review.openstack.org/267865 starts both builder and nodepoold and they both log... i note you have not approved likely because of rechecks... would you object to my approving?17:42
anteayathanks for your patience with me17:42
clarkbjeblair: no objection, I was hoping to confirm things work more generally but I agree there are logs there which is an improvmeent17:42
clarkbjeblair: will make debugging everything else simpler too17:43
jeblairclarkb: k, i'll approve then17:43
AJaegersdague: you were right with the multinode change. Thanks for not trusting me ;)17:43
*** otter768 has joined #openstack-infra17:44
jeblairpabelanger: the memory graphs appear to not be constantly increasing, however, we are using 3x the amount of memory we are using before, and there are occasional serious usage bumps.17:44
AJaegeranteaya: ok, need to ask another friendly soul to review 26827617:44
jeblaircody-somerville-, jhesketh: ^17:44
jeblairpabelanger: i do not feel comfortable making a zuul release under those circumstances.17:44
openstackgerritafazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys  https://review.openstack.org/26966017:45
jeblairpabelanger: i certainly can't stand behind it.17:45
AJaegeranteaya: could you review https://review.openstack.org/#/c/269460/ as well, please?17:45
clarkbjeblair: the other one that should be safe to approve is ianw's stack17:46
*** ddieterly has quit IRC17:46
anteayais it just me or is there a really big spike in zuul outbound eth0 traffic? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=397&rra_id=all17:46
*** bardia has joined #openstack-infra17:47
*** ddieterl_ has joined #openstack-infra17:47
jeblairclarkb: yeah working on that now17:47
clarkbanteaya: as the size of status.json gets bigger and more people wonder why their tests haven't reported back yet that will likely cascade17:47
jeblairclarkb: your vote is absent on https://review.openstack.org/26930317:47
anteayaclarkb: oh okay, I think I understand17:48
sdagueAJaeger: thanks for providing the specific reason why you thought it was wrong, that let me do the compares to make sure I wasn't imagining things17:48
clarkbjeblair: oh I wanted to see test results on that one since ssh is potentially affected17:48
clarkbjeblair: let me look17:48
AJaegersdague: ;)17:48
*** zeih has quit IRC17:48
openstackgerritMerged openstack-infra/nodepool: Use common logging for nodepool applications  https://review.openstack.org/26786517:48
*** jistr has quit IRC17:48
AJaegersdague: could you review https://review.openstack.org/#/c/268276/ , please?17:48
clarkbjeblair: http://logs.openstack.org/03/269303/1/check/gate-dsvm-nodepool/a5bbdf2/logs/nodepool-image-list.txt.gz doesn't look too good17:49
*** otter768 has quit IRC17:49
*** ZZelle has quit IRC17:49
anteayaAJaeger: it appears you are going for making linters the new fashion trend17:49
AJaegeranteaya: yeah, I suggest it - but will not -1 anything here17:49
sdagueAJaeger: +A17:49
clarkbjeblair: it hit http://logs.openstack.org/03/269303/1/check/gate-dsvm-nodepool/a5bbdf2/logs/screen-nodepool.txt.gz#_2016-01-18_23_03_56_76917:50
AJaegerthanks, sdague and anteaya17:50
*** ZZelle has joined #openstack-infra17:50
clarkbjeblair: which may or may not be related to paramiko changing17:50
clarkbjeblair: maybe we should recheck it and see if the dib build works at least? that seems more reliable generally17:50
*** burgerk_ has joined #openstack-infra17:50
AJaegeranteaya: I'm currently trying to remove the extra bandit jobs and merge them in the lint target...17:50
*** burgerk has quit IRC17:51
anteayaoh okay17:52
openstackgerritMerged openstack-infra/project-config: Use linters job for keystone starting with mitaka  https://review.openstack.org/26199417:52
*** angdraug has joined #openstack-infra17:52
jeblairclarkb: okay.  my reasoning was that if ssh failed, i expected it to do in some less subtle way (like on import, or when trying to initiate the connection).  but i also agree that test hasn't fully exercised all of the ssh paths, so we can wait if you want.17:52
AJaegeranteaya: keystone repos where the only ones having it voting, so patches are up for all of these. keystone is merged, keystoneclient approved, keystonemiddleware under review - and each has a corresponding project-config change17:53
openstackgerritMerged openstack-infra/project-config: Define criteria for skipping multinode-live-migration  https://review.openstack.org/26766917:53
*** unicell has quit IRC17:53
*** acruz has joined #openstack-infra17:53
*** flwang has quit IRC17:53
clarkbjeblair: on the flip side it is easy enough to revert that change should we have any trouble with it17:53
AJaegeranteaya: so next one to go in is https://review.openstack.org/269270 - but depends-on is not merged yet17:53
AJaegerbut if you want to review already...17:53
clarkbI will go ahead and approve it (I just realized that we don't daemonize in dsvm jobs anyways)17:54
openstackgerritMiles Gould proposed openstack/requirements: Bump minimum python-ironicclient version  https://review.openstack.org/26632017:54
jeblairclarkb: heh, how true :)17:54
*** arxcruz has quit IRC17:54
jeblairclarkb, greghaynes, ianw, pabelanger: i think that's all the nodepool changes approved17:54
*** bardia has quit IRC17:54
clarkbgerrit says it is in merge conflict now17:54
clarkbI am guessing with common longging?17:55
jeblairclarkb, greghaynes, ianw, pabelanger: except mine: https://review.openstack.org/269784  :(17:55
clarkbjeblair: want ot rebase ianw's stack to address conflicts and I will review your change?17:55
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941117:55
jeblairclarkb: can do17:55
openstackgerritMerged openstack-infra/nodepool: Don't use global statsd  https://review.openstack.org/26788517:58
pabelangerjeblair: great!17:58
anteayaAJaeger: okay thanks for letting me know what is going on, I appreciate it18:00
*** mdenny has joined #openstack-infra18:00
*** zhurong has quit IRC18:01
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Remove workaround for early paramiko import  https://review.openstack.org/26930318:02
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Revert "Import nodepool.builder late for statsd"  https://review.openstack.org/26926618:02
jeblairclarkb: i couldn't help myself and i made the imports longer :)18:02
openstackgerritMerged openstack-infra/project-config: Add py27 and docs to steth  https://review.openstack.org/26946018:03
*** sdake has joined #openstack-infra18:03
openstackgerritMerged openstack-infra/project-config: Use zuul-cloner for hacking integration test  https://review.openstack.org/26827618:03
anteayaAJaeger: thanks then I don't have to figure it out again later18:03
clarkbjeblair: +2 to both18:04
openstackgerritWalter A. Boring IV (hemna) proposed openstack/requirements: Update the os-brick version  https://review.openstack.org/26979318:04
anteayaI'm going to grab some lunch before meeting18:04
anteayaI'm ready to answer the channel best I can when the security patch comes out18:05
*** hemna has joined #openstack-infra18:05
AJaegeranteaya: thanks - in that case https://review.openstack.org/#/c/269261/ for after your lunch ;)18:06
*** sdake_ has quit IRC18:06
*** sputnik13 has joined #openstack-infra18:06
*** ashishb has joined #openstack-infra18:06
anteayaokay thanks18:07
*** fawadkhaliq has quit IRC18:08
rhedlindI have been having issue with jobs not getting kicked off on my CI ever since I upgraded to the latest zuul last week. http://paste.openstack.org/show/484316/ shows the output in the zuul debug log and layout.yaml (has not changed). If anyone has a minute to spare to help me come up with ideas what might be wrong I would appreciate it.18:08
tdurakovAJaeger, hi18:08
*** Swami has joined #openstack-infra18:08
AJaegerhi tdurakov, sorry for wrong review ;(18:09
AJaegerfungi, ok to still add https://review.openstack.org/#/c/262545/ as spec for approving to the agenda?18:09
tdurakovAJaeger, np, could you help me to check things?18:10
*** kzaitsev_mb has quit IRC18:10
fungiAJaeger: sure, if you think it's ready to go up for a council vote now18:10
fungiif people want a longer warning, they can speak up in the meeting and we'll adjust the voting deadline accordingly18:11
*** ashishb has quit IRC18:11
tdurakovthe change was merged recently, so I've triggered check experimental for https://review.openstack.org/#/c/252283/ but haven't seen new job in zuul18:11
*** nmagnezi has joined #openstack-infra18:11
*** zeih has joined #openstack-infra18:12
AJaegertdurakov: I'll check18:13
AJaegerfungi, fine with me18:13
*** BobBall is now known as BobBall_AWOL18:13
*** Hal has joined #openstack-infra18:14
tdurakovAJaeger, thank you, wanted to know whether this change already populated on ci servers or not18:14
*** Hal is now known as Guest9543518:14
sdaguejeblair / clarkb - http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22bash:%20install_package:%20command%20not%20found%5C%22%20AND%20tags:console%20AND%20voting:1%20AND%20build_queue:gate&from=864000s18:14
AJaegertdurakov: seems not - zuul still says "Tue Jan 19 2016 09:55:41 GMT+0100 (CET)"18:14
*** jtomasek_ has quit IRC18:14
AJaegerSo, let's wait until zuul is ready...18:14
sdagueit might make sense to turn of rax-iad from the 2 node allocator as well18:14
*** bardia has joined #openstack-infra18:14
sdagueit seems like it can't reliably get the second node up18:14
AJaegertdurakov: I'm having some jobs merged as well and wait for them, keep an eye on it.18:14
tdurakovAJaeger, ok, acked18:15
sdague20 gate fails in the past week18:15
sdague12 in the last 24 hours18:16
openstackgerritGraham Hayes proposed openstack/requirements: Bump version of debtcollector to 1.2.0  https://review.openstack.org/26769318:16
*** e0ne has joined #openstack-infra18:16
*** qeas has joined #openstack-infra18:17
AJaegersomething fishy going on with zuul? Last reconfigure was 9:55, so 9 hours ago and a change merged two hours ago that should have updated layout.yaml (269284 )18:17
*** sabeen1 has quit IRC18:17
*** moravec has joined #openstack-infra18:17
*** eil397 has joined #openstack-infra18:18
AJaegerlooking at puppetmaster, I see that zuul is unreported since 4 hours ;818:18
jeblairmordred: any chance some puppet stuff caused that ^ ?18:18
*** pradk has quit IRC18:19
clarkbsdague: that looks like grenade is failing to setup the devstack repo on the subnode18:19
mordredjeblair: yes. we are waitingo n a fix to land18:19
clarkbbut I don't see any logs18:19
openstackgerritMerged openstack-infra/system-config: Use /opt not /etc as the system-config location  https://review.openstack.org/26973218:19
mordredthere is patch number one (right on queue)18:19
openstackgerritSean Dague proposed openstack-infra/project-config: don't do multinode jobs in rax-iad  https://review.openstack.org/26979918:19
jeblairmordred: ah heh :)18:19
AJaegermordred: how many do we need? ;)18:19
*** ashishb has joined #openstack-infra18:19
mordredone more18:20
AJaegerthanks, jeblair and mordred18:20
AJaegertdurakov: see above, we first need to fix puppet18:20
*** knikolla has joined #openstack-infra18:21
*** piet has joined #openstack-infra18:21
tdurakovAJaeger, ok18:21
jeblairclarkb, sdague: because of this? http://logs.openstack.org/27/266827/1/gate/gate-grenade-dsvm-multinode/a453e43/console.html#_2016-01-19_09_00_45_03918:22
clarkbsdague: http://logs.openstack.org/27/266827/1/gate/gate-grenade-dsvm-multinode/a453e43/logs/devstack-gate-setup-workspace-new.txt.gz18:22
*** abregman has joined #openstack-infra18:22
clarkbjeblair: yup that seems to be the genesis of the problem which leads to the thing I linked18:22
*** qeas has quit IRC18:22
sdagueclarkb: that's before grenade touches anything18:23
clarkbsdague: yes18:23
clarkbI should s/grenade/the job/18:23
*** salv-orlando has quit IRC18:23
jeblairsdague, clarkb: so perhaps some kind of intra-rax-iad network issue on the private network18:23
sdaguejeblair: perhaps18:23
jeblairclarkb, sdague: is there a reason we use the private network?18:23
sdagueit's been happening for a long time18:23
sdagueand only this region18:24
clarkbjeblair: yes, public networks traverse NAT sometimes18:24
clarkband NAT is terrible18:24
jeblairclarkb, sdague: nodepool verifies external connectivity to nodes via the public network...18:24
jeblairclarkb: i agree in principle -- however, has that caused us problems?18:24
clarkbjeblair: I believe they did, we intentionally switched at one point18:24
jeblairclarkb: okay.  i ask because clearly the internal network is not particularly reliable.18:25
clarkbyup I agree, not disputing that18:25
clarkband in this region using the public ip would be fine18:25
jeblairclarkb: was it on hpcloud or rax that the public network was causing us problems due to nat?18:25
clarkbwould have been hpcloud18:25
clarkbsince rax doesn't NAT aiui18:26
jeblairclarkb: was the nat affecting the tunnel that's built?18:26
knikollaclarkb: Thanks for the help. I’m able to login on review.openstack.org with my launchpad account and that pointed me to the ‘knikolla’ user.18:26
*** pradk has joined #openstack-infra18:26
*** pradk has quit IRC18:26
sdagueanyway, if we could disable that region for now for this set of jobs, it would help reduce the failure rate, and provide time to explore options18:26
clarkbjeblair: I0c8cf8f5627d1c00f278827ab0cee708f7741e89 thats the change, apparently it was for debugging purposes18:27
jeblairclarkb: and were the nat problems consistent or intermittent?18:27
*** unicell has joined #openstack-infra18:27
openstackgerritMerged openstack-infra/system-config: Don't rsync puppet on localhost  https://review.openstack.org/26973618:28
knikollaclarkb: And through git review -s also. All looks good.18:28
clarkbknikolla: excellent let us know if you run into any problems18:28
jeblairclarkb: i'm having trouble mapping that change onto a world where we have providers with only public ips18:29
*** dizquierdo has quit IRC18:29
*** baoli has quit IRC18:29
*** knikolla has quit IRC18:29
clarkbjeblair: if there is no explicit private IP nodepool writes the public IPs to the private IP list18:29
jeblairclarkb: so that change is just a noop in that case?18:29
jeblairclarkb: that makes its logic somewhat less compelling in my mind.18:30
*** trown is now known as trown|lunch18:30
*** fhubik has quit IRC18:30
*** rossella_s has quit IRC18:30
clarkbI am still spooling up stuff from a year ago to figure out what was done18:31
jeblairclarkb: *nod*18:31
clarkbthere was definitely a thing with live migrations18:31
*** rossella_s has joined #openstack-infra18:31
jeblairsdague: do live migrations work on ovh?18:31
clarkband unfortunately we don't have the parent child relationship in gerrit anymore so I can't easily look at the related changes18:32
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: strip _ prefix from independent series  https://review.openstack.org/26980218:32
mordredjeblair, fungi: ok - the current puppet run looks like it's running puppet18:32
sdaguejeblair: I do not know18:32
mordredjeblair, fungi: it has run puppet on the git nodes so far18:32
fungiclarkb: jeblair: sdague: if memory serves, there are multiple rfc-1918 backend nets in some rax regions and to be able to communicate between them you may need additional routes on eth118:32
lucasagomeshi all, quick question... there's anyway to disable that CTRL+F keybind from gerrit? I looked at the preferences but couldn't find a way to disable it18:33
jeblairfungi, clarkb: this is only making me more interested in only using public ips18:33
lucasagomes(I probably will have to do it at browser level)18:33
mordredjeblair: ++18:33
*** igorbelikov has quit IRC18:33
*** zhiyan has quit IRC18:33
*** maximov has quit IRC18:33
*** etoews has quit IRC18:33
*** edwarnicke has quit IRC18:33
*** odyssey4me has quit IRC18:33
lucasagomesI just dislike the fact that gerrit is overwriting existing browser functionality I use all the time18:33
jeblairclarkb: I3afa113f731687be7bacf924e7f26a0635f95d46 may be relevant18:34
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API  https://review.openstack.org/22196418:34
*** tpeoples has quit IRC18:34
*** kozhukalov has quit IRC18:34
mordredjeblair, fungi: and the nodes have begun reporting to puppetdb as expected18:34
*** pvaneck has joined #openstack-infra18:34
jeblairmordred: yay!  AJaeger ^18:34
*** woodster_ has quit IRC18:34
*** rguillebert has quit IRC18:34
*** jraim has quit IRC18:34
*** auggy has quit IRC18:34
*** Ng has quit IRC18:34
*** kozhukalov has joined #openstack-infra18:34
*** igorbelikov has joined #openstack-infra18:34
*** mwhahaha_ has joined #openstack-infra18:34
*** Ng_ has joined #openstack-infra18:34
fungimordred: thanks!18:34
AJaegeryah, thanks!18:34
*** Guest95435 has quit IRC18:34
nibalizermordred: woot18:34
sdaguejeblair: looks like yes - http://logs.openstack.org/15/255515/4/check/gate-tempest-dsvm-multinode-full/8db66f3/console.html#_2016-01-19_09_59_36_45718:35
clarkbfuzzy memory said it had to do with libvirt not know what the public IPs are in a NAT/floating ip siutation18:35
*** othermordred has quit IRC18:35
*** mwhahaha has quit IRC18:35
*** rackertom has quit IRC18:35
clarkbso live migration would fail18:35
jeblairsdague: cool18:35
openstackgerritMerged openstack-infra/tripleo-ci: Remove flavor creation  https://review.openstack.org/25155318:35
*** mwhahaha_ is now known as mwhahaha18:35
jeblairclarkb: aha18:35
clarkbits the whole "host doesn't know its actual IP problem" iirc18:35
*** etoews has joined #openstack-infra18:35
jeblairmordred: ^ wow18:35
*** dham1 has quit IRC18:35
*** teran has quit IRC18:35
*** mhayden has quit IRC18:35
jeblairmordred: it even affects openstack :)18:35
clarkbbut I need to confirm that still18:35
jeblairmordred: but still, somehow, it's "not a problem"18:35
sdagueclarkb: that would be true of floating ips18:36
*** hockeynut has quit IRC18:36
*** odyssey4me has joined #openstack-infra18:36
clarkbjeblair: that said it is easy enough to push a patch up to use only public IPs18:36
clarkband see what happens18:36
sdaguebut if there is some public network that's not floating ips, then I'm not sure it's an issue18:36
jeblairclarkb: i am in favor of taht18:36
clarkbthen recheck until you run somewhere with nat18:36
*** tpeoples has joined #openstack-infra18:36
*** zhiyan has joined #openstack-infra18:36
*** maximov has joined #openstack-infra18:36
*** hockeynut has joined #openstack-infra18:36
*** bastelfreak has quit IRC18:37
*** othermordred has joined #openstack-infra18:37
*** jraim has joined #openstack-infra18:37
*** rguillebert has joined #openstack-infra18:37
*** dham1 has joined #openstack-infra18:37
*** teran has joined #openstack-infra18:37
*** auggy has joined #openstack-infra18:37
*** woodster_ has joined #openstack-infra18:37
clarkbI will write that change18:37
jeblairclarkb, mordred: aiui, bluebox will soon be the only provider we use that has nat18:37
mordredjeblair: yup18:38
*** annegentle has joined #openstack-infra18:38
*** bastelfreak has joined #openstack-infra18:38
*** mhayden has joined #openstack-infra18:38
*** angdraug has quit IRC18:38
sdaguecan we do https://review.openstack.org/#/c/269799/ in parallel, as that would help relieve the 14 hr backlog in the gate18:38
mordredso - should we perhaps treat the problem as "we should use the network to which the vm is actually attached" ?18:38
*** rackertom has joined #openstack-infra18:38
armaxregXboi: ^18:39
*** edwarnicke has joined #openstack-infra18:39
mordredwhich is to say - on bluebox, since we are natted, we know that the network to which the vm is attached, the 'private' network, is a _real_ network that communication between vms can happen on18:39
openstackgerritClark Boylan proposed openstack-infra/devstack-gate: Use public IPs for all comms  https://review.openstack.org/26980718:39
mordredit is possible to know to use this from the metadata18:39
openstackgerritJames E. Blair proposed openstack-infra/system-config: Require non-NAT from providers  https://review.openstack.org/26980818:40
armaxregXboi: it looks like bug 1535850 can be marked as duplicate18:40
openstackbug 1535850 in neutron "multinode check pipeline failing on Permission denied for scp" [Undecided,New] https://launchpad.net/bugs/153585018:40
jeblairmordred: for discussion, an alternate approach ^18:40
mordredI mean - I hate NAT as much as the next guy18:40
regXboiarmax: of which?18:40
mordredbut I believe we can fix this by considering the metainfo we have currently in a different light18:40
*** ybathia has joined #openstack-infra18:40
*** salv-orlando has joined #openstack-infra18:41
jeblairmordred: but should we have to?  it's about to become a minority configuration.18:41
*** shashank_hegde has joined #openstack-infra18:41
armaxregXboi:  bug 153118718:41
openstackbug 1531187 in OpenStack-Gate "subnode fails to be accessable over ssh" [Undecided,New] https://launchpad.net/bugs/153118718:41
mordredit's about to become a minority configuration in our current set of providers18:41
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds fairy-slipper build jobs for api-site repo  https://review.openstack.org/26980918:41
* regXboi reads18:41
mordredbut it's still a normal enough configuratoin, much as I hate that it is18:41
mordredthat I tihnk us standing our ground and demanding that a cloud behave in a certian way does not fit with our stance up to this point which is "we will work around brokenness exhibited by clouds"18:42
jeblairmordred: i'm not sure we're in a position to demand anything of clouds :)18:42
mordredalso, having talked with folks, the chances of us getting a direct public network in bluebox in the near term is none18:42
regXboiarmax: ok we'll mark it as a dup for now and unmark it if this all doesn't fix things18:43
zaroanteaya: what's this deal with the sandbox repo?18:43
armaxregXboi: I already did18:43
*** jsavak has quit IRC18:44
* regXboi wonders if the ansible issues might be pegged to the same thing18:44
clarkbregXboi: ansible issues?18:44
clarkboh you mean the thing we are discussing, yes thats how we found it :)18:44
*** jsavak has joined #openstack-infra18:44
*** sridhar_ram has quit IRC18:44
*** ari is now known as _ari_18:45
regXboiclarkb: I was seeing two different error signatures in the multinode job - one was ansible failing on the primary and the other was scp failing - I am now thinking the two have the same source18:45
regXboier *in the multinode job I was looking at"18:45
clarkbjeblair: it is worth pointing out that ssh seems to work sometimes18:45
clarkbjeblair: so I don't think it is a route issue, but it is definitely unreliable18:46
clarkbyou can see it failed on your link but it ran on mine and failed to find the file18:46
clarkball in the same job18:46
jeblairclarkb: oh, that casts doubt on fungi's theory18:46
fungiyep, agreed18:46
fungiso this could be "backend network in iad is unreliable"18:47
clarkbor possibly even "network" in iad is unreliable18:47
jeblairmordred: okay, my concern is the amount of engineering that might go into what, for us, is a minority configuration18:47
fungior possibly even "iad is unreliable"18:47
clarkbwe just don't typically notice because jenkins and ndepool delete nodes with problems if it can find them18:47
clarkbbut once they are attached they may be fine18:47
mordredjeblair: yah - I hear that - I think it's an almost trivial amount of work though18:48
jeblairmordred: but i can step off my soapbox for a moment and actually consider what that engineering should be :)18:48
mordredjeblair: so I will make a POC script that shows that18:48
AJaegertdurakov: try again, zuul has been reconfigured and most jenkins are up to date18:48
jeblairmordred: okay cool18:48
clarkbwe can collect host uuids and provide them back to rax and see if they can figure it out18:48
*** ashishb has quit IRC18:49
*** pradk has joined #openstack-infra18:49
*** ashishb has joined #openstack-infra18:49
mordredthe logic goes something a little like this: "do you have a direct-attached public network? yes? use it. do you have a floating ip and only one direct attached network? use the direct attached. do you have a floating ip and more than one direct attached network - fail, because that's an unknown combination"18:49
anteayazaro: https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access18:49
anteayazaro: https://review.openstack.org/#/c/269387/18:49
mordredjeblair: does that logic seem like it makes sense to you?18:49
anteayazaro: gerrit says sandbox gives +2 to all registered users18:50
anteayazaro: except it doesn't18:50
jeblairmordred: yes, and i think you can implement it by checking whether the nodepool reported public ip is on a local interface18:50
*** Ramanjaneya has quit IRC18:50
mordredjeblair: yes indeed18:50
jeblairmordred, clarkb: however, is it also potentially possible to solve this by configuration -- like "configure libvirt to know that its ip address is the nodepool supplied public ip, regardless of whether it appears locally?"18:51
mordredjeblair: in fact, if we do not care about being comprehensive - we could simply make it "is the public ip on your local interface? use it. else? use private"18:51
jeblairmordred, clarkb: that's essentially how you solve the problem with afs -- it's "i know you can't figure out your public ip because of nat, so here it is"18:51
jeblairmordred: yeah, i think that works18:52
fungianteaya: zaro: i wonder if that's inheriting the -1..+1 range got registered users from the all-projects acl and so not getting overridden in the sandbox?18:52
mordredjeblair: that seems like a thing we can do entirely in the setup for this without any nodepool or ready script changes18:52
mordredsince it's logic that works with the informatoin that is already written to the node anyway18:53
tdurakovAJaeger, works as expected, thank you18:53
anteayafungi zaro oh perhaps that is it18:53
anteayaI wonder how we address that if that is what is happening18:53
andreafclarkb: if you have a sec, I have another small puppet change on log_processor (add if ! defined to deps) https://review.openstack.org/#/c/269136/18:53
*** boris-42 has quit IRC18:53
fungianteaya: probably need to compare the ref pattern for that label in all-projects and see if maybe it's more specific than the one in the sandbox acl18:54
fungiif so, that would account for what we're seeing18:54
*** salv-orlando has quit IRC18:54
anteayafungi zaro I will see if I can figure out how to do that (if I am able with my level of permissions)18:54
fungianteaya: you can either look at it via gitweb, or just rely on what's in the gerrit configuration section of our system-config documentation (which should be accurate for that at least)18:55
*** sdake_ has joined #openstack-infra18:55
anteayaI'll start there18:55
mordredfungi: we're down to 10 unreported nodes18:55
mordredfungi: so I'll declare ansible/puppet fixed again18:56
* anteaya still doesn't know what gitweb is, she thinks it is the gerrit ui18:56
pleia2anteaya: gitweb is the web interface that actually comes with git18:56
fungianteaya: there's a gitweb interface served from review.openstack.org, and it's linked from the acl view pages in gerrit too18:56
pleia2anteaya: it's not very beautiful, so everyone uses something else (like we use cgit)18:56
pleia2for user interactions18:56
*** arxcruz has joined #openstack-infra18:57
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds release notes jobs for fairy-slipper, app dev doc tool  https://review.openstack.org/26981318:57
anteayapleia2: ah thank you18:57
anteayaI found this: https://review.openstack.org/gitweb?p=openstack-dev/sandbox.git;a=history;hb=refs/meta/config;f=project.config18:57
fungihowever the gitweb on review.o.o has the up side that it serves repos/refs/branches that gerrit doesn't push to our mirrors (for example, the all-projects acl)18:57
anteayafungi: thank you18:57
*** sdake has quit IRC18:57
anteayafungi: ah ha, thank you18:58
fungiwhich is why our cgit doesn't have that18:58
mordredpleia2: at some point in the past ...18:58
clarkbjeblair: https://review.openstack.org/#/c/135768/ is the change I was hoping would shed more light on this but our logs hvae rolled off18:58
mordredfungi answered the question I was about to ask18:58
AJaegertdurakov: glad to hear!18:58
mordredwhich is why we didn't just use the cgit18:58
pleia2yeah, fungi had usefulness/context data :)18:59
*** acruz has quit IRC18:59
fungimordred: well, i'd still not be entirely opposed to switching gerrit's links to point to our cgit instead, even if we can't get it to still serve the all-projects acl18:59
openstackgerritAndreas Jaeger proposed openstack-dev/hacking: Just a test...  https://review.openstack.org/26981419:00
mordredfungi: I think it would be niceish19:00
fungimeeting time... join us in #openstack-meeting19:00
openstackgerritSergey Lukjanov proposed openstack-infra/project-config: Rename sahara-scenario -> sahara-tests  https://review.openstack.org/26981519:00
*** tiswanso has quit IRC19:01
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Run instack-test-overcloud  https://review.openstack.org/26981619:01
*** tiswanso has joined #openstack-infra19:02
zaromordred: review change 24949219:02
*** jsavak has quit IRC19:03
*** alex_xu has quit IRC19:04
*** salv-orlando has joined #openstack-infra19:04
*** sabeen has joined #openstack-infra19:04
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949219:05
*** acruz has joined #openstack-infra19:05
*** arxcruz has quit IRC19:06
*** sabeen2 has joined #openstack-infra19:06
*** zeih has quit IRC19:06
*** alex_xu has joined #openstack-infra19:07
*** jsavak has joined #openstack-infra19:08
*** sabeen has quit IRC19:09
*** eliqiao has quit IRC19:11
*** eliqiao has joined #openstack-infra19:13
*** jsavak has quit IRC19:14
*** tdurakov has quit IRC19:15
*** tdurakov has joined #openstack-infra19:16
*** angdraug has joined #openstack-infra19:17
*** acruz has quit IRC19:17
*** ihrachys has quit IRC19:18
*** jcoufal has quit IRC19:19
*** sridhar_ram has joined #openstack-infra19:23
openstackgerritAnne Gentle proposed openstack-infra/project-config: Adds release notes jobs for fairy-slipper, app dev doc tool  https://review.openstack.org/26981319:23
*** erikwilson has joined #openstack-infra19:24
*** jaosorior has quit IRC19:25
*** jaosorior has joined #openstack-infra19:25
greghaynesclarkb: did nodepool get a redeploy today?19:25
clarkbgreghaynes: not yet19:25
clarkbgreghaynes: I think there is one or two changes we are waiting on19:26
* greghaynes looks19:26
greghayneshas 2x+219:26
greghaynesalso https://review.openstack.org/#/c/269266/219:30
*** trown|lunch is now known as trown19:31
*** dkranz has quit IRC19:32
*** Ng_ is now known as Ng19:32
*** dkranz has joined #openstack-infra19:33
*** _nadya_ has joined #openstack-infra19:33
*** piet has quit IRC19:33
*** sridhar_ram has quit IRC19:34
*** sridhar_ram1 has joined #openstack-infra19:34
*** jsavak has joined #openstack-infra19:35
openstackgerritDoug Hellmann proposed openstack-infra/project-config: publish from openstack/releases to releases.openstack.org  https://review.openstack.org/26651519:39
*** fitoduarte has joined #openstack-infra19:39
*** jsavak has quit IRC19:40
anteayadhellmann: you can just remove the whole target: line19:42
*** jsavak has joined #openstack-infra19:42
anteayadhellmann: though setting the value to empty string will accomplish the same thing19:42
anteayatarget: isn't a parameter that is required19:43
dhellmannanteaya : ah, sure19:43
openstackgerritDoug Hellmann proposed openstack-infra/project-config: publish from openstack/releases to releases.openstack.org  https://review.openstack.org/26651519:43
*** jsavak has quit IRC19:43
*** acruz has joined #openstack-infra19:43
anteayadhellmann: so pretty19:44
*** otter768 has joined #openstack-infra19:45
AJaegerdhellmann: publishing to static.openstack.org is done differently than publishing to docs.o.o19:46
dhellmannAJaeger : did I edit the wrong file?19:47
*** dslev has joined #openstack-infra19:47
dhellmannoops, wrong patch, changing context...19:47
zaropabelanger: not seeing apache::vhost::custom change merged. were you guys talking about a different change?19:47
dhellmannAJaeger : ok, I'm not sure what  your comment is related to?19:47
AJaegerdhellmann: 266515, see my inline comment19:48
greghaynesclarkb: oh hey that requirements change merged for dib19:48
AJaegerdhellmann: I suggest you mark your changes as WIP until the spec is approved.19:48
*** gyee has quit IRC19:49
AJaegeror make it dependent on the spec ;)19:49
dhellmannAJaeger : ok, I'll take a look at it again tomorrow (meeting time starting soon)19:49
*** rockyg has joined #openstack-infra19:49
*** santoshk has joined #openstack-infra19:49
*** krtaylor has quit IRC19:49
*** otter768 has quit IRC19:50
*** Swanson has joined #openstack-infra19:50
dhellmannAJaeger : thanks!19:50
anteayaSwanson: what is the situation please?19:50
SwansonI'm currently spamming the universe.  For a CI I want to chop down the jobs section of zuul's layout.yaml file to just what I care about, right?19:50
anteayawhat do you mean?19:50
anteayaplease explain the details of spamming the universe19:51
SwansonI'm working on a new CI.  I just got zuul up and running and I seem to be attaching myself to every possible review.19:51
anteayawhat is the name of your ci system?19:51
AJaegerstevemar: there are 5 or so keystone patches that were started before the linters target was in and they need a recheck - you just rechecked the first one. They're still building... Everything newer than those is fine.19:51
SwansonDELL SC CI19:51
anteayaSwanson: have you tried shutting it off?19:51
anteayaif you haven't tried shutting it off, shut if off19:52
Swansonanteaya, In theory it is off.19:52
*** eharney has quit IRC19:52
*** piet has joined #openstack-infra19:52
anteayadoes it have any running processes?19:52
pabelangerzaro: puppetlabs-apache19:53
dougwigSwanson: do you have a link to your zuul layout.yaml file?  it's normal for it to *see* every review.19:53
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463919:53
dougwigSwanson: and a good starting point is here: https://github.com/openstack-infra/project-config-example19:53
pabelangerzaro: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost/custom.pp19:53
greghaynesSpamapS: ^ I have some confidence that dib boot test patch will pass this time19:53
SpamapSgreghaynes: yay19:53
greghaynesSpamapS: FYI since I really want that to merge after our bootloader breakage19:53
Swansondougwig, It is probably the default.  It didn't want to work and I followed someone elses suggestion and I think I got the default again.19:54
anteayaSwanson: I see that your ci system has reviewed 8 patches: https://review.openstack.org/#/q/reviewer:%22DELL+SC+CI+%253Copenstack-cinder-eql-ci%2540dell.com%253E%2219:54
*** jsavak has joined #openstack-infra19:54
*** kzaitsev_mb has joined #openstack-infra19:54
*** salv-orlando has quit IRC19:54
anteayadougwig: thank you19:54
dougwigSwanson: can you pastebin  your layout file here?19:54
krotscheckAFS-supportive system-config patches for wheel mirror are up for review. https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open19:55
*** ashishb has quit IRC19:56
*** PsionTheory has joined #openstack-infra19:56
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Just let shade and os-client-config handle clouds  https://review.openstack.org/22675819:56
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions  https://review.openstack.org/22675119:56
*** rfolco has quit IRC19:56
AJaegermordred: one shade to rule them all?19:57
Swansondougwig, anteaya It looks like it is the default. Let me see about pasting it.19:57
*** sdake_ is now known as sdaake19:57
*** sdaake is now known as sdake19:57
anteayaSwanson: paste.openstack.org is suggested unless you like something else19:57
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Log task name more succinctly  https://review.openstack.org/24765819:57
Swansondougwig, the entire universe is in it ...  http://paste.openstack.org/show/484330/19:57
mordredAJaeger: :)19:58
dougwigSwanson: yegads, turn off your zuul right now.19:58
Swansonanteaya, dougwig the pipelines don't matter, right?  It is the jobs, right?19:58
dougwigSwanson: start with this one, https://github.com/openstack-infra/project-config-example/blob/master/zuul/layout.yaml19:58
dougwigSwanson: and built it up.19:58
Swansondougwig, Zuul should be off.  That layout looks way closer to what I started with.19:59
*** aysyd has quit IRC19:59
dougwigSwanson: did you really intend to define that many jobs across that many projects? that is the beefiest 3rd party ci that i've ever seen. what are you trying to test?20:00
*** megm has quit IRC20:00
*** jsavak has quit IRC20:00
Clintjeblair: https://review.openstack.org/#/c/259705/ stole my lunch money20:00
Swansondougwig, I didn't.  I was debugging why zuul didn't start via this page ...  https://etherpad.openstack.org/p/thirdpartyci20:01
Swansondougwig, ended up here https://www.irccloud.com/pastebin/00tjSkvr/20:01
Swansondougwig, and while it DID start it sure started an awful lot of things.20:01
jeblairClint: that sounds important20:01
*** jsavak has joined #openstack-infra20:01
jeblairClint: wait, is it good if your lunch money is stolen or bad?20:02
dougwigSwanson: that layout file you posted has a huge number of jobs. i'd focus on the no-op job on ci-sandbox until it's all solid.20:02
Clintjeblair: gertty will get into a state where it keeps sending the same request with n=500&start=500 over and over multiple times a second20:02
*** megm has joined #openstack-infra20:02
*** tonytan4ever has quit IRC20:02
Swansondougwig, That's what I thought.  I wasn't sure if I was supposed to be defining all the jobs there and then limiting them else where or not.20:03
jeblairClint: neat -- can you save a copy of your gertty.db real quick to make sure we don't lose the ability to reproduce?20:03
*** lucasagomes is now known as lucas-pto20:03
dougwigSwanson: zuul tells jenkins what to build. you don't want it doing that for anything but a test job yet, and in the silent queue at that.20:03
*** austin81 has joined #openstack-infra20:03
*** krtaylor has joined #openstack-infra20:03
*** salv-orlando has joined #openstack-infra20:03
Swansondougwig, anteaya thank you for the help.  Will replace my layout.yaml file with a more appropriate one.20:04
anteayaSwanson: perhaps attend some third party meetings?20:04
gtmanfredmordred: Shrews any chance yall can review this one? https://review.openstack.org/#/c/268404/ for shade20:04
anteayaSwanson: maybe get to know some other operators20:04
jeblairClint: then can you run it with debug logging enabled, and send me the full output from startup till when it gets into that state?  it will be huge, but it should have everything i need to debug and should be safe to share from a privacy pov.  maybe a .bz2 via personal email.20:04
Clintjeblair: afraid not, i reverted that change yesterday and kept on using20:04
anteayaavoid the common issues20:04
*** e0ne has quit IRC20:04
Clintjeblair: i do have a debug log with the problem20:05
anteayaSwanson: and stick with the ci-sandbox for a while20:05
zaropabelanger: i was actually curious about the change with WIP set and still merged.  is there such a change?20:05
Swansonanteaya, That would be the plan.  Thanks again.20:06
fungizaro: "WIP" was in the change subject i think20:06
anteayaSwanson: thank you20:06
jeblairClint: ok.  be aware you are probably not getting all the data without that.20:06
anteayaSwanson: thanks for asking for help20:06
anteayaSwanson: perhaps the first thing to work on is confirming when you system is off20:06
fungizaro: yeah, that's what it was https://review.openstack.org/24483520:06
anteayaSwanson: so you can shut it off with confidence when you need to20:07
Swansonanteaya, It took me way too long to realize my email was blowing up or I would have shut it off sooner.20:07
*** keedya has joined #openstack-infra20:07
anteayaSwanson: okay but when I asked you if your system was off, you replied with "in theory"20:08
anteayaSwanson: I need you to have more confidence in your ability to know your system is off20:08
*** jaosorior has quit IRC20:08
zarofungi: ahh, makes sense then. thought it might be a bug in gerrit.20:08
anteayaplease consider that a priority20:08
Swansonanteaya, okay.20:09
*** dprince has quit IRC20:09
*** salv-orlando has quit IRC20:09
fungizaro: nah, just someone overlooking the change subject, which was not in agreement with the workflow votes20:09
clarkbjeblair: https://jenkins04.openstack.org/job/gate-tempest-dsvm-multinode-full/2689/console is failing fwiw with the use public IPs change20:10
*** kgiusti has quit IRC20:10
anteayaSwanson: thank you20:10
clarkbjeblair: and it ran on bluebox with NAT so will have to do some digging into that20:10
*** abitha has joined #openstack-infra20:11
*** ociuhandu has quit IRC20:11
Clintjeblair: log sent, it's seeing that there's more to fetch but it's setting start=500 unconditionally when it should be going to 1000, i believe20:11
AJaegeranteaya: could you review https://review.openstack.org/#/c/269813/ , please?20:12
anteayaAJaeger: in a second20:12
AJaegersure, thanks20:12
anteayafungi zaro so I see sandbox has inherited from all projects: https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access20:12
anteayaso gerrit reports that all registered users have +2 except they don't20:12
anteayaI don't know where else to check or look or change20:13
* anteaya hopes someone else also cares about getting sandbox permissions working20:13
clarkbanteaya: I wsa thinking about that more and it is possible that the all projects acl sets registered users to +/-120:14
anteayaclarkb: yes it does20:14
clarkbanteaya: and our acl there isn't overriding. So to fix it we may attempt making that an exclusive group permission20:14
anteayaokay can we try that on -dev?20:14
clarkbanteaya: or maybe the refs don't match causing the ocnfusion and we can make them match20:14
anteayaoh that's interesting20:15
anteayahow might I check the refs don't match theory?20:15
fungianteaya: zaro: specifically https://review.openstack.org/gitweb?p=All-Projects.git;a=blob;f=project.config;hb=HEAD#l21 is applied to refs/heads/* but so is the equivalent entry on the sandbox acl20:15
*** tonytan4ever has joined #openstack-infra20:15
clarkblooks to be the same ref20:15
*** ldnunes has quit IRC20:15
anteayayes it is20:15
anteayasorry I mis-understood your refs don't match point, clarkb20:16
clarkbso I am going to guess we need thta to be exclusive to override the all-projects and if that works file a bug wit hgerrit saying that acl inheritance should override20:16
anteayayes they both reference refs/heads/*20:16
anteayaclarkb: do we have to create a new group to implement your approach?20:16
clarkbanteaya: no, we just have to update the acl to use exclusive permissions on code-review20:17
anteayathe acl for sandbox?20:17
*** Sukhdev has joined #openstack-infra20:18
anteayaI can take a try at that20:18
*** tiswanso has quit IRC20:18
*** salv-orlando has joined #openstack-infra20:19
clarkbjeblair: looks like metadata server may not be working in that setup20:19
clarkband we can't config drive with mulitnode because config drive isn't migrateable20:19
*** tiswanso has joined #openstack-infra20:19
*** e0ne has joined #openstack-infra20:22
*** claudiub has quit IRC20:22
openstackgerritMerged openstack-infra/project-config: Adds release notes jobs for fairy-slipper, app dev doc tool  https://review.openstack.org/26981320:22
*** e0ne has quit IRC20:22
*** tdurakov has quit IRC20:23
openstackgerritgordon chung proposed openstack-infra/project-config: fix ceilometer integration job to handle slow gate  https://review.openstack.org/26984620:23
*** flwang has joined #openstack-infra20:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Run instack-test-overcloud  https://review.openstack.org/26981620:24
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Deploy with --libvirt-type qemu  https://review.openstack.org/26984820:24
openstackgerritAnita Kuno proposed openstack-infra/project-config: Add exclusiveGroupPermissions to sandbox's acl  https://review.openstack.org/26984920:24
anteayaclarkb fungi zaro ^^20:25
*** MarkAtwood1 has quit IRC20:25
*** kzaitsev_mb has quit IRC20:26
*** binbincong has quit IRC20:27
anteayaclarkb: thanks20:28
pabelangerzaro: this was the WIP patch, https://github.com/openstack-infra/system-config/commit/7910a841977a39a20a9b35889216d6bf8d6450bd20:29
pabelangerzaro: this is the new change using puppetlabs-apache https://review.openstack.org/#/c/205596/20:29
zaropabelanger: thanks, fungi explained it to me. wip was only on title.20:29
*** kzaitsev_mb has joined #openstack-infra20:30
*** binbincong has joined #openstack-infra20:30
*** salv-orlando has quit IRC20:32
*** sabeen2 has quit IRC20:33
*** ZZelle_ has joined #openstack-infra20:34
anteayafungi: thanks20:35
*** electrofelix has quit IRC20:35
*** denisra_ has quit IRC20:36
*** denisra_ has joined #openstack-infra20:36
*** tdurakov has joined #openstack-infra20:36
*** rguillebert has quit IRC20:37
*** dtardivel has quit IRC20:37
*** gildub has joined #openstack-infra20:39
*** xyang1 has joined #openstack-infra20:39
*** kzaitsev_mb has quit IRC20:40
zaroanteaya: docs seems to indicate that it should work.20:41
anteayazaro: what is "it" in your above assertion?20:42
anteayamy change, or the original configuration that doesn't work20:42
*** boris-42 has joined #openstack-infra20:42
openstackgerritMathieu Gagné proposed openstack-infra/project-config: Increase internap max servers from 36 to 72  https://review.openstack.org/26985720:42
anteayamgagne: thank you20:43
mgagneanteaya =)20:43
zaroanteaya: your change, https://gerrit-review.googlesource.com/Documentation/access-control.html20:44
fungizaro: yeah, we're mainly concerned that a project-specific acl is not overriding an equivalent ref section from the all-projects acl20:44
mgagneI find it curious that some jenkins masters are getting more jobs/slaves scheduled to them than others20:44
anteayamgagne: what are you seeing?20:45
fungizaro: i had always assumed that project-specific acls override the inherited all-projects acl when there were equivalent sections defined?20:45
mgagnecurious lines in graphs: http://grafana.openstack.org/dashboard/db/nodepool20:45
*** bardia has quit IRC20:45
*** bardia has joined #openstack-infra20:46
fungimgagne: that indicates that jenkins02 is having issues and is probably not responding to nodepool's attempts to add or remove slaves20:46
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463920:46
anteayamgagne: I'm going with jenkins02 is off doing its own thing for some reason20:46
fungimgagne: likely the thread leak we battle continually in jenkins (at this point we have a cron job which reboots all our jenkins masters once a week just to work around it)20:46
mgagnewe just have to wait for graphs to form a skull shape :D20:46
clarkbfungi: those restarts seem to help but may not be aggressive enough, maybe we should do it every other day20:47
anteayamgagne: ha ha ha20:47
greghaynesianw: SpamapS can I get a +3 on https://review.openstack.org/#/c/268318/120:48
*** nmagnezi has quit IRC20:49
*** moravec has quit IRC20:51
prometheanfiresure gating is alright, been waiting 8 hours, some are up to 1620:51
zarofungi: it's unclear in the doc, but i think it's an addition too what was specified in the parent to narrow the set of groups. although i'm not sure so might be worth testing and making the docs more clear.20:51
clarkbprometheanfire: the infra systems themselves seem to be operating properly20:52
clarkbrunning >1k jobs per hour20:52
clarkbthe jobs themselves are failing at a relatively high rate which some people are addressing20:53
prometheanfireah, k20:53
clarkbprometheanfire: when jobs fial in the gate everything behind that change must be restarted without that change20:53
prometheanfirethat is painful20:54
fungibasically we end up with a serialized testing scenario within a set of projects sharing common integration test jobs20:54
*** MarkAtwood1 has joined #openstack-infra20:54
fungirather than being able to effectively test them in parallel any longer20:54
*** dslev has quit IRC20:55
openstackgerritMerged openstack-infra/project-config: Add exclusiveGroupPermissions to sandbox's acl  https://review.openstack.org/26984920:56
*** vgridnev has quit IRC20:56
anteayamadorn lbragstad so we are hoping that 269849 gives sandbox users +2 +A, it will take about 30 minutes to update (I believe)20:58
lbragstadanteaya sweet, thanks for the update!20:58
openstackgerritAndre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack  https://review.openstack.org/26941120:58
anteayalbragstad: thanks for bringing it to our attention20:59
*** daemontool_ has quit IRC20:59
anteayalbragstad: let's see what actually happens in 30 minutes but I am hopeful20:59
*** pc_m has quit IRC21:00
*** annegentle has quit IRC21:00
*** sridhar_ram1 has quit IRC21:01
*** _nadya_ has quit IRC21:01
ianwclarkb: mind if i re +w the nodepool import patches that rebased? ... i'd dearly love to get a build and some logs for f23!21:02
clarkbianw: go for it21:03
*** kgiusti has joined #openstack-infra21:04
*** kgiusti has quit IRC21:06
openstackgerritMerged openstack-infra/project-config: Increase internap max servers from 36 to 72  https://review.openstack.org/26985721:07
*** bpokorny_ has joined #openstack-infra21:07
*** vgridnev has joined #openstack-infra21:08
*** adduarte has joined #openstack-infra21:08
*** dslevin has joined #openstack-infra21:09
openstackgerritIan Wienand proposed openstack-infra/nodepool: Normalize some log lines in builder  https://review.openstack.org/26978421:09
*** kzaitsev_mb has joined #openstack-infra21:09
*** pc_m has joined #openstack-infra21:10
*** bpokorny has quit IRC21:11
*** bpokorny_ has quit IRC21:11
mordredclarkb: turns out that option is only for git format-patch21:11
mordredclarkb: not for git commit21:11
*** fitoduarte has quit IRC21:11
mordredclarkb: so it's totally unuseful21:12
mordredlifeless: ^^21:12
*** bpokorny has joined #openstack-infra21:12
lifelessmordred: yay git.21:13
lifelesswith the experience I had from my last patch I sent there, I'm disinclined to offer one for this21:13
jeblairmordred: that sort of makes a weird sort of kind of sense from the "only add this when you are _sending_ the patch"21:14
jeblairmordred, lifeless, fungi: which does sort of suggest that git-review is a not inappropriate place for a similar change21:14
mordredjeblair: it does21:14
*** sputnik13 has quit IRC21:14
mordredbut yeah - I think git review21:14
mordredand I think a git review config option to opt-in to it21:14
mordredso that it has to be a conscious decision21:14
mordredwould be nice21:14
jeblairright; should probably _not_ be a .gitreview file21:14
*** tdurakov has quit IRC21:15
mordredthat way if you wanted to do git config --global gitreview.signoff - so be it - that's your choice21:15
mordredor you could do --local21:15
zarofungi: i think gerritlib release is safe.  i don't see any changes in functionality, all changes are additive.21:15
jeblairthose both sound good to me.  and it might be worth a code comment about why that particular option should never be handled in .gitreview.21:15
openstackgerritMerged openstack-infra/nodepool: Revert "Import nodepool.builder late for statsd"  https://review.openstack.org/26926621:15
*** jpr has quit IRC21:15
anteayaoh general infra announcement, magnum sprint is Feb 18-19, they announced in their meeting today, location to be determined21:16
openstackgerritMerged openstack-infra/nodepool: Remove workaround for early paramiko import  https://review.openstack.org/26930321:16
anteayaI was hoping someone from infra could attend as they seem to really have difficulty with their testing21:16
anteayahow ever I can't make those dates21:16
*** amrith is now known as _amrith_21:16
anteayaas I'll be in the uk21:16
jeblairttx, dhellmann: https://review.openstack.org/269862 sorry for the delay21:16
anteayaoh sorry location is sunnyvale21:16
*** sdake has quit IRC21:16
anteayahoping someone else feels like attending and perhaps offering some testing setup guidance21:17
*** gildub has quit IRC21:17
*** tdurakov has joined #openstack-infra21:17
*** gyee has joined #openstack-infra21:17
*** tdurakov has quit IRC21:17
*** eharney has joined #openstack-infra21:18
fungizaro: thanks, i'll try to work a gerritlib tag in before or between upcoming server reboots21:18
mordredanteaya: I also cannot attend21:18
anteayamordred: thanks for considering it21:18
anteayaor at least reading what I wrote21:18
*** vgridnev has quit IRC21:19
*** vgridnev has joined #openstack-infra21:19
*** jtomasek has quit IRC21:20
*** vgridnev has quit IRC21:20
pabelangerjeblair: care to review: https://review.openstack.org/#/c/264994/ templating support for nodepool rackspace21:20
openstackgerritKhai Do proposed openstack-infra/nodepool: Remove duplicate python-jenkins code from nodepool  https://review.openstack.org/25915721:21
*** tdurakov has joined #openstack-infra21:23
*** e0ne has joined #openstack-infra21:23
*** Sukhdev has quit IRC21:25
*** moravec has joined #openstack-infra21:26
*** rguillebert has joined #openstack-infra21:26
openstackgerritMerged openstack-infra/project-config: don't do multinode jobs in rax-iad  https://review.openstack.org/26979921:27
anteayamy goodness it is going to be warm in bristol next week21:27
*** sputnik13 has joined #openstack-infra21:28
ianwclarkb / greghaynes: do we need https://review.openstack.org/#/c/268867/3 (Handle dib build failures) before we try doing a restart?21:28
greghaynesianw: it would be nice to have21:29
greghaynesit is a definite bug, it does still work without it though21:29
*** mriedem has quit IRC21:29
*** moravec1 has joined #openstack-infra21:29
ianwgreghaynes: so then also https://review.openstack.org/#/c/269774 ?21:29
SpamapSwith the new HORRIBLE gerrit UI, how can I see previous CI results?21:29
SpamapSI don't have a toggle CI button anymore21:30
fungiSpamapS: i don't think that's changed? just click the toggle ci button at the bottom of the page and look in the review comments21:30
mtreinishSpamapS: it's there, in the bottom left of the page21:30
greghaynesSpamapS: there is a patch sets menu on the top right21:30
SpamapSoh its just well hidden21:30
greghaynesalso the toggle ci should still be there21:30
SpamapSVERY well hidden21:30
fungiSpamapS: was it cowering in your blind spot?21:31
SpamapSfungi: yeah, used to be a prominent feature21:31
SpamapSnow it's as tucked away as possible.21:31
*** gordc has quit IRC21:31
SpamapSI'm probably not giving the UI enough of a chance though21:31
SpamapStoo busy puking on it21:31
fungipainted pink with an improbability field erected around it?21:31
fungiimprobability field21:31
fungisuch a useful tool21:31
ianwgreghaynes: ok, not familiar enough with 268867 to merge but maybe clarkb can be convinced21:31
greghaynesheh, I think most folks are not entirely fond of the new UI but not a whole lot we can do :(21:31
greghaynesaside from use gertty21:32
SpamapSOh and I missed ianw's already +A'd that21:32
SpamapSso meh21:32
fungior join the effort to write the replacement webui21:32
SpamapSgreghaynes: I use gertty21:32
*** moravec has quit IRC21:33
SpamapSI used to casually sometimes use the web UI21:33
SpamapSuntil this upgrade21:33
SpamapSfungi: you definitely _do not_ want me writing UI's21:33
greghaynesya, working on the replacement UI has been too low on my todo list for way too long21:33
SpamapSI mean, unless you want code review to look a lot like Zork21:33
fungialso the t key in gertty's default keymap will toggle ci comments21:33
fungiSpamapS: surprisingly, i do, yes!21:33
*** e0ne has quit IRC21:33
SpamapSYou are in a python file, it has 6000 lines. You see a diff at line 4509.21:33
greghaynesturn around21:34
sdagueSpamapS: I thought toggle ci was hidden the same way as before21:34
SpamapSsdague: I may be just raging on the UI in general and assuming _everything_ was always better before. ;)21:34
SpamapSThis new UI needs to get off my lawn.21:34
* fungi shakes cane vigorously21:34
*** kzaitsev_mb has quit IRC21:35
*** ihrachys has joined #openstack-infra21:35
*** sridhar_ram has joined #openstack-infra21:35
ianwSpamapS: it annoys me when it gets less information dense.  showing all the merged changes by default (three ticks) is really useless info because they're done and dusted.  at least there isn't acres of white-space on either side.  anyway, let's not all whinge in here :)21:36
*** sdake has joined #openstack-infra21:37
openstackgerritMerged openstack/diskimage-builder: Add pkg-map for redhat  https://review.openstack.org/26831821:37
SpamapSNo I did not want this to be a UI whambulance. Just needed some details.21:37
anteayaclarkb fungi zaro lbragstad madorn so this is fun: https://review.openstack.org/#/c/269387/ I have successfully removed any permissions for registered users on sandbox21:37
anteayayay me21:37
*** tiswanso has quit IRC21:38
*** dslevin has quit IRC21:38
*** baoli has joined #openstack-infra21:38
openstackgerritAnita Kuno proposed openstack-infra/project-config: Revert "Add exclusiveGroupPermissions to sandbox's acl"  https://review.openstack.org/26987321:38
*** ksantoshk has joined #openstack-infra21:39
*** baoli_ has joined #openstack-infra21:39
fungianteaya: wow. even as an administrator i can no longer vote on those changes21:39
anteayathe power21:40
anteayaso now we know what not to do21:40
anteayalet me fix the commit message on that21:41
*** acruz has quit IRC21:41
fungilooking at the current acl applied in gerrit for that repo, it seems to have done the things we expected (checked the exclusive checkbox on those permissions)21:42
*** tdurakov has quit IRC21:42
openstackgerritAnita Kuno proposed openstack-infra/project-config: Revert "Add exclusiveGroupPermissions to sandbox's acl"  https://review.openstack.org/26987321:42
*** santoshk has quit IRC21:42
fungibut that has obviously not had the desired effect, and implies there's something odd about the registered users group in general21:42
*** bpokorny has quit IRC21:42
anteayafungi: for some reason it is reading the acl for that file and then just making stuff up21:42
*** baoli has quit IRC21:43
anteayafungi: and thanks for checking that effect, as I had not21:43
fungii see what's going on21:43
anteayaah good21:43
*** bpokorny has joined #openstack-infra21:43
anteayawhat is that?21:43
anteayawell obviously a group called Registered Users, but how did it get created?21:44
fungithe empty "Registered Users" group manage-projects just created21:44
anteayaoh I bet we have an acl file derp somewhere21:44
anteayawas that created in response to my patch edit?21:44
anteayamy acl file derp21:44
*** dslev has joined #openstack-infra21:45
fungii'm not sure21:45
*** otter768 has joined #openstack-infra21:46
*** ihrachys_ has joined #openstack-infra21:46
fungino, i don't think so, it was created before sahara-scenario-core, the next group number in sequence21:46
anteayaokay so at least it wasn't me, but perhaps its existance is the reason sandbox permissions are... interesting21:46
anteayamorning jhesketh21:46
*** ihrachys has quit IRC21:46
fungicreated by https://review.openstack.org/259373 which merged Mon, 21 Dec 2015 10:01:18 +000021:47
anteayajhesketh: when you have coffeed up take a peruse of the meeting logs21:47
anteayajhesketh: we had some chat about swift logs and want to hear your thoughts21:47
jheskethanteaya: okay, will do, thanks21:47
fungianteaya: zaro: it looks like the first change to an acl after our gerrit 2.11 upgrade caused manage-projects to create an empty Registered Users group21:47
anteayafungi: ah21:48
anteayajhesketh: thank you21:48
*** jsavak has quit IRC21:48
anteayafungi: yeah, cause that acl file looks as I would expect21:48
*** jsavak has joined #openstack-infra21:48
*** kzaitsev_mb has joined #openstack-infra21:49
*** baoli_ has quit IRC21:49
*** baoli has joined #openstack-infra21:49
*** baoli has quit IRC21:50
*** otter768 has quit IRC21:50
clarkbianw: if you review https://review.openstack.org/#/c/269784/2 I can go through and push the whole stack through21:50
*** baoli has joined #openstack-infra21:50
zarovery odd. not sure what's going on.  but definately bug somewhere. either on override or exclusive.  maybe we can test a little on review-dev21:50
clarkbthen I can do a service restart once we get those installed21:50
fungizaro: see my comment about the empty registered users group. it seems that acl is referencing an actual group rather than the global meta-group for registered users21:51
anteayamadorn: hopefully we get https://review.openstack.org/#/c/269873/ merged shortly, then share the common topic for the patches and hopefully someone can approve and merge them21:52
fungiwhich explains all the weirdness we're seeing21:52
zarofungi: yeah, saw that.  not sure what's up with that.21:52
anteayamadorn: you seem to have helped us uncover a bug, thank you21:52
keedyahi there21:52
keedyai have a review request for a new project https://review.openstack.org/#/c/269411/21:52
anteayafungi clarkb can we get 269873 in the process?21:53
keedya Do I need to add reviewers?21:53
*** _ari_ has quit IRC21:53
anteayawith my gratitude21:53
fungizaro: so something is 1. allowing us to create a regular group which shadows a built-in group name, and 2. causing the acl to refer to that new group rather than the builtin of the same name21:53
clarkbfungi: oh!21:53
clarkbI bet we have a bug in jeepyb then21:53
fungiclarkb: entirely possible21:54
anteayaclarkb: thanks21:54
fungiclarkb: could be a behavior change in gerrit, for example no longer reporting the builting groups in a group listing query or something21:54
clarkbfungi: right well it treats them as system groups21:54
clarkbI thought we had addressed that but apparently not21:55
anteayazaro dougwig thank you21:55
fungiwhich then caused manage-projects to think it needed to create it21:55
clarkband since all-projects isn't jeepyb managed it likely uses the correct uuid in its group file so the default +-1 works21:55
*** angdraug has quit IRC21:55
*** mrmartin has joined #openstack-infra21:56
clarkbgreghaynes: want to review https://review.openstack.org/#/c/269774/1 before I approve it?21:56
*** cburgess has quit IRC21:56
fungialso if you view the all-projects acl in gerrit you can see it's using the correct group reference for registered users (it's not a clickable url like the one in the sandbox acl)21:56
greghaynesclarkb: oh, I missed that one21:56
*** cburgess has joined #openstack-infra21:57
*** eil397 has quit IRC21:57
greghaynesclarkb: lgtm21:58
*** salv-orlando has joined #openstack-infra21:58
*** aeng has joined #openstack-infra21:59
openstackgerritMerged openstack-infra/nodepool: Handle dib build failures  https://review.openstack.org/26886721:59
*** aeng has quit IRC22:00
clarkbfungi: anteaya so the quick ninja fix would be to update the groups file but I think jeepyb will overwrite that the next time it runs22:00
*** Guest51217 is now known as med_22:00
*** med_ has quit IRC22:00
*** med_ has joined #openstack-infra22:00
*** peristeri has quit IRC22:00
*** baoli has quit IRC22:01
clarkbmordred: we just need to trick gerrit into accepting format patch output22:01
*** jpr1 has joined #openstack-infra22:01
*** abregman has quit IRC22:01
zarolets just fix jeepyb22:01
anteayafungi: thanks now I see the clickable url, missed that difference before22:01
*** baoli has joined #openstack-infra22:01
anteayaI'm for helping madorn with tonight's class by someone with gerrit admin approving the patches associated with the common-gerrit topic the class is using22:02
anteayaand fixing jeepyb22:02
*** jpr1 has quit IRC22:02
openstackgerritMerged openstack-infra/nodepool: Also handle work exceptions in builders  https://review.openstack.org/26977422:02
*** erikwilson has quit IRC22:03
clarkbzaro: are you volunteering to do that?22:03
anteayamadorn: when you have that common gerrit topic for your patches, do post it22:03
zaroif anteaya can wait for it22:03
anteayaI can wait sure22:03
zarook. i'll take a look..22:04
anteayasince we have a workaround to help madorn if there is a gerrit admin willing to assist22:04
anteayaif we get the gerrit topic22:04
*** sabeen1 has joined #openstack-infra22:05
*** jsavak has quit IRC22:06
*** sabeen3 has joined #openstack-infra22:07
clarkbhttps://review.openstack.org/#/c/269784/ is the last nodepool change that needs review before I can restart it22:09
* jhesketh finishes reading the logs 22:09
jheskethIt was quite a meeting I missed!22:10
*** sabeen1 has quit IRC22:10
*** Sukhdev has joined #openstack-infra22:11
*** ybathia has quit IRC22:12
*** eliqiao has quit IRC22:12
*** baoli has quit IRC22:13
*** eliqiao has joined #openstack-infra22:13
*** baoli has joined #openstack-infra22:14
*** ybathia has joined #openstack-infra22:15
*** sridhar_ram has quit IRC22:15
pleia2mordred: noted inline the fix you need for https://review.openstack.org/#/c/249492/22:15
pleia2mordred: naturally, it's a single character :) wooo sphinx22:16
ianwclarkb: will do, just waiting for the jenkins22:16
clarkbianw: roger22:16
*** gildub has joined #openstack-infra22:17
anteayajhesketh: yes we missed you22:17
*** sridhar_ram has joined #openstack-infra22:17
*** tiswanso has joined #openstack-infra22:17
*** erikwilson has joined #openstack-infra22:18
*** erikwilson has quit IRC22:18
*** tiswanso has quit IRC22:18
anteayafungi pleia2 jhesketh any one feel like helping by giving this a push? I broke sandbox, this fixes it: https://review.openstack.org/#/c/269873/22:18
*** abregman has joined #openstack-infra22:18
*** mrmartin has quit IRC22:18
anteayawell to the extend it worked before that patch was merged22:18
*** tiswanso has joined #openstack-infra22:18
*** ybathia has quit IRC22:19
pleia2anteaya: done22:19
*** thorst has quit IRC22:19
anteayaand I have to offline for dinner and a meeting, will check back after that22:19
anteayapleia2: thank you22:19
anteayajhesketh: I do want to hear your thoughts but can't ask now as I don't have time to properly listen22:19
fungidoubly approved for double the approval22:20
anteayafungi: thank you22:20
fungisorry, my head is mostly in -incident channel for the moment22:20
anteayafungi: I understand22:20
*** sridhar_ram has quit IRC22:20
anteayafungi: I hope to be back by the time you have to restart servers so I can answer the channel but I don't know22:20
*** rcernin has quit IRC22:21
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Run instack-test-overcloud  https://review.openstack.org/26981622:21
fungianteaya: thanks!22:21
*** thorst has joined #openstack-infra22:22
*** sabeen3 has quit IRC22:22
nibalizermordred: hi22:22
nibalizerim around now22:22
nibalizerhow did puppet apply go22:22
*** sabeen has joined #openstack-infra22:22
*** HeOS has joined #openstack-infra22:22
*** Sukhdev has quit IRC22:22
nibalizerfungi: pabelanger jeblair yes we should move to the new puppetlabs-apache, I think we are waiting on them to release though22:23
*** Sukhdev has joined #openstack-infra22:23
*** akshai has quit IRC22:24
openstackgerritMatthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output  https://review.openstack.org/23444722:24
*** trown is now known as trown|outttypeww22:25
*** baoli has quit IRC22:25
*** baoli has joined #openstack-infra22:26
*** thorst has quit IRC22:26
nibalizergreghaynes: SpamapS the SAS vpn is a different vpn right?22:27
*** fitoduarte has joined #openstack-infra22:27
openstackgerritMerged openstack-infra/project-config: Revert "Add exclusiveGroupPermissions to sandbox's acl"  https://review.openstack.org/26987322:27
*** tlian has quit IRC22:27
*** abregman has quit IRC22:28
*** adduarte has quit IRC22:29
*** sdake has quit IRC22:30
*** rossella_s has quit IRC22:31
*** baoli has quit IRC22:31
dmsimardhey so I have a question about zuul-cloner22:31
*** rossella_s has joined #openstack-infra22:31
dmsimardwhen run from within a gate job, will it magically clone and checkout the current review patchset or what ?22:31
clarkbdmsimard: yes, though I think you still have to tell it which projects to do it for22:32
dmsimardclarkb: I love that kind of magic. Thanks.22:32
*** aeng has joined #openstack-infra22:33
openstackgerritMerged openstack-infra/nodepool: Log task name more succinctly  https://review.openstack.org/24765822:33
*** jamielennox|away is now known as jamielennox22:35
*** akshai has joined #openstack-infra22:35
*** baoli has joined #openstack-infra22:36
*** davidlenwell has quit IRC22:37
*** tonytan4ever has quit IRC22:38
*** davidlenwell has joined #openstack-infra22:38
*** tiswanso has quit IRC22:40
*** thorst has joined #openstack-infra22:40
openstackgerritBrant Knudson proposed openstack-infra/project-config: Keystone run py34 unit tests only on master  https://review.openstack.org/26989622:41
*** tiswanso has joined #openstack-infra22:41
*** eil397 has joined #openstack-infra22:42
*** ybathia has joined #openstack-infra22:42
mordrednibalizer: I have not rolled apply live yet22:43
mordrednibalizer: but it works22:43
jeblairkrotscheck: did you get a volunteer to push through the mirror stuff yet?22:43
nibalizermordred: ok cool22:44
*** amitgandhinz has quit IRC22:45
*** thorst has quit IRC22:45
*** bastelfreak is now known as lieblingsrasi22:45
*** lieblingsrasi is now known as bastelfreak22:45
*** baoli_ has joined #openstack-infra22:46
*** dkranz has quit IRC22:46
*** baoli_ has quit IRC22:48
*** baoli_ has joined #openstack-infra22:49
*** baoli has quit IRC22:49
*** austin81 has quit IRC22:52
*** austin81 has joined #openstack-infra22:53
*** mudassirlatif has quit IRC22:53
*** moravec1 has quit IRC22:55
*** moravec has joined #openstack-infra22:57
openstackgerritBoris Pavlovic proposed openstack-infra/project-config: Add lib-forward-testing for OSprofiler  https://review.openstack.org/26990522:57
*** baoli_ has quit IRC23:00
*** baoli has joined #openstack-infra23:01
*** rkukura_ has joined #openstack-infra23:01
*** mudassirlatif has joined #openstack-infra23:01
clarkbianw: https://review.openstack.org/#/c/269784/ has test results23:02
ianwclarkb: ok, just put it in ... that's the last one right?23:02
*** rkukura has quit IRC23:03
*** rkukura_ is now known as rkukura23:03
clarkbianw: yup23:03
clarkbI will likely restart that service as part of kernel patching reboots23:03
*** rockyg has quit IRC23:04
*** sputnik13 has quit IRC23:05
ianwok ... i would really love to see some build logs! :)  and a fedora back in the gate23:06
*** moravec has quit IRC23:06
clarkbianw: soon!23:06
openstackgerritMerged openstack-infra/nodepool: Normalize some log lines in builder  https://review.openstack.org/26978423:06
*** dingyichen has joined #openstack-infra23:06
*** sdake has joined #openstack-infra23:06
*** dslev has quit IRC23:07
fungijeblair: i said i would try to look at it, but that was before i realized we have kernel updates23:08
jeblairfungi: i want to update my bandersnatch thing and was thinking i might be able to get pypi in afs working in conjunction23:09
clarkbianw: any idea when centos7 will receive kernel updates for recent cve?23:09
clarkbianw: best I can tell the latest kernel is from the 6th of januarar23:10
fungijeblair: sounds worth a shot23:10
*** annegentle has joined #openstack-infra23:10
ianwclarkb: hmm, not off the top of my head23:10
jeblairfungi: i expect to work on bandersnatch today and be ready to push on mirrors for real tomorrow23:11
ianwCVE-2016-0728 ?23:11
fungizaro: can puppet agent be reenabled on review-dev or were you testing something there that puppet might revert?23:11
clarkbianw: yes that is the one23:11
fungiianw: yeah, we're working through it in #openstack-infra-incident but have to wait to do the centos servers23:12
*** baoli has quit IRC23:12
jeblairfungi: oh i missed the red-alert :)23:12
*** dimtruck is now known as zz_dimtruck23:12
fungiheh. it's not really an all-hands-on-deck emergency (yet anyway)23:13
fungistill getting package updates applied everywhere23:13
clarkbianw: oh there are reports that selinux may be sufficient to protect us (need to check on that more but we do have selinux enabled)23:13
zarofungi: you can reenable.23:13
*** ihrachys has joined #openstack-infra23:14
*** sputnik13 has joined #openstack-infra23:14
fungizaro: thanks!23:14
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet/spec_helper: run the script from project dir  https://review.openstack.org/26991423:15
*** baoli has joined #openstack-infra23:15
*** regXboi has quit IRC23:16
*** ihrachys_ has quit IRC23:16
ianwclarkb / fungi : so there's the systemtap workaround in https://bugzilla.redhat.com/show_bug.cgi?id=129747523:18
openstackbugzilla.redhat.com bug 1297475 in vulnerability "CVE-2016-0728 kernel: Possible use-after-free vulnerability in keyring facility" [High,New] - Assigned to security-response-team23:18
*** sputnik13 has quit IRC23:19
*** sputnik13 has joined #openstack-infra23:19
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: WIP: Add Packstack integration test gate jobs  https://review.openstack.org/26991823:20
*** ihrachys has quit IRC23:20
fungiinteresting that red hat knew about the bug for more than a week but didn't actually have a new package brewing to put out once disclosed23:21
ianwfungi: i think there must be more too it than i can see, the bug is restricted.  what i can see from internal responses to important customers is basically a "we're working on it" message.  so i guess the answer is "we're working on it" :)23:23
openstackgerritClint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate devstack plugins list  https://review.openstack.org/26992123:23
fungiianw: i figured they were working on it, but thanks for the confirmation!23:24
*** baoli has quit IRC23:25
*** yamamoto_ has joined #openstack-infra23:27
*** thorst has joined #openstack-infra23:28
* ianw strongly considers less worrying about nodepool logs and more eating23:29
*** alex_xu has quit IRC23:30
openstackgerritDavid Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs  https://review.openstack.org/26991823:30
jheskethjeblair, clarkb: so, re the swift logs from this mornings meeting. I thought we discussed this during the last summit and decided we did want to continue. We acknowledged that we're largely reimplementing a posix filesystem but since we were rather close to being there (or at least it seems we're close to having the features devs need) then we should continue23:31
*** alex_xu has joined #openstack-infra23:31
openstackgerritDaniel Wallace proposed openstack-infra/shade: add the ability to get an object back from swift  https://review.openstack.org/26992523:32
*** Swami has quit IRC23:32
fungijhesketh: for me the current worry is how we shore this up against malicious changes subverting the write access for the metadata container. do we have an easy way to recover if something scribbles garbage over top of what's there?23:32
*** Swami has joined #openstack-infra23:32
*** sigmavirus24 is now known as sigmavirus24_awa23:33
jheskethfungi: we could create a script to recalcaulate the metadata in the worst case23:33
clarkbjhesketh: yes I am ok with continuing, but I am apprehensive simply because its been so much work in general (I am still reviewing those changes though)23:33
fungior am i misreading and the workers don't actually get write access to the metadata?23:33
jeblairjhesketh: hrm, i thought it was more along the lines of "we need to write a routine to query swift to get the last modified time" -- the new container/metadata stuff seems much more complicated.  but maybe i'm just mis-remembering.23:33
jheskethfungi: by looking at the object change times23:33
jheskethfungi: the workers get write access, yes23:33
jheskethjeblair: that is discussed as an alternative option in the spec23:34
fungii also think it's worth making sure we evaluate the effort involved and not fall into a sunk costs fallacy. pressing forward may still make the most sense, but i want to be sure it's not primarily because we've already got so much effort invested23:34
jheskeththe spec has been for review for a while, I've been trying to get attention on it to make sure it was heading in the right way and it seemed like this was what we wanted23:34
jeblairjhesketh: yeah, i just thought we could query things a little better than "traverse the entire system"23:34
jeblairjhesketh: i'm really glad for the spec, because i didn't understand what the zuul changes were about until i started reading it23:35
*** watanabe_isao has joined #openstack-infra23:35
openstackgerritJames E. Blair proposed openstack-infra/infra-specs: Add AFS logs spec  https://review.openstack.org/26992823:36
jheskethjeblair: I'm not sure how we'd do it accurately without traversing the entire system.. we could make assumptions and take the first object in a path and set the pseudo folders to that, but I don't like it23:36
*** xyang1 has quit IRC23:37
jeblairjhesketh, clarkb, fungi: it at least made me want to step back and ask 'would i want to do swift logs knowing what i know now?'23:37
jeblairso i wrote what i think the afs alternative looks like in https://review.openstack.org/26992823:37
*** ybathia has quit IRC23:38
jheskethI probably should know this, but what do we currently have that is using afs?23:39
*** austin81 has left #openstack-infra23:39
jeblairand while i think we can do that pretty quickly -- i also know we've been saying 'afs' a lot but not doing 'afs' a lot, so maybe we should pause this for a week, see if we can get pypi afs mirrors going, and then go from there23:39
*** doug-fish has quit IRC23:39
jeblairjhesketh: nothing seriously yet, i think they have all been lower-than-back-burner priority projects so far23:40
clarkbI am not sure that afs is the right answer fwiw either23:40
clarkbwhen I tested it the performance was really abd23:40
clarkbthat could've been my local connection or poor caching settings though23:41
jeblairit's not exactly iscsi, no23:41
*** thorst has quit IRC23:41
clarkb(also NAT)23:41
jheskethso the original problem of disk space seems to have been less of an issue anyway right? ie throwing disks and lowing retention has largely solved the problem23:41
*** thorst has joined #openstack-infra23:42
jheskethmaybe we're just wasting effort altogether here and we should focus on more important things23:42
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949223:42
clarkbjhesketh: I think it has largely solved it if we are ok with 4 momths of logs23:42
clarkbI don't know that we are actually happy with that number23:42
jeblairclarkb: but i do think it maps well to both log serving and pypi mirroring -- both can benefit from caching23:42
jeblairclarkb: i would like us to be able to get back to 6; i don't think i'd push for more than that.23:42
jeblairbut yeah, right now we are 'only' using 6TB.23:43
jheskethclarkb: right, but what number of months are we happy with and how much more disk do we need for that23:43
jheskethI haven't heard anybody complain about 4 months fwiw23:43
jeblairi think we might even be able to bump back up to 6months now.23:43
clarkbjhesketh: I ran into it today (granted it was 10 months I needed)23:43
clarkband was easy enough to just make change and get new logs :)23:44
jheskethclarkb: sure it'll happen.. just thinking through our options23:44
*** aeng has quit IRC23:44
*** sflanigan has quit IRC23:44
*** dingyichen has quit IRC23:44
jheskethclarkb: right, there's an argument for how useful are the 10month old logs really when you can re-run jobs23:44
jheskethie even if you have the 10month old logs you may still be better re-running23:44
clarkbyes especailly if you want ot address an issue today23:44
clarkbsince deps change and all that23:44
jeblairjhesketh: the older the job, the less likely you'll be able to re-run it :)23:44
jheskethsure, but the problem may have shifted23:45
jheskethI do agree, there is a use for older logs, but exactly how useful I'm less sure on23:45
*** mudassirlatif has quit IRC23:45
*** thorst has quit IRC23:46
jeblairprobably not enough to justify an extra 8TB of space for 1 year retention23:46
*** otter768 has joined #openstack-infra23:47
clarkbjeblair: with afs and mirrors I expect that the http server can also act as a cache and keep things very warm. With logs I worry more because we have very large directories and tend to not access them in patterns that work well for caching23:47
jeblairclarkb: there may be a caching benefit for 'recently created logs'  the old ones will certainly have to be pulled in over the network23:47
mordredthat, of course, is similar to rolling things into colder-storage, fwiw23:48
jeblair(in that people are probably more likely to visit and re-visit recent logs)23:48
*** piet has quit IRC23:49
clarkbwe are still talking almost half a TB of logs per week though23:49
*** doug-fish has joined #openstack-infra23:49
clarkband afs seems to cache on the order of half a gigabyte23:49
jeblairclarkb: .375TB ?23:50
mordredclarkb: afs caches exactly how much you tell it to23:50
clarkbjeblair: ya23:50
mordredclarkb: it's a config setting23:50
clarkbjeblair: with probably some jitter for things that get compressed on the backend23:50
jeblairright, so we can configure like a 50G cache or something if we want23:50
mordredclarkb: in /etc/openafs/cacheinfo ...23:50
mordredclarkb: my laptop has /afs:/var/cache/openafs:500000 set23:50
clarkbmordred: and that doesn't ramp up the cpu/io requirements for the cluster to expire things properly?23:51
mordredif we had web server machines set up to server AFS - I would expect we'd want to dedicate pretty much their entire disk to AFS cache23:51
*** Qiming has joined #openstack-infra23:51
clarkb(I am not sure what is involved in pruning the cache)23:51
*** otter768 has quit IRC23:51
mordredjeblair: ^^ do you know the answer to that? my protocol memory is fading with the lovely Nikka Coffey Grain I just poured23:52
jheskethso coming back to swift... what metrics should we use to determine if the work should stop23:53
fungiwell, also the current logs.o.o giant lvm o' doom has reliability drawbacks23:53
mordredfungi: next year for halloween ...23:53
jheskethperhaps we first need to decide on how many logs we want to store (ie retention)23:53
mordredfungi: PLEASE go as a giant lvm o' doom23:53
fungiwe basically have 14x the chance of the filesystem getting marked offline because we're aggregating a bunch of block devices, each of which is a single point of failure23:53
clarkbjhesketh: I think 6 months seems to be generally agree upon23:53
jeblaircallbacks can be discarded; it's possible there may be an lru thing or something to limit them.  but even if a callback is dropped, it can be re-established without needing to re-fetch the data (ie, it's just a check to see if it's still up to date)23:54
jeblairclarkb, mordred: ^23:54
*** mudassirlatif has joined #openstack-infra23:54
jeblairclarkb, mordred: that is a vague answer based on my limited knowledge of the protocol but it's the best i got right now23:54
*** bpokorny_ has joined #openstack-infra23:54
fungiwe could at our current utilization push back up to 6 months now according to http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all23:54
*** doug-fish has quit IRC23:54
fungithough that's ignoring things that are stored only on swift i guess?23:54
clarkbfungi: yes23:54
fungino idea how statistically significant that is23:55
jeblairhow much is swift-only?23:55
*** bpokorny_ has quit IRC23:55
mordredI thought we were double-writing everything still23:55
clarkbmordred: we are not, some things are swift only23:55
clarkbmostly infra jobs for dog fooding23:55
jeblair(if it's just our jobs, then.. meh..)23:55
fungiafter the qa team took a hatchet to a lot of the logs, the filesystem utilization dwindled dramatically23:55
clarkbjeblair: yes I think that it is basically us. THere may be one or two other jobs that don't double write23:55
*** piet has joined #openstack-infra23:55
mordredfungi: yay qa team23:55
fungibut that aside, i'm still uneasy about having what amounts to a linear array of block devices as a 14-way point of failure23:56
fungiit's already bitten us at least a couple times23:56
*** bpokorny has quit IRC23:57
fungione iscsi hiccough in the provider's backend causing one of those to get switched read-only and it's a day of lost time rebuilding the logical volume and repairing the filesystem (we've been lucky so far that we haven't just lost the whole shebang)23:58
*** yamamoto_ has quit IRC23:59
clarkbaccording to http://csg.sph.umich.edu/docs/unix/afs/ performance is proportional to number of files and number of bytes granted I have no idea how up to date that is and what their setup tuning was like23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!