| fungi | mordred: oh, i should have caught that. yeah, you have the escaping all wrong for traditional grep regex | 00:00 |
|---|---|---|
| *** watanabe_isao has joined #openstack-infra | 00:02 | |
| mordred | fungi: too much perl in the brain | 00:02 |
| *** zz_dimtruck is now known as dimtruck | 00:02 | |
| clarkb | I think http://logs.openstack.org/65/267865/2/check/gate-dsvm-nodepool/f9f0316/logs/screen-nodepool-builder.txt.gz#_2016-01-15_19_28_41_020 is actually why these noepool jobs are all timing out | 00:02 |
| clarkb | so I am going to look into that now ) | 00:02 |
| clarkb | it is like pulling on a thread until you have no more $article_of_clothing | 00:03 |
| *** sabeen has joined #openstack-infra | 00:03 | |
| clarkb | I thought http://logs.openstack.org/65/267865/2/check/gate-dsvm-nodepool/f9f0316/logs/screen-nodepool-builder.txt.gz#_2016-01-15_19_28_39_254 was supposed to be fixed in dib 1.7.1 ? | 00:04 |
| clarkb | greghaynes: ^ | 00:04 |
| greghaynes | clarkb: yes | 00:04 |
| clarkb | oh that ran with 1.7.0 | 00:04 |
| clarkb | http://logs.openstack.org/65/267865/2/check/gate-dsvm-nodepool/f9f0316/logs/pip-freeze.txt.gz | 00:04 |
| clarkb | so let me recheck and see if we can get more infos | 00:04 |
| *** sabeen2 has joined #openstack-infra | 00:06 | |
| *** annegentle has quit IRC | 00:06 | |
| *** lucas-dinner has quit IRC | 00:06 | |
| *** salv-orlando has quit IRC | 00:07 | |
| *** Jeffrey4l has joined #openstack-infra | 00:07 | |
| *** sabeen has quit IRC | 00:08 | |
| *** gildub has quit IRC | 00:11 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Cleanup the early devstack clone script https://review.openstack.org/268888 | 00:11 |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Update remote refs in early devstack clone https://review.openstack.org/269374 | 00:11 |
| *** lucasagomes has joined #openstack-infra | 00:11 | |
| clarkb | if pabelanger's change comes back with logging working and images building I will likely approve that then approve the changes to fix statsd, then approve the change to fix failed builds | 00:12 |
| ianw | clarkb: ^ i think this is finally the solution and explains why we're seeing what we see (remote branches that never die) | 00:12 |
| clarkb | ianw: will you be pushing a similar change to dib? | 00:12 |
| ianw | clarkb: yeah, i'm going to think about that now. it really seems like we should actually create a proper mirror (git clone --mirror -- a bare repo & fetch with "+refs/*:refs/*") | 00:13 |
| clarkb | ++ though fetching the changes for that may be overkill | 00:13 |
| ianw | but, as it stands, the dib mirror has never got the remote refs because it's doing that +master:master fetch | 00:13 |
| *** salv-orlando has joined #openstack-infra | 00:13 | |
| ianw | in any case, we can much more clearly document wtf it is doing | 00:14 |
| *** pradk has quit IRC | 00:14 | |
| *** amotoki has quit IRC | 00:14 | |
| *** eil397 has joined #openstack-infra | 00:15 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: module is a variable, not a name prefix https://review.openstack.org/269375 | 00:17 |
| *** Qiming has quit IRC | 00:17 | |
| *** otter768 has quit IRC | 00:18 | |
| *** salv-orlando has quit IRC | 00:18 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Start copying hieradata and puppet modules https://review.openstack.org/269376 | 00:20 |
| *** kzaitsev1mb is now known as kzaitsev_mb | 00:20 | |
| mordred | nibalizer, clarkb: 269375 please - the other one can happen tomorrow | 00:21 |
| *** yamamoto has joined #openstack-infra | 00:22 | |
| nibalizer | rgr | 00:22 |
| mordred | nibalizer: btw - with that patch applied - it works | 00:22 |
| mordred | things copy as we expect them to | 00:22 |
| mordred | and they do it pretty cleanly | 00:22 |
| nibalizer | neato | 00:22 |
| mordred | yah man | 00:22 |
| clarkb | well we have to do that parent | 00:22 |
| mordred | yah. the parent too | 00:22 |
| * clarkb reviews | 00:23 | |
| mordred | the parent isn't strictly necessary, but it's pretty simple | 00:23 |
| clarkb | btw hbo has a new recut of the godfather movies | 00:23 |
| mordred | a new old recut | 00:23 |
| clarkb | and if you find yourself with 8 hours of free time | 00:23 |
| mordred | right? | 00:23 |
| clarkb | its new old? | 00:24 |
| clarkb | I only just watched them all for the first time in the last year | 00:24 |
| mordred | yeah - it was first released in 1979 | 00:24 |
| mordred | and was aired on NBC one time - except with some things cut out because profanity etc. | 00:25 |
| mordred | it was also released on VHS briefly I think in 1981 | 00:25 |
| mordred | but it's basically unseen :) | 00:25 |
| *** hichihara has joined #openstack-infra | 00:25 | |
| nibalizer | is it meaningfully different? | 00:25 |
| mordred | (I have a friend whose mom still has a betamax player so that she can watch the tape of it she has from taping it off of tv) | 00:25 |
| mordred | nibalizer: it's cut chronologically | 00:26 |
| *** amotoki has joined #openstack-infra | 00:26 | |
| Thelo | Hi all, I quite often see this error message in jenkins jobs logs , is anyone familiar with it? http://paste.openstack.org/show/484222/ | 00:26 |
| mordred | nibalizer: parts 1 and 2 - together - but scenes in different order | 00:26 |
| *** samuelBartel has quit IRC | 00:26 | |
| clarkb | it starts much slower | 00:26 |
| clarkb | but I think it makes more sense | 00:26 |
| clarkb | Thelo: are you reusing nodes? | 00:26 |
| *** mtanino_ has joined #openstack-infra | 00:27 | |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Rename hieraenvironment to puppet_environment https://review.openstack.org/269299 | 00:28 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: module is a variable, not a name prefix https://review.openstack.org/269375 | 00:28 |
| clarkb | wow that was fast | 00:28 |
| *** tjones has joined #openstack-infra | 00:28 | |
| nibalizer | clarkb: i take that as a sign that puppet testing is slow and ansible-puppet doesn't have the best testing right now | 00:29 |
| Thelo | clarkb: I dont think so, nodepool creates a new node each time no? | 00:29 |
| clarkb | Thelo: it should but you have to configure it to do that | 00:29 |
| clarkb | Thelo: but that log looks like the slave is being deleted under the job | 00:29 |
| *** dimtruck is now known as zz_dimtruck | 00:29 | |
| *** mtanino has quit IRC | 00:29 | |
| clarkb | Thelo: so I would make sure you arne't running >1 job on each node and if so put the offline when job starts stuff in your zuul ocnfig | 00:29 |
| Thelo | clarkb: is there a configuration for this, so I can double check? | 00:31 |
| clarkb | Thelo: yes you should have a zuul function that sets up the nodes for single use | 00:31 |
| *** annegentle has joined #openstack-infra | 00:32 | |
| Thelo | clark: where is that function? | 00:32 |
| clarkb | Thelo: in your zuul config | 00:32 |
| *** yamamoto has quit IRC | 00:33 | |
| clarkb | we have it in openstack_functions.py | 00:33 |
| clarkb | then in layout.yaml we associate that function by default with every job | 00:33 |
| Thelo | I see, great | 00:35 |
| Thelo | thanks | 00:35 |
| clarkb | Thelo: what that does is offline the slave when the job starts which prevents a second job from running on it before nodepool can delete it | 00:36 |
| clarkb | if you don't do this it is possible that a second job would start before nodepool deletes it then nodepool deletes it rseulting in errors | 00:36 |
| openstackgerrit | Morgan Fainberg proposed openstack/requirements: Add "positional" package to g-r https://review.openstack.org/267270 | 00:38 |
| *** akshai_ has joined #openstack-infra | 00:40 | |
| *** achanda has joined #openstack-infra | 00:41 | |
| *** achanda has quit IRC | 00:43 | |
| *** akshai has quit IRC | 00:43 | |
| *** achanda has joined #openstack-infra | 00:43 | |
| *** achanda has quit IRC | 00:44 | |
| *** davideagnello has quit IRC | 00:44 | |
| *** achanda has joined #openstack-infra | 00:45 | |
| *** sdake has quit IRC | 00:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Define defaults as defaults, not variables https://review.openstack.org/269378 | 00:46 |
| mordred | nibalizer, clarkb: one more if you don't mind ... ^^ we got the precedence rules wrong (it's not urgent, nothing is _broken_ - just the 'turn it all on' patch won't work right without it | 00:47 |
| *** achanda has quit IRC | 00:47 | |
| *** Sukhdev has quit IRC | 00:48 | |
| clarkb | I really don't like that new gerrit doesnt give a +/- for each file | 00:50 |
| *** achanda has joined #openstack-infra | 00:50 | |
| clarkb | I guess you get the color coded bar graph thing | 00:50 |
| clarkb | but I like concrete numbes | 00:50 |
| *** ddieterly has quit IRC | 00:50 | |
| nibalizer | mordred: i am phone unfortunately | 00:50 |
| *** zhurong has joined #openstack-infra | 00:51 | |
| mordred | nibalizer: it's ok | 00:51 |
| mordred | nibalizer: it's honestly not a complex patch | 00:51 |
| clarkb | I +2'd | 00:51 |
| *** Qiming has joined #openstack-infra | 00:51 | |
| clarkb | feel free to approve if you are babysitting | 00:51 |
| mordred | yah - I am | 00:51 |
| mordred | thanks! | 00:52 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Define defaults as defaults, not variables https://review.openstack.org/269378 | 00:52 |
| *** Liuqing has joined #openstack-infra | 00:53 | |
| *** achanda has quit IRC | 00:55 | |
| *** pahuang has quit IRC | 00:56 | |
| *** camunoz has quit IRC | 00:56 | |
| *** aeng has quit IRC | 00:56 | |
| *** fawadkhaliq has joined #openstack-infra | 01:00 | |
| *** achanda has joined #openstack-infra | 01:00 | |
| *** fawadkhaliq has quit IRC | 01:00 | |
| *** achanda has quit IRC | 01:01 | |
| *** yamahata has quit IRC | 01:01 | |
| *** Jeffrey4l has quit IRC | 01:01 | |
| *** dims has quit IRC | 01:01 | |
| *** angdraug has quit IRC | 01:03 | |
| clarkb | huh something is insisting on installing dib 1.7.0 | 01:03 |
| *** eil397 has quit IRC | 01:03 | |
| * clarkb is going to rant a bit if it is constraints | 01:03 | |
| clarkb | yup | 01:04 |
| clarkb | thats | 01:04 |
| clarkb | ok | 01:04 |
| anteaya | thanks for warning me | 01:04 |
| *** achanda has joined #openstack-infra | 01:05 | |
| clarkb | so we need to wait for https://review.openstack.org/#/c/268285/ to merge | 01:05 |
| clarkb | which I approved earlier today | 01:06 |
| greghaynes | clarkb: wah | 01:07 |
| greghaynes | clarkb: why does that cause 1.7.0 | 01:07 |
| clarkb | because devstack installs everything using constraints | 01:07 |
| clarkb | and constraints says use 1.7.0 | 01:07 |
| greghaynes | clarkb: yes, that says !=1.6.0 currently | 01:07 |
| clarkb | no thats requirements | 01:07 |
| clarkb | contraints says ==1.7.0 | 01:07 |
| greghaynes | wat | 01:08 |
| clarkb | (the thread is about 20 things long at this point) | 01:08 |
| clarkb | greghaynes: requirements says what you can install, constraints says what you must install if used | 01:08 |
| clarkb | and since we use constraints that means constraints wins | 01:08 |
| greghaynes | clarkb: ya, I know, I am wat'ing at why do we have ==1.7.0 in requirements | 01:08 |
| *** eil397 has joined #openstack-infra | 01:08 | |
| clarkb | because it automagically gets proposed to update and I bet none of the gating jobs use dib | 01:09 |
| clarkb | so the 1.7.0 update passed and merged | 01:09 |
| clarkb | I'm not sure its worth adding such a job, but does expose the gaps in our constraints system | 01:09 |
| *** bmjen has quit IRC | 01:10 | |
| *** Hunner has quit IRC | 01:11 | |
| *** gildub has joined #openstack-infra | 01:11 | |
| *** thorst has joined #openstack-infra | 01:11 | |
| greghaynes | clarkb: definitely not worth adding a dib job until they are using local mirrors, way too unreliable | 01:12 |
| *** ZZelle_ has quit IRC | 01:12 | |
| *** thorst_ has joined #openstack-infra | 01:13 | |
| *** markvoelker has quit IRC | 01:14 | |
| *** Hunner has joined #openstack-infra | 01:15 | |
| *** Hunner has quit IRC | 01:15 | |
| *** Hunner has joined #openstack-infra | 01:15 | |
| *** bmjen has joined #openstack-infra | 01:16 | |
| *** thorst has quit IRC | 01:16 | |
| *** sdake has joined #openstack-infra | 01:17 | |
| *** achanda has quit IRC | 01:17 | |
| *** kzaitsev_mb has quit IRC | 01:19 | |
| *** achanda has joined #openstack-infra | 01:21 | |
| *** shashank_hegde has quit IRC | 01:22 | |
| *** pradk has joined #openstack-infra | 01:23 | |
| *** dims has joined #openstack-infra | 01:23 | |
| *** baoli has joined #openstack-infra | 01:24 | |
| jeblair | nodpool is doing a pretty poor job of understanding the demand from zuul right now | 01:25 |
| *** achanda has quit IRC | 01:26 | |
| jeblair | unfortunately, i don't have any suggestions as to why | 01:26 |
| *** sflanigan has joined #openstack-infra | 01:26 | |
| *** sflanigan has joined #openstack-infra | 01:26 | |
| *** dingyichen has joined #openstack-infra | 01:27 | |
| greghaynes | :( | 01:27 |
| clarkb | jeblair: looks like demand is met? what am I missing? | 01:27 |
| greghaynes | is this a sudden change? | 01:27 |
| jeblair | unless maybe we did something where we changed what nodes some jobs run on recently? | 01:27 |
| clarkb | I guess the post jobs are behind when we do have capacity | 01:27 |
| jeblair | clarkb: i see 841 jobs waiting in the zuul queue, and we're using 50% of our capacity | 01:28 |
| clarkb | if you look at the nodepool status page check and gate are satisfied | 01:28 |
| clarkb | and post is about an hour or so behind | 01:28 |
| *** baoli has quit IRC | 01:29 | |
| *** Daisy has joined #openstack-infra | 01:29 | |
| clarkb | er zuul status page | 01:29 |
| jeblair | clarkb: hrm, are the out-of-window jobs counted in the queue? | 01:30 |
| jeblair | clarkb: no... i don't think they should be | 01:30 |
| clarkb | they shouldn't be | 01:30 |
| jeblair | clarkb: look at http://grafana.openstack.org/dashboard/db/zuul-status so you can actually see the zuul graph | 01:31 |
| clarkb | so this is why I was asking about the drop in the graph if we did anything for that | 01:31 |
| jeblair | clarkb: you think there's either a constant or increasing error in that graph? | 01:32 |
| clarkb | I think it is possible that the image build jobs are piling up but no one seemed to know what caused the drop so I focused on other things | 01:32 |
| clarkb | jeblair: I think it may be the nodepool image build jobs piling up | 01:32 |
| jeblair | clarkb: k, let me see if i can sort the 19230 jobs we have by # pending :) | 01:32 |
| clarkb | (progress was made fwiw, there are a stack of nodepool changes that should be good to go once we can confirm they work after the dib constraints update) | 01:32 |
| clarkb | will fix logging, statsd/paramkiko imports, and the handling of dib build failures | 01:33 |
| jeblair | image-delete:17371110800 | 01:34 |
| clarkb | so thats 1/8th of the number and we have multiple images | 01:34 |
| jeblair | clarkb, greghaynes: ^ there are 7 gearman functions like that (108 builds queued) | 01:35 |
| jeblair | so that accounts for 756 entries in the queue | 01:35 |
| greghaynes | jeblair: eek | 01:35 |
| greghaynes | so it is spinning on trying to delete | 01:35 |
| *** thorst_ has quit IRC | 01:35 | |
| clarkb | | 173711 | fedora-22 | /opt/nodepool_dib/fedora-22-1448720157 | 1448720157 | ready | 51:10:40:44 | | 01:35 |
| clarkb | is the imgae | 01:36 |
| *** rfolco has joined #openstack-infra | 01:36 | |
| greghaynes | half of that is probably that we arent handling failure for image-delete jobs on the nodepool side | 01:36 |
| clarkb | ya | 01:36 |
| greghaynes | I wonder if theres a tb for the other half... | 01:36 |
| *** baoli has joined #openstack-infra | 01:38 | |
| *** bardia has quit IRC | 01:38 | |
| jeblair | greghaynes: i see no 'nodepool.builder' log lines | 01:39 |
| clarkb | jeblair: yes, we think we diagnosed that to the late import | 01:39 |
| greghaynes | right - https://review.openstack.org/#/c/267865/ | 01:39 |
| clarkb | so there is a handful of stuff that cna hopefully be better tomorrow allowing us to clean up the straggler issues | 01:40 |
| jeblair | ah k | 01:40 |
| greghaynes | oh, actually - that patch I linked only applies to separate builder process. | 01:40 |
| greghaynes | clarkb: ah, fun | 01:40 |
| clarkb | greghaynes: ya the fix for late imports should fix logging on in process | 01:40 |
| jeblair | cool, so mystery solved; zuul is okay, we can fix nodepool tomorrow... i think i'll dinnertime now :) | 01:41 |
| * greghaynes writes some tests for these failures | 01:41 | |
| clarkb | yup I think we are good enjoy dinner | 01:41 |
| *** baoli has quit IRC | 01:43 | |
| *** watanabe_isao has quit IRC | 01:44 | |
| dims | clarkb : which version of python 3.4 are we using? (random question :) | 01:45 |
| *** rfolco has quit IRC | 01:46 | |
| *** zhurong has quit IRC | 01:46 | |
| *** eil397 has quit IRC | 01:47 | |
| *** bharathm has joined #openstack-infra | 01:47 | |
| *** Liuqing has quit IRC | 01:47 | |
| *** zhurong has joined #openstack-infra | 01:47 | |
| *** zhurong has quit IRC | 01:48 | |
| *** sc68cal has quit IRC | 01:49 | |
| *** shashank_hegde has joined #openstack-infra | 01:49 | |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Handle dib build failures https://review.openstack.org/268867 | 01:49 |
| greghaynes | clarkb: ^ | 01:49 |
| greghaynes | although itll still fail until the requirements deal merges | 01:49 |
| *** ddieterly has joined #openstack-infra | 01:50 | |
| *** bharathm has quit IRC | 01:52 | |
| clarkb | thanks I will try getting all the things in tomkrrow after dib constraint updates | 01:53 |
| clarkb | then restart the service to pick up the changes | 01:53 |
| *** zhurong has joined #openstack-infra | 01:54 | |
| *** sc68cal has joined #openstack-infra | 01:54 | |
| *** weshay_xchat has joined #openstack-infra | 01:54 | |
| greghaynes | cool, I'll get a delete fix up real quick too | 01:54 |
| *** Jeffrey4l has joined #openstack-infra | 01:56 | |
| *** madorn has joined #openstack-infra | 01:56 | |
| madorn | it looks like the merge requirements on openstack-dev/sandbox have changed -- does it really require a sandbox core reviewer to +2 to merge?? | 01:57 |
| *** weshay_xchat has quit IRC | 01:58 | |
| jhesketh | greghaynes: think you might have a small mistake in that handler (see review) | 01:59 |
| greghaynes | jhesketh: ah, yep | 02:00 |
| *** apoorvad has quit IRC | 02:00 | |
| ianw | greghaynes: re your comment, i kind of saw that as a feature, in that if you just specify a port, it will default back to localhost | 02:01 |
| greghaynes | ianw: oh, I didn't realize that works | 02:01 |
| *** yamamoto has joined #openstack-infra | 02:01 | |
| greghaynes | ianw: thats fine by me, then | 02:01 |
| anteaya | madorn: I believe that everyone is a sandbox core reviewer | 02:01 |
| ianw | greghaynes: yeah, statsd 3.0 specifies defaults, it's a bit different to 2.0 in that respect | 02:01 |
| openstackgerrit | greghaynes proposed openstack-infra/nodepool: Handle dib build failures https://review.openstack.org/268867 | 02:02 |
| clarkb | anteaya: madorn yes any registered user can +2 code review https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 02:02 |
| madorn | clarkb anteaya: i dont have plus +2 | 02:03 |
| clarkb | madorn: are you logged in? | 02:03 |
| madorn | it used to work..maybe im doing something wrong. | 02:03 |
| madorn | yes | 02:03 |
| clarkb | madorn: can you link to the change? | 02:03 |
| madorn | sure | 02:04 |
| *** unicell has joined #openstack-infra | 02:04 | |
| madorn | https://review.openstack.org/#/c/269387/ | 02:04 |
| *** unicell1 has quit IRC | 02:04 | |
| *** mtanino_ has quit IRC | 02:04 | |
| lbragstad | I didn't have +2 when I reviewed it | 02:04 |
| lbragstad | cc: madorn clarkb ^ | 02:04 |
| *** sdake_ has joined #openstack-infra | 02:05 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add inetutils-ping to test-deps https://review.openstack.org/269390 | 02:05 |
| lbragstad | I remember reviewing a change a while back in sandbox (for an onboarding developer) and having +2 | 02:05 |
| anteaya | I also do not have +2 | 02:05 |
| anteaya | lbragstad: yes it seems like what gerrit is telling us re: sandbox access is not the whole truth | 02:05 |
| ianw | greghaynes: not sure if upstream will take it but, https://github.com/jsocol/pystatsd/pull/73 would mean we could drop the whole stats.py wrapper, which would be nicer still | 02:06 |
| lbragstad | anteaya ++ | 02:06 |
| lbragstad | anteaya I've parsed the mailing list... was there a thread on that? | 02:06 |
| anteaya | lbragstad: not that I'm aware | 02:06 |
| anteaya | the sandox? | 02:06 |
| anteaya | or the whole truth? | 02:07 |
| lbragstad | anteaya yeah, the policy around who can merge what on the sandbox | 02:07 |
| *** bpokorny_ has joined #openstack-infra | 02:07 | |
| *** shashank_hegde has quit IRC | 02:07 | |
| anteaya | I don't recall a thread on the merging policy of sandbox | 02:07 |
| *** sdake has quit IRC | 02:07 | |
| anteaya | I believe that we set to to all registered users can merge and expect gerrit to do as told | 02:08 |
| anteaya | seems that didn't happen here | 02:08 |
| *** dslev has joined #openstack-infra | 02:08 | |
| anteaya | perhaps it got toggled somehow in the upgrade in December | 02:08 |
| lbragstad | anteaya gotcha | 02:08 |
| madorn | anteaya, yeah looks like that | 02:08 |
| *** sc68cal has quit IRC | 02:09 | |
| anteaya | I'm assuming clark is looking at gerrit back of house things while we chat | 02:09 |
| anteaya | as I don't have access to those things | 02:09 |
| * lbragstad thanks clarkb | 02:09 | |
| clarkb | uh sorry I just got home inspectio nreport so semi distracted | 02:09 |
| clarkb | but let me take a look | 02:09 |
| anteaya | clarkb: no no, house first | 02:09 |
| anteaya | is it a keeper? | 02:09 |
| anteaya | house is more inportant than sandbox | 02:10 |
| clarkb | anteaya: well nothin gtoo major showed up but probably going to get a second oponion on at least one thing | 02:10 |
| anteaya | yay nothing major showed up! | 02:10 |
| anteaya | \o/ | 02:10 |
| clarkb | so my best guess is that gerrit is silly and doesn't allow registered users to +2 | 02:10 |
| clarkb | because why would you want that | 02:10 |
| anteaya | exactly | 02:10 |
| clarkb | but it clearly shows registered users should be able to _2 | 02:10 |
| clarkb | er +2 | 02:10 |
| anteaya | it does indeed tell us that | 02:11 |
| *** bpokorny has quit IRC | 02:11 | |
| *** Liuqing has joined #openstack-infra | 02:11 | |
| lbragstad | when I look to "Reply" on a review, I can't seem to +2 or -2 it | 02:11 |
| lbragstad | in sandbox | 02:11 |
| anteaya | lbragstad: I see the same | 02:11 |
| *** bpokorny_ has quit IRC | 02:11 | |
| lbragstad | I get the option to -1/+1 and 0 | 02:12 |
| anteaya | reply is the new comment | 02:12 |
| *** shashank_hegde has joined #openstack-infra | 02:12 | |
| anteaya | lbragstad: yup, me too | 02:12 |
| * lbragstad might not be using the new gerrit UI properly... | 02:12 | |
| anteaya | ha ha ha | 02:12 |
| anteaya | ha ha ha | 02:12 |
| *** sc68cal has joined #openstack-infra | 02:12 | |
| * anteaya wipes tears from her eyes | 02:12 | |
| anteaya | lbragstad: thanks for that | 02:13 |
| lbragstad | it's been a learning curve :) | 02:13 |
| anteaya | I don't know as the terms new gerrit UI and properly fit in the same sentence | 02:13 |
| anteaya | ha ha ha, you generous soul you | 02:13 |
| anteaya | clarkb: so any suggestions for a way forward, or have we finished when we agreed gerrit is silly? | 02:14 |
| clarkb | I am going with gerrit is silly, if someone has time they can read through code or file a bug | 02:15 |
| clarkb | maybe zaro has insight | 02:16 |
| anteaya | lbragstad madorn guess we are done here | 02:16 |
| *** kzaitsev_mb has joined #openstack-infra | 02:16 | |
| anteaya | nice to have the chat | 02:16 |
| clarkb | if we really need to approve a chnage for testing purposes we can probably do that | 02:16 |
| anteaya | madorn: do we need to do that? | 02:16 |
| *** yamamoto has quit IRC | 02:17 | |
| *** unicell has quit IRC | 02:17 | |
| madorn | well | 02:18 |
| madorn | we run a class where we walk people through submitting their first bug fix | 02:18 |
| *** otter768 has joined #openstack-infra | 02:18 | |
| madorn | via sandbox | 02:18 |
| madorn | curious if there is another sandbox | 02:19 |
| madorn | or some way we can show them what its like to get some code merged | 02:19 |
| lbragstad | curious if there is another sandbox that let's you approve/merge your own changes? | 02:19 |
| madorn | yes | 02:19 |
| *** unicell has joined #openstack-infra | 02:20 | |
| anteaya | ah okay hmmm | 02:20 |
| madorn | it doesn't even need to happen in real time....if we could just get someone to go through their reviews in the sandbox sometime after class and +2 them | 02:20 |
| anteaya | no you are using sandbox for its intended purpose | 02:21 |
| lbragstad | madorn maybe a bot job? | 02:21 |
| clarkb | ya I think we would want this to work | 02:21 |
| anteaya | when is the class? | 02:21 |
| clarkb | but newer gerrit may have broken us | 02:21 |
| clarkb | if it wasn't almost 6:30pm local time I would dig into java more :) | 02:21 |
| anteaya | madorn: when are you running the next class? | 02:21 |
| madorn | no proble, thanks for looking into it. the class is tomorrow here in california...8:30am to 4pm. They will ahve their reviews posted by the end of the day. | 02:22 |
| *** david-lyle has joined #openstack-infra | 02:22 | |
| anteaya | madorn: okay well thanks for letting us know | 02:23 |
| anteaya | madorn: I'll bring it to the attention of folks tomorrow and we will see what we can do | 02:23 |
| madorn | sure thanks for looking ito it. let me know how i can help, or if you'd like me to file a bug | 02:23 |
| *** otter768 has quit IRC | 02:23 | |
| anteaya | if you could have everyone in the class set a common topic that would be great | 02:23 |
| madorn | sure | 02:24 |
| lbragstad | anteaya do you know if we could get one approve on the project? | 02:24 |
| anteaya | then if we don't have the issue fixed by class time then someone can go through and +A all the patches using that topic | 02:24 |
| lbragstad | approver* | 02:24 |
| anteaya | lbragstad: that is what we will be going to try to figure out | 02:24 |
| anteaya | we want everyone to have +2 and +A | 02:24 |
| anteaya | we just might not have that in place by tomorrow | 02:25 |
| lbragstad | anteaya ah, makes sense - I was going to say that if we can't figure out how to do that by tomorrow, I'd be happy to manually +2 patches for the class | 02:25 |
| anteaya | thanks for letting us know ahead of time | 02:25 |
| anteaya | thanks, we might take you up on that | 02:25 |
| anteaya | I don't have gerrit backend permissions and everyone else is end of day or holidaying | 02:26 |
| lbragstad | madorn anteaya just ping me if you need me to do that | 02:26 |
| anteaya | but I'll make a case for this tomorrow | 02:26 |
| madorn | alright, thanks clarkb anteaya and lbragstad for your help! | 02:26 |
| anteaya | lbragstad: thanks well do | 02:26 |
| anteaya | madorn: thanks for using sandbox for its intended purpose | 02:26 |
| lbragstad | madorn not a problem - thanks for letting us know | 02:26 |
| anteaya | it warms my heart | 02:26 |
| clarkb | worst case we can make a group that gets +2 there that includes registered users | 02:26 |
| anteaya | clarkb: cool | 02:26 |
| clarkb | I am hoping that is sufficiently confusing to work around any gerrit weirdness | 02:26 |
| anteaya | ha ha ha | 02:27 |
| anteaya | I have to get up soon to chair my next meeting | 02:27 |
| anteaya | g'night all | 02:27 |
| clarkb | good night | 02:27 |
| lbragstad | o/ | 02:27 |
| *** keedya has quit IRC | 02:27 | |
| *** MarkAtwood has joined #openstack-infra | 02:28 | |
| *** tjones has quit IRC | 02:29 | |
| greghaynes | clarkb: so if a builder fails to delete, should nodepool still delete the record from the db? | 02:31 |
| *** dslev has quit IRC | 02:31 | |
| clarkb | greghaynes: probably not | 02:32 |
| *** shashank_hegde has quit IRC | 02:33 | |
| greghaynes | clarkb: hrm, do we automatically delete dib images? | 02:34 |
| greghaynes | clarkb: I am now confused why nodepool would spin on a delete | 02:34 |
| greghaynes | clarkb: my tests shows nodepool trying the delte once then hanging forever... | 02:34 |
| clarkb | greghaynes: we delete them when we delete the image in glance | 02:35 |
| clarkb | greghaynes: so its delete glance then delete on disk | 02:35 |
| greghaynes | clarkb: ok, but that is manual? | 02:35 |
| greghaynes | looks like periodicCleanup does deletes | 02:35 |
| *** thorst has joined #openstack-infra | 02:36 | |
| *** gfreeman has quit IRC | 02:36 | |
| clarkb | its periodic | 02:37 |
| greghaynes | ok, so I wonder if the periodic cleanup is what is spinning | 02:37 |
| greghaynes | since the row is not being removed | 02:37 |
| greghaynes | if so, we need a way to signal "we tried to delete, it fails, sorry | 02:38 |
| greghaynes | hrm, actually, the issue isn't that nodepool is spinning on trying to delete, the builder must not be consuming those jobs... | 02:40 |
| *** thorst has quit IRC | 02:44 | |
| *** davideagnello has joined #openstack-infra | 02:45 | |
| *** burgerk has joined #openstack-infra | 02:45 | |
| *** kzaitsev_mb has quit IRC | 02:50 | |
| *** davideagnello has quit IRC | 02:51 | |
| *** markvoelker has joined #openstack-infra | 02:52 | |
| openstackgerrit | Merged openstack-infra/subunit2sql: Omit subunit2sql/tests from coverage https://review.openstack.org/268831 | 02:53 |
| *** burgerk has quit IRC | 02:54 | |
| *** markvoelker_ has joined #openstack-infra | 02:56 | |
| *** e0ne has quit IRC | 02:59 | |
| *** otter768 has joined #openstack-infra | 03:00 | |
| *** e0ne has joined #openstack-infra | 03:00 | |
| *** markvoelker has quit IRC | 03:00 | |
| *** markvoelker_ has quit IRC | 03:01 | |
| *** e0ne has quit IRC | 03:01 | |
| *** lezbar has quit IRC | 03:04 | |
| *** markvoelker has joined #openstack-infra | 03:06 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output https://review.openstack.org/234447 | 03:06 |
| *** yamamoto has joined #openstack-infra | 03:09 | |
| *** shashank_hegde has joined #openstack-infra | 03:09 | |
| *** yuanying has quit IRC | 03:15 | |
| *** rguillebert has quit IRC | 03:17 | |
| *** dims has quit IRC | 03:20 | |
| Thelo | Hi all, recently a new version of os-brick was release but the test result are quite unstable ( sometimes it just does not find the os-brick 0.8.0 library : http://logs.openstack.org/25/267725/2/check/gate-tempest-dsvm-postgres-full/7eea4ff//logs/devstacklog.txt.gz#_2016-01-19_02_39_50_853), it looks like some of the pypi mirrors maybe missing the os-brick release (http://pypi.region-b.geo-1.openstack.org/packages/2.7/o/), does anyone know how to fix th | 03:20 |
| *** achanda has joined #openstack-infra | 03:21 | |
| *** Sukhdev has joined #openstack-infra | 03:22 | |
| *** baoli has joined #openstack-infra | 03:24 | |
| *** shashank_hegde has quit IRC | 03:25 | |
| clarkb | Thelo: you should not use our pypi mirrors | 03:25 |
| clarkb | to fix it an infra-root will need to log in to the machine and diagnose the problem | 03:26 |
| *** links has joined #openstack-infra | 03:27 | |
| clarkb | flock: cannot open lock file /var/run/bandersnatch/mirror.lock: No such file or directory | 03:27 |
| clarkb | but there is no running bandersnatch | 03:27 |
| clarkb | there is no /var/run/bandersnatch dir | 03:27 |
| *** ybathia has joined #openstack-infra | 03:30 | |
| *** tjones has joined #openstack-infra | 03:31 | |
| *** ybathia_ has joined #openstack-infra | 03:32 | |
| *** sflanigan has quit IRC | 03:33 | |
| Thelo | clarkb: is there any infra-root in this timezone? | 03:34 |
| clarkb | yes I just posted the issue | 03:35 |
| clarkb | I don't have time to dig further tonight though | 03:35 |
| *** ybathia has quit IRC | 03:36 | |
| *** ybathia_ is now known as ybathia | 03:36 | |
| clarkb | I think the lock file didn't clean up properly so futher runs are failing | 03:36 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 03:36 |
| *** woodster_ has quit IRC | 03:36 | |
| Thelo | clarkb: so you think its not a mirror issue? | 03:36 |
| clarkb | it is a mirror issue | 03:36 |
| clarkb | it isn't updating because a lock can't be obtained | 03:37 |
| clarkb | that said you should not use our mirrors. we prpvode no warranty for their usefulness outside of our tests | 03:37 |
| Thelo | clarkb: this was tested by your test. | 03:37 |
| clarkb | oh for whatever reason I thought it was third party testing | 03:38 |
| clarkb | in that case maybe someone else has time to debug further. I have a kd on my lap and dinner is soon | 03:39 |
| *** ashleighfarnham has joined #openstack-infra | 03:39 | |
| Thelo | clarkb: Sure, enjoy your evening. Just quickly where did you post this bug? | 03:39 |
| clarkb | 03:27:27 clarkb | flock: cannot open lock file /var/run/bandersnatch/mirror.lock: No such file or directory | 03:39 |
| clarkb | 03:27:59 clarkb | there is no /var/run/bandersnatch dir | 03:40 |
| *** ddieterly has quit IRC | 03:42 | |
| *** thorst has joined #openstack-infra | 03:42 | |
| *** tlian has quit IRC | 03:45 | |
| clarkb | mordred: https://github.com/dear-github/dear-github | 03:46 |
| *** kzaitsev_mb has joined #openstack-infra | 03:46 | |
| jhesketh | clarkb: I'm here, just in a meeting... will be done in a second | 03:48 |
| *** davideagnello has joined #openstack-infra | 03:48 | |
| clarkb | jhesketh: I looked in puppet-bandersnatch and that should be a dir so maybe puppet is broken | 03:49 |
| *** thorst has quit IRC | 03:50 | |
| *** mudassirlatif has joined #openstack-infra | 03:50 | |
| clarkb | http://puppetboard.openstack.org/node/pypi.region-b.geo-1.openstack.org | 03:50 |
| *** sflanigan has joined #openstack-infra | 03:50 | |
| *** sflanigan has joined #openstack-infra | 03:50 | |
| *** yamahata has joined #openstack-infra | 03:52 | |
| *** wei-wang has joined #openstack-infra | 03:52 | |
| *** kzaitsev_mb has quit IRC | 03:52 | |
| *** davideagnello has quit IRC | 03:52 | |
| *** zz_dimtruck is now known as dimtruck | 03:52 | |
| *** tjones has quit IRC | 03:54 | |
| jhesketh | clarkb: hmm okay | 03:55 |
| *** flwang has quit IRC | 03:57 | |
| *** achanda has quit IRC | 04:00 | |
| jhesketh | clarkb: that host is in ansibles static disable:children section | 04:01 |
| jhesketh | so not really sure why it's been disabled | 04:01 |
| *** dimtruck is now known as zz_dimtruck | 04:03 | |
| *** fedexo has joined #openstack-infra | 04:05 | |
| *** yuanying has joined #openstack-infra | 04:07 | |
| *** Daisy has quit IRC | 04:10 | |
| *** ybathia has quit IRC | 04:12 | |
| *** cfarquhar has quit IRC | 04:13 | |
| *** cfarquhar has joined #openstack-infra | 04:14 | |
| *** cfarquhar has quit IRC | 04:14 | |
| *** cfarquhar has joined #openstack-infra | 04:14 | |
| *** yamahata has quit IRC | 04:15 | |
| *** asselin has quit IRC | 04:15 | |
| *** Ramanjaneya has joined #openstack-infra | 04:17 | |
| *** sdake_ has quit IRC | 04:22 | |
| *** moravec has joined #openstack-infra | 04:22 | |
| *** Hal has joined #openstack-infra | 04:23 | |
| *** Hal is now known as Guest68188 | 04:24 | |
| *** Ramanjaneya has quit IRC | 04:28 | |
| *** bpokorny has joined #openstack-infra | 04:28 | |
| openstackgerrit | Shamail Tahir proposed openstack-infra/irc-meetings: Add new bi-weekly APAC meeting for user committee https://review.openstack.org/269427 | 04:29 |
| *** [1]Thelo has joined #openstack-infra | 04:29 | |
| *** shashank_hegde has joined #openstack-infra | 04:30 | |
| *** achanda has joined #openstack-infra | 04:30 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 04:31 |
| *** ashishb has joined #openstack-infra | 04:31 | |
| *** baoli has quit IRC | 04:31 | |
| *** zz_dimtruck is now known as dimtruck | 04:32 | |
| *** Thelo has quit IRC | 04:32 | |
| *** [1]Thelo is now known as Thelo | 04:32 | |
| *** ashleighfarnham has quit IRC | 04:33 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 04:34 |
| *** ashishb has quit IRC | 04:37 | |
| *** achanda has quit IRC | 04:38 | |
| *** watanabe_isao has joined #openstack-infra | 04:38 | |
| *** achanda has joined #openstack-infra | 04:39 | |
| *** markvoelker has quit IRC | 04:39 | |
| *** sflanigan has quit IRC | 04:39 | |
| *** markvoelker has joined #openstack-infra | 04:40 | |
| *** dimtruck is now known as zz_dimtruck | 04:41 | |
| *** moravec has quit IRC | 04:41 | |
| *** ddieterly has joined #openstack-infra | 04:43 | |
| openstackgerrit | fumihiko kakuma proposed openstack-infra/project-config: Run the ofagent ci job in the periodic-qa queue https://review.openstack.org/267323 | 04:44 |
| *** annegentle has quit IRC | 04:45 | |
| *** ddieterly has quit IRC | 04:47 | |
| *** thorst has joined #openstack-infra | 04:49 | |
| *** sabeen2 has quit IRC | 04:49 | |
| *** kzaitsev_mb has joined #openstack-infra | 04:49 | |
| *** mudassirlatif has quit IRC | 04:51 | |
| *** akshai_ has quit IRC | 04:51 | |
| *** shashank_hegde has quit IRC | 04:52 | |
| *** sflanigan has joined #openstack-infra | 04:55 | |
| *** sflanigan has joined #openstack-infra | 04:55 | |
| *** thorst has quit IRC | 04:55 | |
| *** asselin has joined #openstack-infra | 04:56 | |
| *** pradk has quit IRC | 04:58 | |
| *** Daisy has joined #openstack-infra | 05:05 | |
| *** fedexo has quit IRC | 05:07 | |
| *** nikil89_ has joined #openstack-infra | 05:11 | |
| *** nikil89_ is now known as naggappan | 05:11 | |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 05:12 |
| *** mrmartin has joined #openstack-infra | 05:14 | |
| *** ashishb has joined #openstack-infra | 05:15 | |
| *** kzaitsev_mb has quit IRC | 05:22 | |
| *** shashank_hegde has joined #openstack-infra | 05:23 | |
| *** bharathm has joined #openstack-infra | 05:24 | |
| *** Ramanjaneya has joined #openstack-infra | 05:25 | |
| *** bharathm has quit IRC | 05:29 | |
| *** Jeffrey4l has quit IRC | 05:31 | |
| *** Ramanjaneya has quit IRC | 05:32 | |
| *** baoli has joined #openstack-infra | 05:32 | |
| *** otter768 has quit IRC | 05:38 | |
| *** amotoki has quit IRC | 05:39 | |
| *** Ramanjaneya has joined #openstack-infra | 05:40 | |
| *** mrmartin has quit IRC | 05:41 | |
| *** shashank_hegde has quit IRC | 05:42 | |
| *** shashank_hegde has joined #openstack-infra | 05:42 | |
| *** salv-orlando has joined #openstack-infra | 05:42 | |
| *** abhishekk has joined #openstack-infra | 05:43 | |
| abhishekk | hi all, I am facing problem while regstering for ICLA on review.openstack.org, its raising internal server error | 05:44 |
| *** ddieterly has joined #openstack-infra | 05:44 | |
| abhishekk | does any one has any idea about this? | 05:44 |
| *** amotoki has joined #openstack-infra | 05:44 | |
| *** diltram has quit IRC | 05:44 | |
| *** sdake has joined #openstack-infra | 05:44 | |
| *** jaosorior has joined #openstack-infra | 05:45 | |
| watanabe_isao | asselin, ping.:) | 05:46 |
| *** diltram has joined #openstack-infra | 05:47 | |
| *** ddieterly has quit IRC | 05:48 | |
| *** vivekd has joined #openstack-infra | 05:49 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Add extlinux pkg-map for redhat to bootloader https://review.openstack.org/269439 | 05:50 |
| *** jtomasek has joined #openstack-infra | 05:50 | |
| *** ashleighfarnham has joined #openstack-infra | 05:50 | |
| *** elo has quit IRC | 05:51 | |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 05:51 |
| *** abregman has joined #openstack-infra | 05:52 | |
| *** Jeffrey4l has joined #openstack-infra | 05:52 | |
| *** thorst has joined #openstack-infra | 05:53 | |
| AJaeger | stevemar: we need to keep the bandit job unless we backport the changes to liberty. | 05:53 |
| openstackgerrit | Eli Qiao proposed openstack-infra/project-config: Magnum: enable mesos voting https://review.openstack.org/269440 | 05:54 |
| AJaeger | ianw: could you review https://review.openstack.org/#/c/261994 , please? | 05:54 |
| AJaeger | stevemar: I commented also on https://review.openstack.org/#/c/269268 why we need these stepy by step | 05:57 |
| *** ildikov has quit IRC | 05:58 | |
| openstackgerrit | Merged openstack/diskimage-builder: Run package-intalls with py3k if we must https://review.openstack.org/254386 | 05:58 |
| *** jamielennox is now known as jamielennox|away | 05:59 | |
| *** thorst has quit IRC | 06:00 | |
| *** elo has joined #openstack-infra | 06:02 | |
| *** ashleighfarnham has quit IRC | 06:04 | |
| *** HeOS has joined #openstack-infra | 06:05 | |
| *** aukhan has joined #openstack-infra | 06:06 | |
| *** Ramanjaneya has quit IRC | 06:06 | |
| *** bharathm has joined #openstack-infra | 06:08 | |
| *** gildub has quit IRC | 06:13 | |
| *** vivekd has quit IRC | 06:15 | |
| *** salv-orlando has quit IRC | 06:15 | |
| *** Ramanjaneya has joined #openstack-infra | 06:16 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:17 | |
| *** vgridnev has joined #openstack-infra | 06:22 | |
| *** Ramanjaneya has quit IRC | 06:23 | |
| *** kzaitsev_mb has quit IRC | 06:23 | |
| *** salv-orlando has joined #openstack-infra | 06:24 | |
| *** achanda_ has joined #openstack-infra | 06:25 | |
| *** yfried has quit IRC | 06:25 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/267962 | 06:27 |
| *** elo has quit IRC | 06:27 | |
| *** elo has joined #openstack-infra | 06:28 | |
| *** achanda has quit IRC | 06:28 | |
| *** amotoki_ has joined #openstack-infra | 06:32 | |
| *** amotoki has quit IRC | 06:34 | |
| *** amotoki_ has quit IRC | 06:35 | |
| *** bharathm has quit IRC | 06:36 | |
| *** davideagnello has joined #openstack-infra | 06:37 | |
| *** elo has quit IRC | 06:37 | |
| *** davideagnello has quit IRC | 06:38 | |
| *** elo has joined #openstack-infra | 06:38 | |
| *** vivekd has joined #openstack-infra | 06:39 | |
| stevemar | AJaeger: yeah, i saw the comments, i think i lost in all the depends-on | 06:43 |
| *** achanda_ has quit IRC | 06:43 | |
| AJaeger | stevemar: sorry for losing you | 06:44 |
| *** amotoki has joined #openstack-infra | 06:44 | |
| stevemar | AJaeger: don't sweat it | 06:44 |
| stevemar | AJaeger: i shouldn't have rushed it | 06:44 |
| *** shashank_hegde has quit IRC | 06:44 | |
| *** bharathm_ has joined #openstack-infra | 06:44 | |
| *** ddieterly has joined #openstack-infra | 06:45 | |
| *** bharathm_ has quit IRC | 06:45 | |
| AJaeger | stevemar: no worries, thanks for the reviews | 06:45 |
| *** vgridnev has quit IRC | 06:45 | |
| *** annegentle has joined #openstack-infra | 06:46 | |
| stevemar | AJaeger: always a pleasure sir | 06:46 |
| *** ddieterly has quit IRC | 06:49 | |
| *** nihilifer has quit IRC | 06:49 | |
| *** nihilifer has joined #openstack-infra | 06:50 | |
| *** annegentle has quit IRC | 06:50 | |
| Thelo | HI, is there any infra-root that could help on this : recently a new version of os-brick was release but the test result are quite unstable ( sometimes it just does not find the os-brick 0.8.0 library : http://logs.openstack.org/25/267725/2/check/gate-tempest-dsvm-postgres-full/7eea4ff//logs/devstacklog.txt.gz#_2016-01-19_02_39_50_853), it looks like some of the pypi mirrors maybe missing the os-brick release (http://pypi.region-b.geo-1.openstack.org/ | 06:51 |
| Thelo | packages/2.7/o/), does anyone know how to fix this | 06:51 |
| *** Jeffrey4l has quit IRC | 06:52 | |
| jhesketh | Thelo: yeah clarkb looked at that earlier and it looks like that mirror isn't syncing with puppet | 06:52 |
| jhesketh | for some reason it's been disabled in the ansible inventory, but I'm not sure why | 06:52 |
| jhesketh | as such, I don't really want to take it out in case there was a good reason | 06:52 |
| Thelo | jhesketh: Do you know who could help with that? | 06:54 |
| jhesketh | Thelo: not sure sorry... I' | 06:56 |
| *** thorst has joined #openstack-infra | 06:57 | |
| *** amotoki has quit IRC | 07:00 | |
| *** shashank_hegde has joined #openstack-infra | 07:00 | |
| * jhesketh is looking to see if there's anything obvious | 07:00 | |
| *** jcoufal has joined #openstack-infra | 07:03 | |
| *** yfried has joined #openstack-infra | 07:04 | |
| *** thorst has quit IRC | 07:04 | |
| *** salv-orlando has quit IRC | 07:05 | |
| *** _nadya_ has joined #openstack-infra | 07:09 | |
| *** markvoelker_ has joined #openstack-infra | 07:11 | |
| *** markvoelker has quit IRC | 07:15 | |
| *** vgridnev has joined #openstack-infra | 07:16 | |
| *** tdurakov_ has joined #openstack-infra | 07:16 | |
| *** amotoki has joined #openstack-infra | 07:17 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add py27 to steth https://review.openstack.org/269460 | 07:18 |
| *** tdurakov_ has quit IRC | 07:19 | |
| *** mudassirlatif has joined #openstack-infra | 07:19 | |
| *** flepied1 has quit IRC | 07:20 | |
| *** kzaitsev_mb has joined #openstack-infra | 07:20 | |
| *** shashank_hegde has quit IRC | 07:21 | |
| *** elo has quit IRC | 07:23 | |
| *** ccrouch has quit IRC | 07:26 | |
| *** Ramanjaneya has joined #openstack-infra | 07:27 | |
| *** nmagnezi has joined #openstack-infra | 07:27 | |
| *** mpavone has joined #openstack-infra | 07:28 | |
| *** scheuran has joined #openstack-infra | 07:34 | |
| *** hdd has quit IRC | 07:35 | |
| *** changzhi has joined #openstack-infra | 07:35 | |
| *** placko has joined #openstack-infra | 07:36 | |
| *** otter768 has joined #openstack-infra | 07:39 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add py27 and docs to steth https://review.openstack.org/269460 | 07:40 |
| *** Jeffrey4l has joined #openstack-infra | 07:42 | |
| *** tdurakov_ has joined #openstack-infra | 07:43 | |
| *** hdd has joined #openstack-infra | 07:43 | |
| *** otter768 has quit IRC | 07:44 | |
| *** kzaitsev_mb has quit IRC | 07:45 | |
| *** ddieterly has joined #openstack-infra | 07:46 | |
| *** jlanoux has joined #openstack-infra | 07:46 | |
| *** vgridnev has quit IRC | 07:49 | |
| *** binbincong has quit IRC | 07:49 | |
| *** jpr1 has joined #openstack-infra | 07:50 | |
| *** jpr2 has joined #openstack-infra | 07:50 | |
| *** amotoki has quit IRC | 07:50 | |
| *** aopincar has joined #openstack-infra | 07:51 | |
| *** samuelBartel has joined #openstack-infra | 07:51 | |
| *** ddieterly has quit IRC | 07:52 | |
| *** eliqiao_ is now known as eliqiao | 07:52 | |
| *** wei-wang has quit IRC | 07:53 | |
| *** mudassirlatif has quit IRC | 07:54 | |
| *** jpr1 has quit IRC | 07:54 | |
| *** Jeffrey4l has quit IRC | 07:56 | |
| *** wei-wang has joined #openstack-infra | 07:56 | |
| *** markvoelker_ has quit IRC | 07:57 | |
| *** wei-wang has quit IRC | 07:58 | |
| *** _nadya_ has quit IRC | 07:59 | |
| anteaya | abhishekk: try reading this: https://ask.openstack.org/en/question/56720/cannot-store-contact-information-when-updating-info-in-openstack-gerrit/ | 08:00 |
| *** rcernin has joined #openstack-infra | 08:01 | |
| *** binbincong has joined #openstack-infra | 08:02 | |
| openstackgerrit | Zhi Chang proposed openstack-infra/project-config: Add py27 and docs to steth https://review.openstack.org/269460 | 08:03 |
| *** tdurakov_ is now known as tdurakov | 08:04 | |
| *** thorst has joined #openstack-infra | 08:05 | |
| *** salv-orlando has joined #openstack-infra | 08:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Set Ceilometer backend to Gnocchi in integration test https://review.openstack.org/263325 | 08:06 |
| *** rossella_s has quit IRC | 08:08 | |
| *** ildikov has joined #openstack-infra | 08:09 | |
| *** flepied has joined #openstack-infra | 08:10 | |
| *** thorst has quit IRC | 08:10 | |
| *** markus_z has joined #openstack-infra | 08:10 | |
| *** salv-orlando has quit IRC | 08:11 | |
| *** HeOS has quit IRC | 08:12 | |
| *** Jeffrey4l has joined #openstack-infra | 08:14 | |
| *** flepied has quit IRC | 08:17 | |
| *** vgridnev has joined #openstack-infra | 08:19 | |
| *** isaacb has joined #openstack-infra | 08:22 | |
| *** arxcruz has joined #openstack-infra | 08:23 | |
| *** dingyichen has quit IRC | 08:25 | |
| *** mrmartin has joined #openstack-infra | 08:26 | |
| *** chlong has quit IRC | 08:27 | |
| *** matrohon has joined #openstack-infra | 08:28 | |
| *** sdake_ has joined #openstack-infra | 08:28 | |
| *** ifarkas has joined #openstack-infra | 08:29 | |
| *** sdake has quit IRC | 08:31 | |
| *** vivekd has quit IRC | 08:32 | |
| *** acruz has joined #openstack-infra | 08:35 | |
| *** yfried_ has joined #openstack-infra | 08:35 | |
| *** yfried has quit IRC | 08:35 | |
| *** binbincong has quit IRC | 08:35 | |
| *** arxcruz has quit IRC | 08:36 | |
| *** markvoelker has joined #openstack-infra | 08:38 | |
| *** davideagnello has joined #openstack-infra | 08:38 | |
| *** kzaitsev_mb has joined #openstack-infra | 08:39 | |
| *** daemontool has joined #openstack-infra | 08:41 | |
| *** placko has quit IRC | 08:41 | |
| *** vivekd has joined #openstack-infra | 08:42 | |
| *** Sukhdev has quit IRC | 08:43 | |
| *** markvoelker has quit IRC | 08:43 | |
| *** davideagnello has quit IRC | 08:44 | |
| openstackgerrit | Merged openstack-infra/project-config: Add neutron -plus job template for advanced features https://review.openstack.org/247697 | 08:45 |
| *** markvoelker has joined #openstack-infra | 08:46 | |
| *** jordanP has joined #openstack-infra | 08:48 | |
| *** ddieterly has joined #openstack-infra | 08:49 | |
| *** ifarkas has quit IRC | 08:49 | |
| *** hdd has quit IRC | 08:50 | |
| *** tdurakov has quit IRC | 08:50 | |
| abhishekk | anteya: thank you | 08:51 |
| *** tdurakov has joined #openstack-infra | 08:52 | |
| *** flepied has joined #openstack-infra | 08:53 | |
| *** ddieterly has quit IRC | 08:53 | |
| *** fhubik has joined #openstack-infra | 08:54 | |
| *** fhubik is now known as fhubik_brb | 08:54 | |
| *** lezbar has joined #openstack-infra | 08:56 | |
| *** ifarkas has joined #openstack-infra | 08:58 | |
| *** ifarkas has quit IRC | 08:59 | |
| *** ifarkas has joined #openstack-infra | 08:59 | |
| *** vivekd has quit IRC | 08:59 | |
| *** binbincong has joined #openstack-infra | 09:00 | |
| *** fhubik_brb is now known as fhubik | 09:01 | |
| *** markvoelker has quit IRC | 09:02 | |
| *** daemontool has quit IRC | 09:02 | |
| *** watanabe_isao has quit IRC | 09:02 | |
| *** e0ne has joined #openstack-infra | 09:03 | |
| AJaeger | anteaya: FYI: https://review.openstack.org/#/c/269440 -1 for voting job only in check | 09:03 |
| *** daemontool has joined #openstack-infra | 09:03 | |
| *** vivekd has joined #openstack-infra | 09:04 | |
| *** fhubik is now known as fhubik_brb | 09:04 | |
| *** sdake_ has quit IRC | 09:07 | |
| *** thorst has joined #openstack-infra | 09:07 | |
| *** MarkAtwood has quit IRC | 09:08 | |
| *** jordanP has quit IRC | 09:08 | |
| openstackgerrit | Merged openstack-infra/project-config: Update remote refs in early devstack clone https://review.openstack.org/269374 | 09:09 |
| *** sputnik13 has joined #openstack-infra | 09:11 | |
| *** amotoki has joined #openstack-infra | 09:11 | |
| *** dizquierdo has joined #openstack-infra | 09:11 | |
| openstackgerrit | Vitaly Gridnev proposed openstack-infra/project-config: Skip one test to unlock saharaclient gate https://review.openstack.org/267561 | 09:11 |
| *** HeOS has joined #openstack-infra | 09:12 | |
| *** thorst has quit IRC | 09:15 | |
| *** kzaitsev_mb has quit IRC | 09:16 | |
| daemontool | clarkb, ping | 09:17 |
| daemontool | or anyone available, have issues with http://logs.openstack.org/06/269206/3/check/gate-freezer-devstack-dsvm/405ec04/ | 09:17 |
| daemontool | related to: http://logs.openstack.org/06/269206/3/check/gate-freezer-devstack-dsvm/405ec04/ | 09:17 |
| daemontool | but I'm not sure how to solve it | 09:17 |
| *** markvoelker has joined #openstack-infra | 09:19 | |
| *** jistr has joined #openstack-infra | 09:20 | |
| *** matrohon has quit IRC | 09:20 | |
| *** hichihara has quit IRC | 09:22 | |
| *** hichihara has joined #openstack-infra | 09:22 | |
| *** fhubik_brb is now known as fhubik | 09:23 | |
| *** markvoelker has quit IRC | 09:24 | |
| *** hashar has joined #openstack-infra | 09:33 | |
| *** e0ne has quit IRC | 09:34 | |
| *** matrohon has joined #openstack-infra | 09:36 | |
| *** Daisy has quit IRC | 09:37 | |
| *** electrofelix has joined #openstack-infra | 09:38 | |
| *** daemontool has quit IRC | 09:39 | |
| *** daemontool has joined #openstack-infra | 09:39 | |
| *** markvoelker has joined #openstack-infra | 09:40 | |
| *** otter768 has joined #openstack-infra | 09:40 | |
| *** fhubik is now known as fhubik_brb | 09:40 | |
| *** derekh has joined #openstack-infra | 09:43 | |
| *** boris-42 has quit IRC | 09:43 | |
| *** markvoelker has quit IRC | 09:45 | |
| *** otter768 has quit IRC | 09:45 | |
| *** fhubik_brb is now known as fhubik | 09:46 | |
| *** fhubik is now known as fhubik_brb | 09:47 | |
| *** amotoki has quit IRC | 09:48 | |
| *** e0ne has joined #openstack-infra | 09:48 | |
| *** ddieterly has joined #openstack-infra | 09:50 | |
| *** sdake has joined #openstack-infra | 09:51 | |
| *** dtardivel has joined #openstack-infra | 09:51 | |
| *** abhishekk has left #openstack-infra | 09:53 | |
| *** aukhan has quit IRC | 09:53 | |
| *** aukhan has joined #openstack-infra | 09:54 | |
| *** ddieterly has quit IRC | 09:54 | |
| *** markus_z is now known as markus_z_meeting | 09:56 | |
| *** e0ne has quit IRC | 09:57 | |
| *** sshnaidm has quit IRC | 09:57 | |
| *** dizquierdo has quit IRC | 09:59 | |
| *** Jeffrey4l has quit IRC | 10:00 | |
| *** vgridnev has quit IRC | 10:06 | |
| *** vgridnev has joined #openstack-infra | 10:07 | |
| *** salv-orlando has joined #openstack-infra | 10:07 | |
| *** daemontool has quit IRC | 10:08 | |
| *** timothyb89 has quit IRC | 10:08 | |
| *** daemontool has joined #openstack-infra | 10:08 | |
| *** timothyb89 has joined #openstack-infra | 10:10 | |
| *** sshnaidm has joined #openstack-infra | 10:10 | |
| *** fhubik_brb is now known as fhubik | 10:10 | |
| *** Ramanjaneya has quit IRC | 10:10 | |
| *** daemontool has quit IRC | 10:11 | |
| *** wei-wang has joined #openstack-infra | 10:11 | |
| *** daemontool has joined #openstack-infra | 10:11 | |
| *** Qiming has quit IRC | 10:12 | |
| *** jordanP has joined #openstack-infra | 10:12 | |
| *** thorst has joined #openstack-infra | 10:12 | |
| *** salv-orlando has quit IRC | 10:13 | |
| *** vgridnev has quit IRC | 10:13 | |
| *** _nadya_ has joined #openstack-infra | 10:14 | |
| *** pigmej has joined #openstack-infra | 10:15 | |
| *** sdake has quit IRC | 10:15 | |
| *** markvoelker has joined #openstack-infra | 10:16 | |
| *** daemontool has quit IRC | 10:17 | |
| *** daemontool has joined #openstack-infra | 10:17 | |
| *** rguillebert has joined #openstack-infra | 10:17 | |
| *** dtantsur|afk is now known as dtantsur | 10:18 | |
| *** placko_ has joined #openstack-infra | 10:18 | |
| *** amotoki has joined #openstack-infra | 10:18 | |
| *** Jeffrey4l has joined #openstack-infra | 10:18 | |
| *** daemontool has quit IRC | 10:19 | |
| *** daemontool has joined #openstack-infra | 10:19 | |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Adds ironic neutron integration testing https://review.openstack.org/252814 | 10:20 |
| *** thorst has quit IRC | 10:20 | |
| *** ociuhandu has quit IRC | 10:21 | |
| *** markvoelker has quit IRC | 10:21 | |
| *** yfried has joined #openstack-infra | 10:22 | |
| *** vgridnev has joined #openstack-infra | 10:26 | |
| *** yfried_ has quit IRC | 10:27 | |
| *** aspiers has quit IRC | 10:27 | |
| Thelo | Hi, any infra-root here? | 10:29 |
| *** ldnunes has joined #openstack-infra | 10:30 | |
| *** AzherKhan has joined #openstack-infra | 10:30 | |
| *** Jeffrey4l has quit IRC | 10:31 | |
| *** aukhan has quit IRC | 10:31 | |
| *** sputnik13 has quit IRC | 10:32 | |
| *** zhurong has quit IRC | 10:33 | |
| *** zeih has joined #openstack-infra | 10:37 | |
| *** hichihara has quit IRC | 10:38 | |
| *** daemontool has quit IRC | 10:38 | |
| *** davideagnello has joined #openstack-infra | 10:40 | |
| *** daemontool has joined #openstack-infra | 10:42 | |
| *** ildikov has quit IRC | 10:44 | |
| *** markvoelker has joined #openstack-infra | 10:44 | |
| *** davideagnello has quit IRC | 10:46 | |
| *** dizquierdo has joined #openstack-infra | 10:48 | |
| *** dims has joined #openstack-infra | 10:48 | |
| *** rfolco has joined #openstack-infra | 10:48 | |
| *** vgridnev has quit IRC | 10:49 | |
| *** markvoelker has quit IRC | 10:49 | |
| *** ddieterly has joined #openstack-infra | 10:50 | |
| *** AzherKhan has quit IRC | 10:54 | |
| *** ihrachys has joined #openstack-infra | 10:54 | |
| *** ddieterly has quit IRC | 10:55 | |
| *** yamamoto has quit IRC | 10:55 | |
| *** tdurakov has quit IRC | 10:56 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/requirements: Pin more broken DIB versions: 1.6.0-1.7.1 https://review.openstack.org/269553 | 10:57 |
| *** erlon has joined #openstack-infra | 11:00 | |
| *** Qiming has joined #openstack-infra | 11:05 | |
| *** Roamer` has quit IRC | 11:06 | |
| *** aysyd has joined #openstack-infra | 11:06 | |
| *** rossella_s has joined #openstack-infra | 11:07 | |
| *** ildikov has joined #openstack-infra | 11:09 | |
| *** wei-wang has quit IRC | 11:11 | |
| *** pigmej has left #openstack-infra | 11:12 | |
| andreaf | hello, I'm looking for a puppet review in https://review.openstack.org/#/c/269136/ - it's just adding "if ! defined" statements to imports in log_processor puppet module | 11:13 |
| andreaf | and a tiny infra doc update as well: https://review.openstack.org/#/c/254725/ (periodic pipeline test results are collected into the DB as well now) | 11:14 |
| *** markvoelker has joined #openstack-infra | 11:14 | |
| *** changzhi has quit IRC | 11:15 | |
| *** vgridnev has joined #openstack-infra | 11:15 | |
| *** tdurakov has joined #openstack-infra | 11:16 | |
| *** zeih has quit IRC | 11:16 | |
| openstackgerrit | Merged openstack/requirements: Blacklist DIB 1.7.0 https://review.openstack.org/268285 | 11:17 |
| *** thorst has joined #openstack-infra | 11:17 | |
| *** markvoelker has quit IRC | 11:20 | |
| *** sfinucan has joined #openstack-infra | 11:20 | |
| *** jaosorior has quit IRC | 11:22 | |
| *** jaosorior has joined #openstack-infra | 11:22 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/requirements: Pin more broken DIB versions: 1.6.0-1.7.1 https://review.openstack.org/269553 | 11:23 |
| dtantsur | ttx, had to rebase ^^ | 11:23 |
| *** thorst has quit IRC | 11:25 | |
| *** fhubik has quit IRC | 11:27 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins https://review.openstack.org/269567 | 11:28 |
| *** aopincar has quit IRC | 11:30 | |
| *** aspiers has joined #openstack-infra | 11:30 | |
| *** markvoelker has joined #openstack-infra | 11:32 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/267962 | 11:32 |
| *** zeih has joined #openstack-infra | 11:35 | |
| *** ericksonsantos has joined #openstack-infra | 11:36 | |
| *** markvoelker has quit IRC | 11:36 | |
| *** ociuhandu has joined #openstack-infra | 11:37 | |
| *** e0ne has joined #openstack-infra | 11:38 | |
| *** zeih has quit IRC | 11:40 | |
| *** binbincong has quit IRC | 11:40 | |
| *** otter768 has joined #openstack-infra | 11:41 | |
| *** yamamoto has joined #openstack-infra | 11:45 | |
| *** otter768 has quit IRC | 11:46 | |
| *** Ramanjaneya has joined #openstack-infra | 11:47 | |
| *** markus_z_meeting is now known as markus_z | 11:49 | |
| openstackgerrit | Timur Sufiev proposed openstack-infra/project-config: Source exports from horizon repo for horizon dsvm-job https://review.openstack.org/269578 | 11:49 |
| *** treaki has joined #openstack-infra | 11:50 | |
| *** Ramanjaneya has quit IRC | 11:50 | |
| *** Ramanjaneya has joined #openstack-infra | 11:51 | |
| *** ddieterly has joined #openstack-infra | 11:51 | |
| *** zeih has joined #openstack-infra | 11:54 | |
| *** jlanoux has quit IRC | 11:55 | |
| *** ddieterly has quit IRC | 11:55 | |
| *** tdurakov has quit IRC | 12:00 | |
| *** binbincong has joined #openstack-infra | 12:01 | |
| openstackgerrit | Quan Zhou proposed openstack-infra/activity-board: Update JS library 15.06-38-gcd52ae4 https://review.openstack.org/269582 | 12:02 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements https://review.openstack.org/268438 | 12:02 |
| *** yamamoto has quit IRC | 12:03 | |
| *** weshay_xchat has joined #openstack-infra | 12:04 | |
| *** daemontool has quit IRC | 12:05 | |
| *** Thelo_ has joined #openstack-infra | 12:05 | |
| *** daemontool has joined #openstack-infra | 12:06 | |
| *** Liuqing has quit IRC | 12:06 | |
| *** yamamoto has joined #openstack-infra | 12:07 | |
| *** baoli has quit IRC | 12:07 | |
| *** salv-orlando has joined #openstack-infra | 12:08 | |
| *** Ramanjaneya_ has joined #openstack-infra | 12:08 | |
| *** ashishb has quit IRC | 12:10 | |
| *** yamamoto_ has joined #openstack-infra | 12:11 | |
| *** vivekd_ has joined #openstack-infra | 12:11 | |
| *** markvoelker has joined #openstack-infra | 12:11 | |
| *** vivekd has quit IRC | 12:11 | |
| *** vivekd_ is now known as vivekd | 12:11 | |
| *** vivekd_ has joined #openstack-infra | 12:12 | |
| *** salv-orlando has quit IRC | 12:13 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** hashar has quit IRC | 12:15 | |
| openstackgerrit | Merged openstack-infra/activity-board: Update JS library 15.06-38-gcd52ae4 https://review.openstack.org/269582 | 12:15 |
| *** vivekd has quit IRC | 12:16 | |
| *** vivekd_ is now known as vivekd | 12:16 | |
| *** markvoelker has quit IRC | 12:16 | |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Upload metadata for pseudo folder https://review.openstack.org/263240 | 12:16 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/project-config: Send instructions to workers to upload metadata https://review.openstack.org/263218 | 12:16 |
| *** _nadya_ has quit IRC | 12:16 | |
| openstackgerrit | Angus Lees proposed openstack/requirements: Add oslo.privsep to global-requirements.txt https://review.openstack.org/266189 | 12:16 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_libvirt job https://review.openstack.org/269592 | 12:17 |
| *** julim has joined #openstack-infra | 12:19 | |
| *** yamamoto_ has quit IRC | 12:23 | |
| *** _nadya_ has joined #openstack-infra | 12:26 | |
| *** tdurakov has joined #openstack-infra | 12:27 | |
| ttx | dtantsur: re +2ed | 12:30 |
| *** jtomasek__ has joined #openstack-infra | 12:30 | |
| *** jtomasek__ has quit IRC | 12:30 | |
| *** jtomasek_ has joined #openstack-infra | 12:30 | |
| *** dizquierdo has quit IRC | 12:33 | |
| *** ildikov has quit IRC | 12:35 | |
| *** dizquierdo has joined #openstack-infra | 12:36 | |
| *** davideagnello has joined #openstack-infra | 12:42 | |
| *** links has quit IRC | 12:46 | |
| *** davideagnello has quit IRC | 12:47 | |
| *** thorst has joined #openstack-infra | 12:48 | |
| *** markvoelker has joined #openstack-infra | 12:49 | |
| *** amrith is now known as _amrith_ | 12:50 | |
| *** moravec has joined #openstack-infra | 12:50 | |
| *** binbincong has quit IRC | 12:51 | |
| *** ddieterly has joined #openstack-infra | 12:52 | |
| *** Guest68188 has quit IRC | 12:52 | |
| *** mpavone has quit IRC | 12:52 | |
| *** markvoelker has quit IRC | 12:54 | |
| *** binbincong has joined #openstack-infra | 12:56 | |
| *** ddieterly has quit IRC | 12:56 | |
| *** ildikov has joined #openstack-infra | 12:57 | |
| *** yamamoto has joined #openstack-infra | 12:59 | |
| *** Hal has joined #openstack-infra | 13:01 | |
| *** mpavone has joined #openstack-infra | 13:01 | |
| *** mpavone has quit IRC | 13:01 | |
| *** Hal is now known as Guest17173 | 13:02 | |
| *** jlanoux has joined #openstack-infra | 13:02 | |
| *** sdake has joined #openstack-infra | 13:06 | |
| *** aopincar has joined #openstack-infra | 13:07 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper https://review.openstack.org/269284 | 13:07 |
| *** dprince has joined #openstack-infra | 13:07 | |
| *** ddieterly has joined #openstack-infra | 13:08 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-puppet: Properly suppress undefined warning https://review.openstack.org/269620 | 13:09 |
| *** moravec has quit IRC | 13:09 | |
| *** markvoelker has joined #openstack-infra | 13:10 | |
| *** daemontool_ has joined #openstack-infra | 13:10 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use the puppet role to copy the puppet code https://review.openstack.org/251581 | 13:10 |
| mordred | morning fungi - when you awaken, could you bless those two? ^^ | 13:11 |
| *** daemontool has quit IRC | 13:11 | |
| *** gordc has joined #openstack-infra | 13:11 | |
| *** vgridnev has quit IRC | 13:12 | |
| *** wei-wang has joined #openstack-infra | 13:13 | |
| *** vgridnev has joined #openstack-infra | 13:14 | |
| *** markvoelker has quit IRC | 13:15 | |
| daemontool_ | hi, if anyone is available, we need some help with http://logs.openstack.org/06/269206/3/check/gate-freezer-devstack-dsvm/405ec04/console.html | 13:16 |
| daemontool_ | after branch creation the error we get at the gate is "ERROR: branch not allowed by features matrix: 1.2.0" | 13:17 |
| daemontool_ | branch create is stable/1.2.0 | 13:17 |
| *** arxcruz has joined #openstack-infra | 13:18 | |
| *** ekarlso has quit IRC | 13:18 | |
| *** ekarlso has joined #openstack-infra | 13:18 | |
| *** acruz has quit IRC | 13:19 | |
| *** dizquierdo has quit IRC | 13:23 | |
| *** isaacb has quit IRC | 13:24 | |
| *** rhallisey has quit IRC | 13:26 | |
| *** rhallisey has joined #openstack-infra | 13:27 | |
| *** dslev has joined #openstack-infra | 13:30 | |
| *** edmondsw has joined #openstack-infra | 13:30 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper https://review.openstack.org/269284 | 13:31 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Fix some minor project group issues https://review.openstack.org/269627 | 13:31 |
| EmilienM | AJaeger: it that better now? https://review.openstack.org/#/c/269284/2..3/zuul/layout.yaml | 13:31 |
| EmilienM | AJaeger: I was wondering if noop-jobs would also in check, or only gate in this case | 13:32 |
| openstackgerrit | Merged openstack/requirements: bump versions for each keystone library https://review.openstack.org/266629 | 13:37 |
| *** kgiusti has joined #openstack-infra | 13:37 | |
| *** placko_ has quit IRC | 13:39 | |
| *** otter768 has joined #openstack-infra | 13:42 | |
| *** vivekd_ has joined #openstack-infra | 13:42 | |
| *** tlian has joined #openstack-infra | 13:43 | |
| *** yfried has quit IRC | 13:44 | |
| *** vivekd has quit IRC | 13:45 | |
| *** vivekd_ is now known as vivekd | 13:45 | |
| *** Ramanjaneya has quit IRC | 13:46 | |
| *** Ramanjaneya_ has quit IRC | 13:46 | |
| *** otter768 has quit IRC | 13:47 | |
| mordred | clarkb: whence you awake, the nodepool-src jobs in shade have gone red, but for the life of me I cannot find any actual errors | 13:47 |
| mordred | clarkb: http://logs.openstack.org/05/267405/5/check/gate-dsvm-nodepool-src-shade/c3e9585/ for instance | 13:47 |
| mordred | clarkb: it seems to be timing out - but all of the things seem to be successing | 13:48 |
| *** aopincar has quit IRC | 13:48 | |
| *** jaosorior has quit IRC | 13:48 | |
| *** jaosorior has joined #openstack-infra | 13:48 | |
| clarkb | mordred it is due to dib 1.7.0 breaking image builds. New dib constraint that went in overnight should fix | 13:49 |
| mordred | ok. cool | 13:49 |
| clarkb | also I am not actually awake right now | 13:49 |
| *** daemontool__ has joined #openstack-infra | 13:49 | |
| mordred | clarkb: sweet. don't be awake then | 13:49 |
| krotscheck | Any non-mordred cores around to review the wheel-mirror patches? https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open | 13:49 |
| mordred | awww. | 13:50 |
| krotscheck | clarkb: I know how you feel. | 13:50 |
| * mordred feels left out | 13:50 | |
| *** daemontool_ has quit IRC | 13:50 | |
| krotscheck | mordred: Feel like a pioneer! You were there first! | 13:50 |
| mordred | Shrews: ^^ see above | 13:50 |
| mordred | krotscheck: \o/ | 13:50 |
| mordred | krotscheck: did I receive dissentary for my troubles? | 13:50 |
| krotscheck | Dysentery? No, but you broke your leg near chimney rock. | 13:50 |
| *** ccrouch has joined #openstack-infra | 13:50 | |
| mordred | woohoo! | 13:51 |
| *** markvoelker has joined #openstack-infra | 13:52 | |
| *** dkranz has joined #openstack-infra | 13:53 | |
| *** Roamer` has joined #openstack-infra | 13:53 | |
| *** daemontool__ is now known as daemontool | 13:54 | |
| *** burgerk has joined #openstack-infra | 13:55 | |
| *** markvoelker has quit IRC | 13:57 | |
| *** nelsnelson has joined #openstack-infra | 13:58 | |
| *** amotoki has quit IRC | 13:58 | |
| *** ccrouch has quit IRC | 13:58 | |
| *** markvoelker has joined #openstack-infra | 14:00 | |
| openstackgerrit | Merged openstack-infra/system-config: Use the puppet role to copy the puppet code https://review.openstack.org/251581 | 14:00 |
| AJaeger | sdague: while you might be fine with the direction of https://review.openstack.org/#/c/267669/1 I still think the +2 is wrong, the change does *nothing* | 14:00 |
| *** nelsnels_ has quit IRC | 14:00 | |
| *** aopincar has joined #openstack-infra | 14:01 | |
| *** markvoelker_ has joined #openstack-infra | 14:01 | |
| clarkb | mordred though my rechecks have aboht a 50% pass rate so there may be another issue | 14:01 |
| AJaeger | EmilienM: just add noop to the gate, see openstack/tripleo-heat-templates for an example | 14:01 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Add Create Story Button in Worklists https://review.openstack.org/269656 | 14:02 |
| sdague | AJaeger: why ... ? | 14:02 |
| EmilienM | AJaeger: ok, I see thanks | 14:02 |
| *** ari has joined #openstack-infra | 14:02 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper https://review.openstack.org/269284 | 14:02 |
| AJaeger | sdague: see what I posted in the comment, the change has no effect | 14:02 |
| AJaeger | sdague: sorry not time to discuss now further, have to go offline for a bit now. | 14:03 |
| clarkb | mordred we may just need a longer timeout | 14:05 |
| *** tiswanso has joined #openstack-infra | 14:05 | |
| *** markvoelker has quit IRC | 14:05 | |
| *** daemontool has quit IRC | 14:06 | |
| *** daemontool has joined #openstack-infra | 14:07 | |
| *** hashar has joined #openstack-infra | 14:07 | |
| *** sabeen1 has joined #openstack-infra | 14:08 | |
| *** annegentle has joined #openstack-infra | 14:09 | |
| *** doug-fish has joined #openstack-infra | 14:10 | |
| *** akshai has joined #openstack-infra | 14:10 | |
| *** sabeen3 has joined #openstack-infra | 14:11 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys https://review.openstack.org/269660 | 14:11 |
| *** sabeen3 has left #openstack-infra | 14:11 | |
| *** sabeen1 has quit IRC | 14:13 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 14:13 |
| mordred | clarkb: excellent | 14:13 |
| mordred | clarkb: btw, I know you're not awake, but ^^ I think I'm going to be able to land that today :) | 14:13 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Fix some minor project group issues https://review.openstack.org/269627 | 14:14 |
| *** ddieterly has quit IRC | 14:15 | |
| *** peristeri has joined #openstack-infra | 14:15 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Fix width of "no result" and "loading" rows in list tables https://review.openstack.org/269663 | 14:15 |
| *** vivekd has quit IRC | 14:16 | |
| *** vivekd has joined #openstack-infra | 14:16 | |
| *** lucasagomes is now known as lucas-afk | 14:16 | |
| *** ddieterly has joined #openstack-infra | 14:17 | |
| sdague | AJaeger: I'm looking at before and after match criteria, they are different, the important bit I highlighted in a follow on comment | 14:17 |
| *** ddieterly has quit IRC | 14:18 | |
| *** amitgandhinz has joined #openstack-infra | 14:19 | |
| *** Daisy has joined #openstack-infra | 14:20 | |
| openstackgerrit | Merged openstack-infra/shade: allow for updating passwords in keystone v2 https://review.openstack.org/268361 | 14:22 |
| mrmartin | mordred: with this local puppet apply puppetboard.o.o will consume reports of puppet runs? | 14:22 |
| *** jsavak has joined #openstack-infra | 14:23 | |
| *** baoli has joined #openstack-infra | 14:23 | |
| mordred | mrmartin: it will indeed | 14:23 |
| openstackgerrit | Merged openstack-infra/shade: Support neutron subnets without gateway IPs https://review.openstack.org/267405 | 14:24 |
| mordred | mrmartin: we have the code to grab the report files from the hosts after we run puppet on them and upload to the puppetdb server | 14:24 |
| mrmartin | oh ok. | 14:24 |
| *** annegentle has quit IRC | 14:27 | |
| *** baoli_ has joined #openstack-infra | 14:28 | |
| *** daemontool_ has joined #openstack-infra | 14:30 | |
| *** rossella_s has quit IRC | 14:30 | |
| *** [1]Thelo has joined #openstack-infra | 14:30 | |
| *** baoli has quit IRC | 14:31 | |
| *** mriedem_away is now known as mriedem | 14:31 | |
| *** daemontool has quit IRC | 14:31 | |
| *** rossella_s has joined #openstack-infra | 14:31 | |
| *** sabeen1 has joined #openstack-infra | 14:31 | |
| *** tiswanso has quit IRC | 14:31 | |
| *** Thelo has quit IRC | 14:33 | |
| *** [1]Thelo is now known as Thelo | 14:33 | |
| *** tiswanso has joined #openstack-infra | 14:33 | |
| *** acruz has joined #openstack-infra | 14:33 | |
| *** daemontool_ has quit IRC | 14:34 | |
| *** ccrouch has joined #openstack-infra | 14:34 | |
| *** daemontool_ has joined #openstack-infra | 14:34 | |
| *** naggappan has quit IRC | 14:35 | |
| *** arxcruz has quit IRC | 14:35 | |
| *** gordc has quit IRC | 14:36 | |
| *** ccrouch has quit IRC | 14:36 | |
| *** ccrouch has joined #openstack-infra | 14:36 | |
| *** yamamoto has quit IRC | 14:36 | |
| openstackgerrit | Julien Danjou proposed openstack/requirements: Fix voluptuous version https://review.openstack.org/269677 | 14:36 |
| *** daemontool_ has quit IRC | 14:37 | |
| *** daemontool_ has joined #openstack-infra | 14:38 | |
| *** vivekd_ has joined #openstack-infra | 14:39 | |
| *** baoli_ has quit IRC | 14:39 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 14:39 |
| smcginnis | clarkb: When you wake up... :) | 14:39 |
| smcginnis | clarkb: Any resolution of the mirrors not getting updated from last night? | 14:39 |
| smcginnis | clarkb: Trying to get a requirements update through, but tests keep failing due to that. | 14:40 |
| *** fhubik has joined #openstack-infra | 14:41 | |
| *** vivekd has quit IRC | 14:42 | |
| *** vivekd_ is now known as vivekd | 14:42 | |
| *** aopincar has quit IRC | 14:42 | |
| *** davideagnello has joined #openstack-infra | 14:43 | |
| clarkb | smcginnis: I havent done anything mkre than what I posted here | 14:45 |
| clarkb | jhesketh looked and found puppet was disabled on those machines | 14:45 |
| clarkb | I do not know why | 14:45 |
| *** _amrith_ is now known as amrith | 14:46 | |
| *** ddieterly has joined #openstack-infra | 14:47 | |
| *** davideagnello has quit IRC | 14:48 | |
| *** gordc has joined #openstack-infra | 14:48 | |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic https://review.openstack.org/269684 | 14:51 |
| *** notmorgan has quit IRC | 14:51 | |
| *** flepied has quit IRC | 14:51 | |
| *** dizquierdo has joined #openstack-infra | 14:54 | |
| tdurakov | AJaeger, sdague hi, talking about https://review.openstack.org/#/c/267669/ could we merge it now, or changes required? | 14:54 |
| *** eharney has quit IRC | 14:54 | |
| openstackgerrit | Mateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins https://review.openstack.org/269007 | 14:54 |
| *** knikolla has joined #openstack-infra | 14:55 | |
| *** regXboi has joined #openstack-infra | 14:55 | |
| knikolla | Hi guys. I’m currently setting up on Gerrit, but I think I’ve previously done the setup with a different launchpad account which I no longer have (merged it) and reserved my preferred username. So I can no longer select that username now. Would it be possible to free that username up so I can reregister it with a different launchpad account? | 14:57 |
| *** placko_ has joined #openstack-infra | 14:58 | |
| *** akshai has quit IRC | 14:59 | |
| *** watanabe_isao has joined #openstack-infra | 14:59 | |
| *** bryan_att has joined #openstack-infra | 15:00 | |
| *** akshai has joined #openstack-infra | 15:00 | |
| *** notmorgan has joined #openstack-infra | 15:00 | |
| *** notmorgan has quit IRC | 15:00 | |
| *** notmorgan has joined #openstack-infra | 15:00 | |
| *** notmorgan has quit IRC | 15:01 | |
| *** notmorgan has joined #openstack-infra | 15:01 | |
| *** notmorgan has quit IRC | 15:01 | |
| *** notmorgan has joined #openstack-infra | 15:01 | |
| *** baoli has joined #openstack-infra | 15:03 | |
| *** tdurakov has quit IRC | 15:03 | |
| *** Daisy has quit IRC | 15:03 | |
| *** baoli has quit IRC | 15:03 | |
| *** annegentle has joined #openstack-infra | 15:04 | |
| *** watanabe_isao has quit IRC | 15:04 | |
| *** baoli has joined #openstack-infra | 15:04 | |
| *** annegent_ has joined #openstack-infra | 15:04 | |
| *** moravec has joined #openstack-infra | 15:05 | |
| *** tdurakov has joined #openstack-infra | 15:06 | |
| *** jcoufal_ has joined #openstack-infra | 15:06 | |
| *** baoli has quit IRC | 15:07 | |
| clarkb | knikolla: generally no, but it may be possible yo associate new openid to old account. Would that work? | 15:07 |
| *** baoli has joined #openstack-infra | 15:08 | |
| *** annegentle has quit IRC | 15:08 | |
| knikolla | clarkb: Can the new openid assigned be the one I’m trying to set up now with gerrit? | 15:09 |
| *** peristeri has quit IRC | 15:09 | |
| clarkb | yes | 15:09 |
| *** jcoufal has quit IRC | 15:09 | |
| knikolla | clarkb: That would work. What other information would be needed? | 15:10 |
| clarkb | we need to know what the two accounts are and make sure they belong together | 15:10 |
| *** dslev has quit IRC | 15:11 | |
| fungi | Thelo: jhesketh: clarkb: it looks like when i rebooted the hpcloud pypi server a few days ago to more thoroughly clean up its full rootfs, /var/run/bandersnatch never got created and so the cronjob couldn't create the lockfile. as a result it's not been updating for ~3 days. i just made that directory by hand and so it should start syncing again (at least until we finally take down hplcoud in a week | 15:12 |
| fungi | or so) | 15:12 |
| *** daemontool_ has quit IRC | 15:12 | |
| clarkb | fungi becaude puppet isnt running | 15:12 |
| knikolla | clarkb: The account with the username I want to get is “knikolla”. The account I’m currently on is tied to the launchpad account with email “knikolla@bu.edu” and id gerrit id 20163 | 15:12 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Wire up the new --overcloud-pingtest into the tripleo.sh heredoc https://review.openstack.org/262028 | 15:12 |
| *** eharney has joined #openstack-infra | 15:13 | |
| fungi | daemontool_: it sounds like you're trying to use devstack-gate with a nonstandard stable branch name. as far as i know that's not a solved problem unless you want it tested with master of everything else | 15:13 |
| *** jcoufal has joined #openstack-infra | 15:13 | |
| *** tdurakov has quit IRC | 15:13 | |
| *** vivekd has quit IRC | 15:13 | |
| *** tonytan4ever has joined #openstack-infra | 15:14 | |
| fungi | clarkb: yeah, though that also seems sort of chicken-and-eggish to puppet an ephemeral directory in a tmpfs | 15:14 |
| *** zz_dimtruck is now known as dimtruck | 15:14 | |
| *** tdurakov has joined #openstack-infra | 15:14 | |
| fungi | basically leaves things broken until the first puppet run after a reboot. probably not a great pattern for us to be following | 15:14 |
| *** tdurakov has quit IRC | 15:16 | |
| *** jcoufal_ has quit IRC | 15:16 | |
| *** annegent_ is now known as annegentle | 15:16 | |
| *** tdurakov has joined #openstack-infra | 15:17 | |
| *** yfried has joined #openstack-infra | 15:17 | |
| *** peristeri has joined #openstack-infra | 15:18 | |
| * mordred suggests that bandersnatch should create its own directory in /var/run given the nature of such things | 15:19 | |
| anteaya | AJaeger: thank you | 15:19 |
| *** lucas-afk is now known as lucasagomes | 15:19 | |
| clarkb | knikolla: and the old ope id doesnt exist because it was merged into the current lp account? | 15:20 |
| *** jpr2 has quit IRC | 15:20 | |
| fungi | mordred: well, the cron job should anyway, since it's the cron job that wants that directory there (bandersnatch doesn't use it, our flock wrapper does) | 15:21 |
| mordred | ah | 15:22 |
| mordred | yes | 15:22 |
| mordred | I agree that the cron job should ensure the directory, not puppet | 15:22 |
| knikolla | clarkb: I believe so, I’m sure I’ve gone through the gerrit setup before about half-year ago. But since that I’ve done some kung fu with my launchpad accounts merging them because I realized I had 3 different ones. | 15:22 |
| *** MarkAtwood has joined #openstack-infra | 15:23 | |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys https://review.openstack.org/269660 | 15:23 |
| fungi | knikolla: clarkb: keep in mind that merging launchpad accounts doesn't actually merge openids. login.launchpad.net is a separate service from launchpad.net. login.l.n provides a different openid for each e-mail address, while launchpad.net is simply accepting all of them as being for the same lp account | 15:24 |
| *** daemontool_ has joined #openstack-infra | 15:24 | |
| *** rbrndt has joined #openstack-infra | 15:24 | |
| fungi | that is the actual reason for the constant confusion over duplicate gerrit accounts | 15:24 |
| clarkb | yup thatd the issue here | 15:25 |
| fungi | we mistakenly think that launchpad openids are a 1:1 mapping to launchpad accounts, but they're completely separate things | 15:25 |
| openstackgerrit | Merged openstack-infra/ansible-puppet: Properly suppress undefined warning https://review.openstack.org/269620 | 15:25 |
| fungi | which just happen to share a domain name | 15:25 |
| clarkb | so old account is no longer accessible amd new one is created | 15:25 |
| *** annegentle has quit IRC | 15:26 | |
| fungi | well, it's not so much that, if the user has all those e-mail addresses associated with the same lp account, then they'll still authenticate with a different openid to gerrit depending on which e-mail address they provide when logging into login.l.n | 15:26 |
| *** annegentle has joined #openstack-infra | 15:26 | |
| *** salv-orlando has joined #openstack-infra | 15:27 | |
| clarkb | fungi: thats not what we saw with amrith iirc | 15:27 |
| fungi | unless the account merge has been improved to update login.launchpad.net now as well (it didn't used to) | 15:27 |
| *** pradk has joined #openstack-infra | 15:27 | |
| clarkb | the old openid is actaually a 404 | 15:27 |
| fungi | that could be a more recent behavior change then | 15:27 |
| clarkb | so its still brokem just in new and better eays | 15:27 |
| *** flepied has joined #openstack-infra | 15:28 | |
| *** burgerk has quit IRC | 15:28 | |
| knikolla | clarkb: I see. I’m gonna see if I’m able to get access to the old openid, but still I’d like the IDs to be switched, so that I can use the new launchpad account with the knikolla username. | 15:28 |
| *** woodster_ has joined #openstack-infra | 15:29 | |
| fungi | knikolla: what he's going to do is make it so that your new launchpad account authenticates you to your original gerrit account rather than causing creation of a new gerrit account | 15:29 |
| *** achanda has joined #openstack-infra | 15:29 | |
| *** flepied has quit IRC | 15:29 | |
| *** flepied has joined #openstack-infra | 15:30 | |
| knikolla | fungi: Oh, I see. That makes sense. | 15:30 |
| *** angdraug has joined #openstack-infra | 15:30 | |
| *** sdake has quit IRC | 15:31 | |
| knikolla | kungi: clarkb: I need to run now to a meeting. Thanks for all the help. | 15:31 |
| *** knikolla has quit IRC | 15:31 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:31 | |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Display Assignees on Cards and Worklists https://review.openstack.org/269707 | 15:32 |
| *** akshai has quit IRC | 15:32 | |
| smcginnis | clarkb: Thanks for the update on the mirroring. | 15:32 |
| smcginnis | clarkb: Is there a plan to resolve that, or is there a way we can get Jenkins to avoid the non-mirrored repos so tests don't fail? | 15:32 |
| *** ajmiller has joined #openstack-infra | 15:33 | |
| clarkb | smcginnis: fungi has resolved it | 15:33 |
| fungi | smcginnis: clarkb: what repos? | 15:33 |
| smcginnis | clarkb: Oh, excellent. Thanks! And thanks fungi! | 15:34 |
| clarkb | fungi: it was the pypi mirror being out of sync | 15:34 |
| openstackgerrit | Giulio Fidente proposed openstack-infra/tripleo-ci: We don't have tripleo.sh --flavors anymore https://review.openstack.org/269710 | 15:34 |
| clarkb | fungi: so was missing some newer releases | 15:34 |
| fungi | oh, smcginnis said something about mirroring repos and avoiding broken repo mirrors | 15:34 |
| smcginnis | clarkb: So no real reason that particular mirror was disabled? | 15:34 |
| fungi | wasn't sure what "avoid the non-mirrored repos" meant | 15:34 |
| clarkb | smcginnis: it was not disabled, it was just behind | 15:34 |
| smcginnis | clarkb: Oh, I misunderstood the earlier statements then. Good to hear. | 15:35 |
| fungi | smcginnis: it hadn't pulled the latest release of os-brick since it was running a three-day-old snapshot of pypi basically | 15:35 |
| Thelo | fungi: Great thanks, so it should be fine now? ( the repo?) | 15:35 |
| clarkb | fungi: https://login.launchpad.net/+id/McTKxdP is the old openid and does 404 | 15:35 |
| fungi | clarkb: that's an improvement in that case. makes things a little simpler for us now | 15:35 |
| smcginnis | fungi: There was a comment earlier that that particular mirror was disabled in some ansible scripts. | 15:36 |
| fungi | Thelo: probably? i haven't checked to see if it's serving the latest os-brick release there yet | 15:36 |
| smcginnis | fungi: It didn't sound like (at the time) there was a desire to reenable, so I was just trying to understand how to proceed. | 15:36 |
| smcginnis | fungi: Thanks for your help. I guess we should be good now. | 15:36 |
| anteaya | oh I'll just toss this out there but it would be great if sandbox could allow all registered users to really +2 +A patches rather than just telling us they have those permissions | 15:37 |
| *** hashar has quit IRC | 15:37 | |
| fungi | smcginnis: well, at this point the server and its public cloud provider are going away entirely in a little over a week, so there's not a lot of incentive to remember who disabled puppet for that server and why | 15:37 |
| smcginnis | fungi: True! ;) | 15:37 |
| anteaya | someone is teaching a class tonight and actually using sandbox as intended (woooo) would be great if they can +2 and approve patches | 15:37 |
| openstackgerrit | Vasyl Saienko proposed openstack-infra/project-config: Add networking-generic-switch project https://review.openstack.org/267664 | 15:37 |
| fungi | anteaya: is there an acl update proposed for that? i take it our documentation contradicts the current acl? | 15:38 |
| anteaya | well the permissions are there | 15:38 |
| clarkb | now I just need to find the launchpad/ubuntu one account to see if I can check the email addresses match up there but maybe that is overly paranoid | 15:38 |
| anteaya | it is just in a actual patch they aren't | 15:38 |
| anteaya | let me get links | 15:39 |
| openstackgerrit | Merged openstack-infra/project-config: Magnum: enable mesos voting https://review.openstack.org/269440 | 15:39 |
| anteaya | https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 15:39 |
| anteaya | https://review.openstack.org/#/c/269387/ | 15:39 |
| anteaya | so right now it seems to me that gerrit is contradicting itself | 15:40 |
| *** achanda has quit IRC | 15:40 | |
| clarkb | right my theory was gerrit does something to prevent registered users from working in that manner | 15:40 |
| clarkb | with a potential workaround of making a new group and adding registered users to that group | 15:40 |
| *** tdurakov has quit IRC | 15:40 | |
| anteaya | I see +1, 0, -1 when I try to review that patch | 15:40 |
| *** tdurakov has joined #openstack-infra | 15:41 | |
| anteaya | when signed in | 15:41 |
| anteaya | clarkb: yup, so far that approach makes sense to me | 15:41 |
| *** tdurakov has quit IRC | 15:43 | |
| *** _nadya_ has quit IRC | 15:43 | |
| *** otter768 has joined #openstack-infra | 15:43 | |
| *** tdurakov has joined #openstack-infra | 15:43 | |
| *** yamahata has joined #openstack-infra | 15:44 | |
| *** akshai has joined #openstack-infra | 15:45 | |
| *** mtanino has joined #openstack-infra | 15:46 | |
| clarkb | fungi: want to sanity check http://paste.openstack.org/show/484297/ I am still half asleep but I think that is what is necessary | 15:47 |
| *** otter768 has quit IRC | 15:47 | |
| clarkb | anteaya: let me set up a test on review-dev really quickly | 15:50 |
| anteaya | clarkb: thank you | 15:50 |
| anteaya | clarkb: I appreciate your help even though you really should be asleep | 15:50 |
| *** eharney has quit IRC | 15:50 | |
| *** ddieterly has quit IRC | 15:50 | |
| anteaya | I just didn't want this item to be lost | 15:50 |
| *** markvoelker_ has quit IRC | 15:51 | |
| *** angdraug has quit IRC | 15:52 | |
| clarkb | anteaya: what can you vote verified on https://review-dev.openstack.org/5399 ? | 15:53 |
| clarkb | registered users should be able to -2 to +2 | 15:53 |
| clarkb | then if that fails as expected we can try the new group thing | 15:53 |
| *** hdd has joined #openstack-infra | 15:54 | |
| *** salv-orlando has quit IRC | 15:54 | |
| fungi | clarkb: i would just do `update account_external_ids set account_id=16465 where external_id="https://login.launchpad.net/+id/McTKxdP";` and then deactivate the new account | 15:57 |
| anteaya | clarkb: code review is still only -1, 0, +2 but I just voted verified +2 on that patch | 15:57 |
| *** burgerk has joined #openstack-infra | 15:57 | |
| anteaya | clarkb: but I still don't have workflow +1 available to me | 15:58 |
| clarkb | anteaya: ok then I think that means registered users can +2 | 15:58 |
| clarkb | so something else must be happening | 15:58 |
| clarkb | fungi: oh derp that is easier | 15:58 |
| *** bradjones has joined #openstack-infra | 15:58 | |
| *** bradjones has quit IRC | 15:58 | |
| *** bradjones has joined #openstack-infra | 15:58 | |
| anteaya | well registered users can vote +2 verified on sandbox in review-dev with that setup | 15:59 |
| krotscheck | fungi: I've got +2 on the system-config whele patches. Any chance you can roll through and take a look at them today-ish? https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open+project:openstack-infra/system-config | 15:59 |
| *** achanda has joined #openstack-infra | 15:59 | |
| clarkb | anteaya: yes which refutes the theory that registered users can't +2 things due to a gerrit bug | 15:59 |
| anteaya | but I wonder if we offer that as a workaround we actually are creating a different problem by confusing our users | 15:59 |
| anteaya | clarkb: I still can't +2 code review | 15:59 |
| anteaya | clarkb: nor can I +1 workflow | 15:59 |
| clarkb | anteaya: I used verified instead of code review because there is no code review +2 on review-dev | 15:59 |
| anteaya | clarkb: ah I see now, sorry I was confused | 16:00 |
| mrmartin | clarkb, fungi: ask-staging.openstack.org is alive again. I've enabled the openstackid auth provider, but I need a little help to roll this out into production during the next week if everything goes well. | 16:00 |
| *** Ramanjaneya has joined #openstack-infra | 16:00 | |
| *** sdake has joined #openstack-infra | 16:01 | |
| *** baoli has quit IRC | 16:01 | |
| *** lucasagomes is now known as lucas-brb | 16:01 | |
| *** eharney has joined #openstack-infra | 16:02 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper https://review.openstack.org/269284 | 16:02 |
| *** boris-42 has joined #openstack-infra | 16:03 | |
| *** achanda has quit IRC | 16:03 | |
| *** jsavak has quit IRC | 16:03 | |
| openstackgerrit | Devananda van der Veen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin https://review.openstack.org/265311 | 16:04 |
| devananda | jroll: ^ | 16:04 |
| *** zeih has quit IRC | 16:04 | |
| *** ddieterly has joined #openstack-infra | 16:06 | |
| fungi | krotscheck: my tuesdays are afflicted with meetingitis, so what little space there is on my dance card is filling up very quickly. i'll see if i can skim those after the infra team meeting while i'm lurking the tc meeting | 16:06 |
| *** hdd has quit IRC | 16:06 | |
| krotscheck | fungi: Thanks. I'll see if I can solicit an infra-core to help me land them and provision the instances before moving on to adding the gate jobs. | 16:07 |
| *** Guest17173 has quit IRC | 16:07 | |
| fungi | mrmartin: exciting! unfortunately i'm not around much at all next week since that's when we're having our quarterly foundation staff meetings/get-together | 16:07 |
| krotscheck | fungi: I just want to make sure you're aware of what we're doing in case the world goes splodey | 16:07 |
| fungi | krotscheck: much appreciated! | 16:08 |
| jroll | devananda: just one comment - also, we'll need to get the +A on that before we land the plugin things, so they land together | 16:08 |
| *** nmagnezi has quit IRC | 16:09 | |
| sdague | clarkb: I think something is wrong with the common flags code and internap | 16:09 |
| sdague | http://logs.openstack.org/34/254434/10/gate/gate-grenade-dsvm-multinode/fda9fbe//logs/old/screen-n-cpu.txt.gz#_2016-01-19_15_50_50_269 | 16:09 |
| openstackgerrit | Angus Lees proposed openstack-infra/zuul: Explicitly set basepython=python2.7 https://review.openstack.org/269731 | 16:09 |
| devananda | jroll: I'd love to test the plugin thing before landing it, honestly | 16:09 |
| *** Qiming has quit IRC | 16:09 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use /opt not /etc as the system-config location https://review.openstack.org/269732 | 16:09 |
| mrmartin | fungi: ok, I no problem, as other infra cores have access to ask.o.o we can do it without you. the askbot upstream repo changes really broke the staging, I hope it won't happen with production, I'll prepare every documentation required for a seamless upgrade. | 16:09 |
| *** bpokorny has quit IRC | 16:10 | |
| clarkb | sdague: we may have to trim the flag list then (I thought it was fairly minimal though) | 16:10 |
| sdague | clarkb: it's not dynamically generated? | 16:10 |
| jroll | devananda: yeah, I agree, but it can be tested locally you know :) | 16:11 |
| clarkb | sdague: no I think its basically a standard libvirt cpu type less a couple flags but I would have to go reread the code | 16:11 |
| *** _nadya_ has joined #openstack-infra | 16:11 | |
| jroll | devananda: the downside to testing it before landing is we'll be running each test twice (once from tempest, once from ironic tree), until we re-update the regex | 16:11 |
| *** markus_z has quit IRC | 16:11 | |
| *** moravec has quit IRC | 16:11 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use /opt not /etc as the system-config location https://review.openstack.org/269732 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 16:12 |
| mordred | fungi, clarkb: real quick - had an oops ^^ | 16:12 |
| clarkb | sdague: # Create a gate64 cpu model that is core2duo less monitor and pse36 | 16:12 |
| fungi | mordred: which one--both? | 16:12 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 16:12 |
| clarkb | which includes vme and ssse3 | 16:12 |
| devananda | jroll: indeed. anyway, I'm around all week and will keep an eye on moving this (and hte neutron work) forward | 16:13 |
| mwhahaha | is there a way to tell which DC a job ran in? We're running into some issues where some jobs are taking excessively long to download source which is causing failures on some nodes but not others | 16:13 |
| *** wei-wang has quit IRC | 16:14 | |
| clarkb | mwhahaha: it is encoded in the test node hostname | 16:14 |
| *** moravec has joined #openstack-infra | 16:14 | |
| mwhahaha | ah so it looks like the hpcloud nodes | 16:14 |
| jroll | devananda: awesome, ty | 16:14 |
| fungi | also try to avoid downloading things in jobs. we cache a lot of source on the workers already--what source are you downloading? | 16:14 |
| * mgagne feels better now | 16:14 | |
| sdague | clarkb: ok, you up for proposing that? Because it's in devstack, we're going to have to update every branch | 16:14 |
| mwhahaha | the puppet modules for unit tests | 16:14 |
| clarkb | sdague: sure | 16:14 |
| mordred | fungi: no - just 269732 | 16:15 |
| sdague | that's the cause of some chunk of the gate backlog | 16:15 |
| fungi | mwhahaha: oh, so not openstack puppet modules, just other random dependencies? | 16:15 |
| *** fhubik is now known as fhubik_brb | 16:15 | |
| *** fhubik_brb is now known as fhubik | 16:15 | |
| sdague | mriedem: any idea if we have a bug for this yet? | 16:15 |
| mwhahaha | for these jobs yea | 16:15 |
| *** Liuqing has joined #openstack-infra | 16:16 | |
| *** zhurong has joined #openstack-infra | 16:16 | |
| mwhahaha | it looks like there might be some overloaded nodes in the hpcloud because some are OK and some are not, https://review.openstack.org/#/c/262587/ the gate-fuel-library-puppet-unit-3.{7,8}-dsvm-centos7 took an hour and timed out while the others were OK in < 10 mins | 16:16 |
| *** gongysh has joined #openstack-infra | 16:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 16:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Don't rsync puppet on localhost https://review.openstack.org/269736 | 16:16 |
| *** jsavak has joined #openstack-infra | 16:16 | |
| mordred | fungi, clarkb: ALSO - 269736 if you get a chance | 16:16 |
| *** jpr has joined #openstack-infra | 16:18 | |
| clarkb | just reading up on cpu flags we may actually want vme, but we can figure that out later | 16:18 |
| openstackgerrit | Sean Dague proposed openstack-infra/elastic-recheck: add signature for internap cpu model issue https://review.openstack.org/269738 | 16:18 |
| clarkb | its all emulated anyways so any performance benefit is likely to be tiny | 16:19 |
| sdague | clarkb: yeh, but internap doesn't have vme | 16:19 |
| sdague | so if that's in our cloud, we can't have it | 16:19 |
| sdague | clarkb: ideally it would be nice for the gate64 model to be defined in devstack-gate, passed to devstack some how, and let devstack change this. But we can do that once things are fixed. | 16:20 |
| *** moravec has quit IRC | 16:20 | |
| clarkb | sdague: I was thinking maybe we provide feedback to them that they should have this | 16:21 |
| clarkb | but I need to do more research before I care that strongly | 16:21 |
| sdague | clarkb: sure, that's fine | 16:22 |
| clarkb | (it apperas that it does allow for more performant interrupts) | 16:22 |
| mriedem | sdague: for what? | 16:22 |
| sdague | when did internap get added to the mix? | 16:22 |
| *** jordanP has quit IRC | 16:22 | |
| clarkb | sdague: first week of janaury | 16:22 |
| mriedem | oh http://logs.openstack.org/34/254434/10/gate/gate-grenade-dsvm-multinode/fda9fbe//logs/old/screen-n-cpu.txt.gz#_2016-01-19_15_50_50_269 | 16:22 |
| sdague | mriedem: yeh that | 16:23 |
| mriedem | sdague: i haven't seen that one | 16:23 |
| mriedem | i can report one quick | 16:23 |
| sdague | mriedem: I already did | 16:23 |
| sdague | https://review.openstack.org/#/c/269738/ | 16:23 |
| clarkb | oh was there a bug I should've used? | 16:23 |
| clarkb | I can update commit messages | 16:23 |
| sdague | clarkb: sure | 16:23 |
| *** fawadkhaliq has joined #openstack-infra | 16:24 | |
| *** egonzalez has joined #openstack-infra | 16:25 | |
| mtreinish | sdague: fwiw: http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj01 | 16:25 |
| anteaya | for me newton will be named in honour of fig newtons | 16:26 |
| anteaya | and I need to greive the loss of Om as a release name for a bit first | 16:26 |
| sdague | mtreinish: I guess internap can't pass on multinode tests | 16:27 |
| sdague | that's where we specify the kvm arch | 16:27 |
| *** Jeffrey4l has joined #openstack-infra | 16:27 | |
| mordred | anteaya: I know - I was hoping for Om myself - but apparently it's a popular enough name that many other non-openstack people got to it before we did | 16:27 |
| clarkb | fungi: http://paste.openstack.org/show/484307/ that look better? | 16:28 |
| mtreinish | sdague: yeah, that's what it looks like | 16:28 |
| *** lucas-brb is now known as lucasagomes | 16:28 | |
| anteaya | mordred: ah I can see that | 16:28 |
| anteaya | I do like fig newtons though | 16:28 |
| sdague | mtreinish: how does your os health handle multinode jobs | 16:28 |
| fungi | clarkb: yep | 16:28 |
| sdague | does it register against both providers? | 16:28 |
| *** Jeffrey4l has quit IRC | 16:28 | |
| sdague | or only the master | 16:28 |
| *** Jeffrey4l has joined #openstack-infra | 16:29 | |
| clarkb | sdague: multinode is always within a single region | 16:30 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: add signature for internap cpu model issue https://review.openstack.org/269738 | 16:30 |
| sdague | clarkb: oh, when did that change? | 16:30 |
| clarkb | sdague: it has always been the case | 16:30 |
| fungi | clarkb: is that the same account as http://lists.openstack.org/pipermail/openstack-dev/2016-January/084382.html is about? | 16:30 |
| sdague | hmmmm... really? | 16:30 |
| clarkb | sdague: yes | 16:30 |
| mordred | zaro, pleia2: when you get in - I updated https://review.openstack.org/249492 based on your reviews and would love follow up reviews | 16:30 |
| fungi | sdague: yeah, that was an original design constraint of the feature | 16:31 |
| clarkb | fungi: no I don't think so, different email address at least | 16:31 |
| mordred | zaro, pleia2: also, I believe we can roll that live today | 16:31 |
| fungi | sdague: same provider, region and az | 16:31 |
| clarkb | fungi: it may not be az I think due to how we have azs in hpcloud | 16:31 |
| clarkb | but once hpcloud goes away that will be true | 16:31 |
| clarkb | I can't remember if we allow the az to float or not | 16:31 |
| *** kzaitsev_mb has joined #openstack-infra | 16:32 | |
| sdague | ok, I thought I remember being told that wasn't going to be constrained, which is why we built all the cpu flags stuff | 16:32 |
| fungi | i thought multinode in hpcloud was tied to all subnodes being in the same neutron network (because we defined them as providers in nodepool) and then each of those neutron networks went to one and only one az? | 16:32 |
| sdague | because live migrating from rax -> hp triggered weird flag cross overs | 16:33 |
| anteaya | mtreinish: so this is the job failure rate: http://status.openstack.org/openstack-health/#/g/node_provider/internap-nyj01 so this is tracking the host's ability to build the node and provision for the job? | 16:33 |
| fungi | sdague: i thought it was that rackspace had inconsistent cpu between instances that we ran into that issue | 16:34 |
| anteaya | mtreinish: it doesn't have anything to do with if the actual tests failed or succeeded? | 16:34 |
| SamYaple | /w/win 1 | 16:34 |
| mordred | fungi: yes - I believe that is true- it is tied to az as a happenstance of how we neutron in hp | 16:34 |
| mtreinish | anteaya: right now it's only counting if the tests failed | 16:34 |
| sdague | fungi: the overlap game I remember playing was hp -> rax | 16:34 |
| mordred | fungi: not because the code itself is az aware | 16:34 |
| mtreinish | if devstack-gate or devstack fails, it's not counted | 16:34 |
| anteaya | mtreinish: oh okay | 16:34 |
| *** sridhar_ram has joined #openstack-infra | 16:35 | |
| sdague | if this is all in one region now, and the same flavor, my guess is we could probably get rid of this entirely | 16:35 |
| fungi | sdague: interesting. i don't recall us ever having the ability to try that (other than setting it up manually) | 16:35 |
| clarkb | sdague: the cpu flag stuff is necessary because within a region we can have vastly different cpu flags | 16:35 |
| anteaya | okay yes, I was aware that was how openstack-health operates I just wondered if it was different for the part where you are tracking providers, thanks for the clarification | 16:35 |
| sdague | clarkb: for the same flavor? | 16:35 |
| clarkb | sdague: yes | 16:35 |
| clarkb | sdague: so the issue wasn't rax vs hp so much as hp vs hp and rax vs rax | 16:35 |
| *** baoli has joined #openstack-infra | 16:35 | |
| sdague | on just rax? | 16:35 |
| *** markvoelker has joined #openstack-infra | 16:36 | |
| sdague | on hp the definitely smoothed it all out | 16:36 |
| clarkb | sdague: yes I think hp may have consistent flags but rax didn't | 16:36 |
| mtreinish | anteaya: see: http://blog.kortar.org/?p=279 the "Failures Before Test Runs aren’t counted" section (I really need to add heading links) | 16:36 |
| mordred | cloud providers not so much with using exactly the same hardware on their hypervisors | 16:36 |
| *** moravec has joined #openstack-infra | 16:36 | |
| *** flepied1 has joined #openstack-infra | 16:36 | |
| mtreinish | anteaya: we do have a way forward on that, so it'll be fixed before too long | 16:36 |
| sdague | mordred: that surprises me at the flavor level | 16:36 |
| *** flepied has quit IRC | 16:36 | |
| mordred | sdague: many things surprise me at all the levels | 16:36 |
| clarkb | sdague: so the idea was find subset that worked for rax and hp not because we mixed rax and hp but because we run jobs on all those cpu flags | 16:36 |
| mordred | sdague: but I've grown to expect the most unexpected and worst things from clouds | 16:37 |
| sdague | right, now that we are mixing all the world, I think we've got to reset expectations. Possibly with a dedicated arch per provider. | 16:37 |
| clarkb | fungi: ok that account is all munged inthe db now | 16:37 |
| anteaya | mtreinish: now I understand, thank you | 16:37 |
| *** salv-orlando has joined #openstack-infra | 16:37 | |
| sdague | because slicing out all the flags for lowest common denominator across all clouds is going to potentially sacrifice performance we want in places where we can have it | 16:37 |
| mordred | would it be useful to do a cpu flag detection at node creation and storing that somewhere that we can read it when setting up the flags for devstack? | 16:38 |
| *** bpokorny has joined #openstack-infra | 16:38 | |
| clarkb | mordred: well its already done for us by the kernel | 16:38 |
| clarkb | the tricky bit is you have to install the flags during the devstack run because libvirt | 16:38 |
| mordred | right | 16:38 |
| mordred | that's why I was saying at node creation | 16:38 |
| clarkb | so we could update devstack to accept a list and do the right thing rather than just being hard coded | 16:38 |
| clarkb | mordred: node creation is too early | 16:38 |
| mordred | or, multi-node association time | 16:39 |
| nibalizer | good morning | 16:39 |
| mordred | rather | 16:39 |
| *** bpokorny has quit IRC | 16:39 | |
| mordred | the first time we, as nodepool, know that the two nodes are going to be associated with each other | 16:39 |
| clarkb | mordred: ya then | 16:39 |
| *** abregman has quit IRC | 16:39 | |
| mordred | we cna communicate to both of them what the cpu flag set on each is | 16:39 |
| anteaya | mtreinish: you only need one more devstack core reviewer +2 for this to go in, well done: https://review.openstack.org/#/c/234232/ | 16:39 |
| *** egonzalez has quit IRC | 16:39 | |
| anteaya | good morning nibalizer | 16:39 |
| mordred | but then the logic for finding the right flags | 16:39 |
| mordred | should likely live in devstack | 16:39 |
| *** bpokorny has joined #openstack-infra | 16:39 | |
| mordred | "hey, devstack, I've got these lists of cpu flag lists" | 16:40 |
| mtreinish | anteaya: I actually think I caught a bug in that last night, so I might need to respin it | 16:40 |
| clarkb | I mean this would be a trivial fix if nova flavors enforced this >_> | 16:40 |
| mordred | cause I agree with sdague - doing this as the LCD for all of our cloud is going to be continually unfun | 16:40 |
| mordred | but mapping discreet combinations is likely less suck | 16:40 |
| clarkb | mordred: yup, that said the cloud should solve this | 16:40 |
| mordred | clarkb: totes | 16:41 |
| clarkb | so this would still be a bandaid | 16:41 |
| sdague | anyway, clarkb's fixes should be promoted | 16:41 |
| fungi | also this introduces yet another possible corner case for differentiation that can impact jobs. it might be a detail we want to add to logstash for the job info too? | 16:41 |
| sdague | 2 current gate resets are that issue | 16:41 |
| anteaya | mtreinish: nice catch | 16:41 |
| mordred | nibalizer: when you wake up ... today is the day - all of the pre-reqs are in and working | 16:42 |
| openstackgerrit | Devananda van der Veen proposed openstack-infra/project-config: Allow ironic jobs to run with tempest plugin https://review.openstack.org/265311 | 16:42 |
| sdague | or internap turned off until this settles out and those get in | 16:42 |
| sdague | or blacklist it from multinode jobs | 16:44 |
| openstackgerrit | Sam Betts proposed openstack-infra/project-config: Add a tinyIPA non-voting test to Ironic https://review.openstack.org/269684 | 16:45 |
| mtreinish | anteaya: well when I looked at: https://review.openstack.org/234447 the subunit stream had the devstack stream had the devstack "test" listed twice | 16:45 |
| jeblair | mordred, clarkb: yeah, probably not at node association time though -- i think d-g should grab the info and either do the intersection calculation or pass it to devstack to do it. | 16:45 |
| openstackgerrit | Devananda van der Veen proposed openstack-infra/project-config: Make the ironic-parallel job experimental https://review.openstack.org/269757 | 16:45 |
| *** annegentle has quit IRC | 16:46 | |
| anteaya | mtreinish: ah | 16:46 |
| *** Liuqing has quit IRC | 16:46 | |
| clarkb | jeblair: I think passing the intersection to each devstack run is probably sanest | 16:46 |
| *** moravec has quit IRC | 16:46 | |
| mordred | clarkb, jeblair: ++ | 16:47 |
| anteaya | mtreinish: quite the prompt you have there | 16:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Don't use internap for 2-node labels https://review.openstack.org/269759 | 16:47 |
| jeblair | sdague: ^ | 16:47 |
| mtreinish | anteaya: ? | 16:47 |
| *** samuelBartel has quit IRC | 16:48 | |
| anteaya | mtreinish: http://paste.openstack.org/show/476207/ | 16:48 |
| jeblair | clarkb: where are your in-flight fixes? | 16:48 |
| anteaya | jeblair: https://review.openstack.org/#/q/topic:bug/1535799 | 16:48 |
| clarkb | jeblair: 269739 269742 269743 | 16:48 |
| jeblair | oh, i didn't add query support for the gertty link thing | 16:49 |
| sdague | jeblair: +A | 16:49 |
| mtreinish | anteaya: hah, I didn't remember that I posted a paste with the same issue back in Oct | 16:49 |
| *** _nadya_ has quit IRC | 16:49 | |
| anteaya | mtreinish: :) well you are consistent in that you found it again | 16:50 |
| *** annegentle has joined #openstack-infra | 16:50 | |
| *** moravec has joined #openstack-infra | 16:50 | |
| mtreinish | anteaya: and the prompt is even better on my terminal, it's got colors | 16:50 |
| anteaya | ha ha ha | 16:50 |
| anteaya | too bad paste can't share that | 16:50 |
| clarkb | the things to keep in mind for this is you need to make an intersection with a complete view of the nodes you have running right now. You then have to map those flags onto the flags libvirt understands (this was mostly 1:1 when I checked a year ago). You have to write the mapping file out after libvirt is installed, then do a service stop start in order for them to be properly loaded | 16:50 |
| *** Liuqing has joined #openstack-infra | 16:50 | |
| *** placko_ has quit IRC | 16:51 | |
| mtreinish | anteaya: http://i.imgur.com/YvGLkSC.png | 16:51 |
| clarkb | for these reasons it was simplest at the time to just make a hardcoded sane cpu type and have devstack install it itself | 16:51 |
| anteaya | mtreinish: aha ha haha | 16:51 |
| *** nelsnelson has quit IRC | 16:51 | |
| *** salv-orlando has quit IRC | 16:51 | |
| *** Liuqing has quit IRC | 16:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/project-config: Revert "Don't use internap for 2-node labels" https://review.openstack.org/269763 | 16:51 |
| *** yamahata has quit IRC | 16:51 | |
| jeblair | clarkb, sdague: ^ revert depends on clarkb's internap fixes | 16:52 |
| *** nelsnelson has joined #openstack-infra | 16:52 | |
| *** gyee has joined #openstack-infra | 16:53 | |
| *** Liuqing has joined #openstack-infra | 16:53 | |
| *** Liuqing has quit IRC | 16:53 | |
| *** rcernin has quit IRC | 16:53 | |
| sdague | jeblair: thanks | 16:54 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Display Assignees on Cards and Worklists https://review.openstack.org/269707 | 16:54 |
| *** apoorvad has joined #openstack-infra | 16:54 | |
| *** matrohon has quit IRC | 16:55 | |
| *** annegentle has quit IRC | 16:55 | |
| clarkb | fungi: it would be nice (though maybe it breaks the protocol) if openids could be redirected from one to another in the case of an account merge | 16:56 |
| clarkb | fungi: and that permanent redirect would indicate to the consumer side to use the new url | 16:56 |
| *** ashleighfarnham has joined #openstack-infra | 16:57 | |
| anteaya | zaro: when you are about, review.o.o sandbox is not allowing all registered users to +2 and +A patches even though it reports all registered users have permission to do so | 16:57 |
| *** scheuran has quit IRC | 16:57 | |
| fungi | clarkb: well, this has got me thinking about how to go about our transition to openstackid.org for gerrit authentication | 16:58 |
| anteaya | zaro: it would be great if we can figure out how to give +2 +A to all registered users on sandbox | 16:58 |
| *** ajmiller has quit IRC | 16:58 | |
| *** david-lyle has quit IRC | 16:58 | |
| *** ajmiller has joined #openstack-infra | 16:58 | |
| *** dslevin has joined #openstack-infra | 16:59 | |
| *** gongysh has quit IRC | 16:59 | |
| *** sshnaidm has quit IRC | 16:59 | |
| fungi | clarkb: if we get to the point where we can make it optional to log in with either login.launchpad.net or openstackid.org, and if we can get openstackid to support id lookups by e-mail address, then i can batch add all the openstackid.org ids to gerrit's database for any existing accounts based on a search for all their known e-mail addresses | 17:00 |
| *** sshnaidm has joined #openstack-infra | 17:00 | |
| *** jsavak has quit IRC | 17:00 | |
| jeblair | infra-root: https://threatpost.com/serious-linux-kernel-vulnerability-patched/115923/ is a kernel local privilege escalation that it scheduled to get patched today: CVE-2016-0728 | 17:00 |
| *** jsavak has joined #openstack-infra | 17:00 | |
| *** claudiub has joined #openstack-infra | 17:00 | |
| fungi | jeblair: oh, yep, i saw that come through oss-security and hadn't looked closely enough at it yet. thanks! | 17:00 |
| jeblair | there's apparently poc code for it out there... which is lovely. | 17:01 |
| *** e0ne has quit IRC | 17:01 | |
| fungi | jeblair: on that note, care to weigh back in on 267778 now that it works? | 17:01 |
| fungi | and i guess we'll want to move over to the incident channel when it comes time to do some reboots | 17:02 |
| *** apoorvad has quit IRC | 17:02 | |
| *** salv-orlando has joined #openstack-infra | 17:03 | |
| jeblair | fungi: any reason not to put the class apt purge into the conditional? | 17:03 |
| *** MarkAtwood1 has joined #openstack-infra | 17:03 | |
| fungi | jeblair: because it's replacing include apt | 17:04 |
| jeblair | fungi: ah, and you want apt in there regardless | 17:04 |
| jeblair | gotcha | 17:04 |
| fungi | i mean, i could do an else with the old include apt if we really want | 17:04 |
| *** sdake has quit IRC | 17:04 | |
| *** sdake_ has joined #openstack-infra | 17:04 | |
| jeblair | fungi: no prob, approved. just felt like i was missing something (and i was) | 17:04 |
| jeblair | clarkb: what should i look at in nodepool-land? | 17:05 |
| jeblair | (before the storm hits) | 17:05 |
| jeblair | i found https://review.openstack.org/268867 | 17:05 |
| *** david-lyle has joined #openstack-infra | 17:06 | |
| *** mrmartin has quit IRC | 17:06 | |
| *** ddieterly has quit IRC | 17:06 | |
| *** jlanoux has quit IRC | 17:07 | |
| *** flwang has joined #openstack-infra | 17:08 | |
| *** mrmartin has joined #openstack-infra | 17:08 | |
| *** ccrouch has quit IRC | 17:08 | |
| *** ddieterly has joined #openstack-infra | 17:08 | |
| clarkb | jeblair: there is that one, ianw's stack at https://review.openstack.org/269266 and https://review.openstack.org/267865 fixes logging | 17:08 |
| *** ddieterly has quit IRC | 17:09 | |
| clarkb | jeblair: I got tests to rerun against them way too early this morning and I think they look good and can start approving them | 17:09 |
| *** rcernin has joined #openstack-infra | 17:09 | |
| *** yamahata has joined #openstack-infra | 17:09 | |
| *** dtantsur is now known as dtantsur|afk | 17:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Don't use internap for 2-node labels https://review.openstack.org/269759 | 17:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Also handle work exceptions in builders https://review.openstack.org/269774 | 17:10 |
| jeblair | clarkb, greghaynes: ^ greghaynes change to add the failure handler lgtm with that addition ^ | 17:10 |
| clarkb | oh there is also ianw's fixes for the dib elements | 17:11 |
| clarkb | let me find that one | 17:11 |
| *** hdd has joined #openstack-infra | 17:11 | |
| clarkb | oh sweet it merged already | 17:11 |
| openstackgerrit | Merged openstack-infra/system-config: Puppet the Ubuntu sources.list files on servers https://review.openstack.org/267778 | 17:11 |
| pleia2 | mordred: yay docs, looking now | 17:12 |
| *** moravec has quit IRC | 17:12 | |
| mordred | pleia2: I apparently have an unhappy docs job | 17:12 |
| jeblair | clarkb: does "from nodepool import nodepool; from nodepool import builder;" read weird to you? like where is builder coming from? nodepool or nodepool.nodepool? :) | 17:13 |
| pleia2 | mordred: I have a comment inline too, posting now | 17:13 |
| jeblair | clarkb: i guess it's doing the right thing.... but i think i might use the slightly more clear but verbose forms | 17:14 |
| clarkb | jeblair: I hadn't thought of it that way :) though when I worked around the statsd thing I did use nodepool.builder to be explicit | 17:14 |
| mordred | pleia2: woot | 17:14 |
| *** asselin has quit IRC | 17:14 | |
| pleia2 | mordred: nit-ish but easier on the eyes link-wise | 17:14 |
| mordred | pleia2: oh - really? | 17:14 |
| jeblair | well, i better check the dsvm test to make sure it is doing the right thing | 17:14 |
| mordred | pleia2: do I need to do any path things? | 17:14 |
| *** prometheanfire has joined #openstack-infra | 17:14 | |
| pleia2 | mordred: should work with magic | 17:14 |
| prometheanfire | gate stuck or backed up? | 17:15 |
| *** Daisy has joined #openstack-infra | 17:15 | |
| jeblair | prometheanfire: what makes you ask? | 17:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 17:15 |
| nibalizer | mordred: today is a pooey day for me but I can try to help | 17:15 |
| prometheanfire | 265662 has been waiting over 5 hours | 17:15 |
| prometheanfire | just curious | 17:15 |
| nibalizer | I'll be around more in the afternoon | 17:15 |
| mordred | pleia2: neat. I've learned something today | 17:16 |
| *** ifarkas has quit IRC | 17:16 | |
| mordred | nibalizer: cool. I'm thinking of rolling it live after the meeting | 17:16 |
| *** ddieterly has joined #openstack-infra | 17:16 | |
| nibalizer | mordred: ya I have an internal meeting at 11 then a tour thing at 12 and then I'm available | 17:17 |
| nibalizer | this means my participation in the infra meeting will be limited to nonexistant :( | 17:17 |
| pleia2 | mordred: the rendering here is an example of how it works: http://docs.openstack.org/infra/system-config/sysadmin.html#adding-a-new-server | 17:17 |
| mordred | nibalizer: well, luckily, so far it all seems to be working, so it should be very little needed | 17:17 |
| jeblair | prometheanfire: according to http://status.openstack.org/zuul/ the top of the gate queue is at 17 hours, and the test node graph at the bottom looks like we're at capacity for our test node use | 17:18 |
| openstackgerrit | Zara proposed openstack-infra/storyboard-webclient: Display Assignees on Cards and Worklists https://review.openstack.org/269707 | 17:18 |
| pleia2 | :file:`manifests/site.pp turns into "system-config: manifests/site.pp" with the link to git.o.o | 17:18 |
| mordred | pleia2: that's so super fancy | 17:18 |
| pleia2 | it's one of my favorite doc-things | 17:18 |
| jeblair | prometheanfire: the jobs launched graph seems to indicate we're launching 1,500 jobs per hour | 17:18 |
| jeblair | prometheanfire: so i think things are moving | 17:18 |
| prometheanfire | ok | 17:19 |
| *** Daisy has quit IRC | 17:19 | |
| jeblair | prometheanfire: the failure rate of some of the common gate jobs has been high recently: http://grafana.openstack.org/dashboard/db/tempest-failure-rate | 17:20 |
| *** Jeffrey4l has quit IRC | 17:22 | |
| jeblair | clarkb: it looks like the nodepool dsvm job may be flaky... have you looked into potential causes? (at least to determine whether it's internal to nodepool or external?) | 17:22 |
| jeblair | clarkb: one i'm looking at says: 2016-01-18 23:03:56,769 ERROR nodepool.SnapshotImageUpdater: Exception updating image trusty-server-cloudimg-amd64-disk1 in devstack: | 17:23 |
| jeblair | Exception: Timeout waiting for ssh access | 17:23 |
| *** HeOS has quit IRC | 17:23 | |
| *** derekh has quit IRC | 17:23 | |
| jeblair | clarkb: it ran on ovh; i wonder if we need to update our ssh timeout in the dsvm job? | 17:24 |
| clarkb | jeblair: it used to be external until last night. global requirements was constraining dib to a broken version but that was updated to version 1.7.1 last night | 17:26 |
| clarkb | jeblair: now the fail rate seems to be closer to 50% instead of 100% and I think its due to timeouts | 17:26 |
| clarkb | qemu performance is not so great | 17:27 |
| *** _et_ has quit IRC | 17:28 | |
| *** mrmartin has quit IRC | 17:28 | |
| *** shashank_hegde has joined #openstack-infra | 17:29 | |
| *** _et_ has joined #openstack-infra | 17:29 | |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/irc-meetings: Move the Neutron drivers meeting https://review.openstack.org/269780 | 17:30 |
| clarkb | it is possible that if we swap in dropbear we could make that go quicker | 17:30 |
| *** mudassirlatif has joined #openstack-infra | 17:30 | |
| * clarkb makes that change for the lols | 17:30 | |
| AJaeger | any project-config core to +2A https://review.openstack.org/#/c/261994/ and https://review.openstack.org/#/c/268276/ ,please? | 17:31 |
| jeblair | okay, i think we need to promote some debug log messages to info... | 17:31 |
| *** zeih has joined #openstack-infra | 17:31 | |
| AJaeger | Also, armax has a patch for jeepyb that I think will help bug triage and it needs another +2: https://review.openstack.org/253229, please | 17:32 |
| *** arxcruz has joined #openstack-infra | 17:32 | |
| *** acruz has quit IRC | 17:32 | |
| clarkb | oh wait no can't do that bceause it doesn't support scp/sftp | 17:34 |
| clarkb | which is why we don't cirros | 17:34 |
| pleia2 | wow, it never occurred to me that dropbear didn't do scp | 17:36 |
| pabelanger | jeblair: fungi: are in good shape for zuul release, or still waiting of a bug fix? | 17:36 |
| *** davideagnello has joined #openstack-infra | 17:37 | |
| clarkb | pleia2: ya I remmeber having a hard time debugging this the first time aroudn, then my memory returned to me | 17:37 |
| *** dslevin has quit IRC | 17:37 | |
| *** apoorvad has joined #openstack-infra | 17:37 | |
| *** ccrouch has joined #openstack-infra | 17:37 | |
| anteaya | AJaeger: you are changing the template running on keystone from python-jobs to python-jobs-linters, thereby removing the python-jobs template from keystone, and you are creating settings for a new job gate-keystone-linters which I don't see being used on keystone | 17:37 |
| anteaya | https://review.openstack.org/#/c/261994/4/zuul/layout.yaml | 17:38 |
| anteaya | AJaeger: so I'm confused | 17:38 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Normalize some log lines in builder https://review.openstack.org/269784 | 17:38 |
| *** sfinucan has quit IRC | 17:39 | |
| AJaeger | anteaya: python-jobs-linters template includes gate-keystone-linters | 17:39 |
| jeblair | clarkb, greghaynes; ^ that will help me from going cross-eyed trying to read the dsvm logs | 17:39 |
| AJaeger | anteaya: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n321 | 17:39 |
| anteaya | okay | 17:40 |
| *** shashank_hegde has quit IRC | 17:41 | |
| AJaeger | thanks, anteaya | 17:41 |
| anteaya | and I'm already +2 on 268276 | 17:42 |
| anteaya | welcome | 17:42 |
| jeblair | clarkb: i'm happy that https://review.openstack.org/267865 starts both builder and nodepoold and they both log... i note you have not approved likely because of rechecks... would you object to my approving? | 17:42 |
| anteaya | thanks for your patience with me | 17:42 |
| clarkb | jeblair: no objection, I was hoping to confirm things work more generally but I agree there are logs there which is an improvmeent | 17:42 |
| clarkb | jeblair: will make debugging everything else simpler too | 17:43 |
| jeblair | clarkb: k, i'll approve then | 17:43 |
| AJaeger | sdague: you were right with the multinode change. Thanks for not trusting me ;) | 17:43 |
| *** otter768 has joined #openstack-infra | 17:44 | |
| jeblair | pabelanger: the memory graphs appear to not be constantly increasing, however, we are using 3x the amount of memory we are using before, and there are occasional serious usage bumps. | 17:44 |
| AJaeger | anteaya: ok, need to ask another friendly soul to review 268276 | 17:44 |
| jeblair | cody-somerville-, jhesketh: ^ | 17:44 |
| jeblair | pabelanger: i do not feel comfortable making a zuul release under those circumstances. | 17:44 |
| openstackgerrit | afazekas proposed openstack-infra/tripleo-ci: scp/ssh do not check for know host keys https://review.openstack.org/269660 | 17:45 |
| jeblair | pabelanger: i certainly can't stand behind it. | 17:45 |
| AJaeger | anteaya: could you review https://review.openstack.org/#/c/269460/ as well, please? | 17:45 |
| clarkb | jeblair: the other one that should be safe to approve is ianw's stack | 17:46 |
| *** ddieterly has quit IRC | 17:46 | |
| anteaya | is it just me or is there a really big spike in zuul outbound eth0 traffic? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=397&rra_id=all | 17:46 |
| *** bardia has joined #openstack-infra | 17:47 | |
| *** ddieterl_ has joined #openstack-infra | 17:47 | |
| jeblair | clarkb: yeah working on that now | 17:47 |
| clarkb | anteaya: as the size of status.json gets bigger and more people wonder why their tests haven't reported back yet that will likely cascade | 17:47 |
| jeblair | clarkb: your vote is absent on https://review.openstack.org/269303 | 17:47 |
| anteaya | clarkb: oh okay, I think I understand | 17:48 |
| sdague | AJaeger: thanks for providing the specific reason why you thought it was wrong, that let me do the compares to make sure I wasn't imagining things | 17:48 |
| clarkb | jeblair: oh I wanted to see test results on that one since ssh is potentially affected | 17:48 |
| clarkb | jeblair: let me look | 17:48 |
| AJaeger | sdague: ;) | 17:48 |
| *** zeih has quit IRC | 17:48 | |
| openstackgerrit | Merged openstack-infra/nodepool: Use common logging for nodepool applications https://review.openstack.org/267865 | 17:48 |
| *** jistr has quit IRC | 17:48 | |
| AJaeger | sdague: could you review https://review.openstack.org/#/c/268276/ , please? | 17:48 |
| clarkb | jeblair: http://logs.openstack.org/03/269303/1/check/gate-dsvm-nodepool/a5bbdf2/logs/nodepool-image-list.txt.gz doesn't look too good | 17:49 |
| *** otter768 has quit IRC | 17:49 | |
| *** ZZelle has quit IRC | 17:49 | |
| anteaya | AJaeger: it appears you are going for making linters the new fashion trend | 17:49 |
| AJaeger | anteaya: yeah, I suggest it - but will not -1 anything here | 17:49 |
| sdague | AJaeger: +A | 17:49 |
| clarkb | jeblair: it hit http://logs.openstack.org/03/269303/1/check/gate-dsvm-nodepool/a5bbdf2/logs/screen-nodepool.txt.gz#_2016-01-18_23_03_56_769 | 17:50 |
| AJaeger | thanks, sdague and anteaya | 17:50 |
| *** ZZelle has joined #openstack-infra | 17:50 | |
| clarkb | jeblair: which may or may not be related to paramiko changing | 17:50 |
| clarkb | jeblair: maybe we should recheck it and see if the dib build works at least? that seems more reliable generally | 17:50 |
| *** burgerk_ has joined #openstack-infra | 17:50 | |
| AJaeger | anteaya: I'm currently trying to remove the extra bandit jobs and merge them in the lint target... | 17:50 |
| *** burgerk has quit IRC | 17:51 | |
| anteaya | oh okay | 17:52 |
| openstackgerrit | Merged openstack-infra/project-config: Use linters job for keystone starting with mitaka https://review.openstack.org/261994 | 17:52 |
| *** angdraug has joined #openstack-infra | 17:52 | |
| jeblair | clarkb: okay. my reasoning was that if ssh failed, i expected it to do in some less subtle way (like on import, or when trying to initiate the connection). but i also agree that test hasn't fully exercised all of the ssh paths, so we can wait if you want. | 17:52 |
| AJaeger | anteaya: keystone repos where the only ones having it voting, so patches are up for all of these. keystone is merged, keystoneclient approved, keystonemiddleware under review - and each has a corresponding project-config change | 17:53 |
| openstackgerrit | Merged openstack-infra/project-config: Define criteria for skipping multinode-live-migration https://review.openstack.org/267669 | 17:53 |
| *** unicell has quit IRC | 17:53 | |
| *** acruz has joined #openstack-infra | 17:53 | |
| *** flwang has quit IRC | 17:53 | |
| clarkb | jeblair: on the flip side it is easy enough to revert that change should we have any trouble with it | 17:53 |
| AJaeger | anteaya: so next one to go in is https://review.openstack.org/269270 - but depends-on is not merged yet | 17:53 |
| AJaeger | but if you want to review already... | 17:53 |
| clarkb | I will go ahead and approve it (I just realized that we don't daemonize in dsvm jobs anyways) | 17:54 |
| openstackgerrit | Miles Gould proposed openstack/requirements: Bump minimum python-ironicclient version https://review.openstack.org/266320 | 17:54 |
| jeblair | clarkb: heh, how true :) | 17:54 |
| *** arxcruz has quit IRC | 17:54 | |
| jeblair | clarkb, greghaynes, ianw, pabelanger: i think that's all the nodepool changes approved | 17:54 |
| *** bardia has quit IRC | 17:54 | |
| clarkb | gerrit says it is in merge conflict now | 17:54 |
| clarkb | I am guessing with common longging? | 17:55 |
| clarkb | *logging | 17:55 |
| jeblair | clarkb, greghaynes, ianw, pabelanger: except mine: https://review.openstack.org/269784 :( | 17:55 |
| clarkb | jeblair: want ot rebase ianw's stack to address conflicts and I will review your change? | 17:55 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 17:55 |
| jeblair | clarkb: can do | 17:55 |
| openstackgerrit | Merged openstack-infra/nodepool: Don't use global statsd https://review.openstack.org/267885 | 17:58 |
| pabelanger | jeblair: great! | 17:58 |
| anteaya | AJaeger: okay thanks for letting me know what is going on, I appreciate it | 18:00 |
| *** mdenny has joined #openstack-infra | 18:00 | |
| *** zhurong has quit IRC | 18:01 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Remove workaround for early paramiko import https://review.openstack.org/269303 | 18:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Revert "Import nodepool.builder late for statsd" https://review.openstack.org/269266 | 18:02 |
| jeblair | clarkb: i couldn't help myself and i made the imports longer :) | 18:02 |
| openstackgerrit | Merged openstack-infra/project-config: Add py27 and docs to steth https://review.openstack.org/269460 | 18:03 |
| *** sdake has joined #openstack-infra | 18:03 | |
| openstackgerrit | Merged openstack-infra/project-config: Use zuul-cloner for hacking integration test https://review.openstack.org/268276 | 18:03 |
| anteaya | AJaeger: thanks then I don't have to figure it out again later | 18:03 |
| clarkb | jeblair: +2 to both | 18:04 |
| openstackgerrit | Walter A. Boring IV (hemna) proposed openstack/requirements: Update the os-brick version https://review.openstack.org/269793 | 18:04 |
| anteaya | I'm going to grab some lunch before meeting | 18:04 |
| anteaya | I'm ready to answer the channel best I can when the security patch comes out | 18:05 |
| *** hemna has joined #openstack-infra | 18:05 | |
| AJaeger | anteaya: thanks - in that case https://review.openstack.org/#/c/269261/ for after your lunch ;) | 18:06 |
| *** sdake_ has quit IRC | 18:06 | |
| *** sputnik13 has joined #openstack-infra | 18:06 | |
| *** ashishb has joined #openstack-infra | 18:06 | |
| anteaya | okay thanks | 18:07 |
| *** fawadkhaliq has quit IRC | 18:08 | |
| rhedlind | I have been having issue with jobs not getting kicked off on my CI ever since I upgraded to the latest zuul last week. http://paste.openstack.org/show/484316/ shows the output in the zuul debug log and layout.yaml (has not changed). If anyone has a minute to spare to help me come up with ideas what might be wrong I would appreciate it. | 18:08 |
| tdurakov | AJaeger, hi | 18:08 |
| *** Swami has joined #openstack-infra | 18:08 | |
| AJaeger | hi tdurakov, sorry for wrong review ;( | 18:09 |
| AJaeger | fungi, ok to still add https://review.openstack.org/#/c/262545/ as spec for approving to the agenda? | 18:09 |
| tdurakov | AJaeger, np, could you help me to check things? | 18:10 |
| *** kzaitsev_mb has quit IRC | 18:10 | |
| fungi | AJaeger: sure, if you think it's ready to go up for a council vote now | 18:10 |
| fungi | if people want a longer warning, they can speak up in the meeting and we'll adjust the voting deadline accordingly | 18:11 |
| *** ashishb has quit IRC | 18:11 | |
| tdurakov | the change was merged recently, so I've triggered check experimental for https://review.openstack.org/#/c/252283/ but haven't seen new job in zuul | 18:11 |
| *** nmagnezi has joined #openstack-infra | 18:11 | |
| *** zeih has joined #openstack-infra | 18:12 | |
| AJaeger | tdurakov: I'll check | 18:13 |
| AJaeger | fungi, fine with me | 18:13 |
| *** BobBall is now known as BobBall_AWOL | 18:13 | |
| *** Hal has joined #openstack-infra | 18:14 | |
| tdurakov | AJaeger, thank you, wanted to know whether this change already populated on ci servers or not | 18:14 |
| *** Hal is now known as Guest95435 | 18:14 | |
| sdague | jeblair / clarkb - http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22bash:%20install_package:%20command%20not%20found%5C%22%20AND%20tags:console%20AND%20voting:1%20AND%20build_queue:gate&from=864000s | 18:14 |
| AJaeger | tdurakov: seems not - zuul still says "Tue Jan 19 2016 09:55:41 GMT+0100 (CET)" | 18:14 |
| *** jtomasek_ has quit IRC | 18:14 | |
| AJaeger | So, let's wait until zuul is ready... | 18:14 |
| sdague | it might make sense to turn of rax-iad from the 2 node allocator as well | 18:14 |
| *** bardia has joined #openstack-infra | 18:14 | |
| sdague | it seems like it can't reliably get the second node up | 18:14 |
| AJaeger | tdurakov: I'm having some jobs merged as well and wait for them, keep an eye on it. | 18:14 |
| tdurakov | AJaeger, ok, acked | 18:15 |
| sdague | 20 gate fails in the past week | 18:15 |
| sdague | 12 in the last 24 hours | 18:16 |
| openstackgerrit | Graham Hayes proposed openstack/requirements: Bump version of debtcollector to 1.2.0 https://review.openstack.org/267693 | 18:16 |
| *** e0ne has joined #openstack-infra | 18:16 | |
| *** qeas has joined #openstack-infra | 18:17 | |
| AJaeger | something fishy going on with zuul? Last reconfigure was 9:55, so 9 hours ago and a change merged two hours ago that should have updated layout.yaml (269284 ) | 18:17 |
| *** sabeen1 has quit IRC | 18:17 | |
| *** moravec has joined #openstack-infra | 18:17 | |
| *** eil397 has joined #openstack-infra | 18:18 | |
| AJaeger | looking at puppetmaster, I see that zuul is unreported since 4 hours ;8 | 18:18 |
| jeblair | mordred: any chance some puppet stuff caused that ^ ? | 18:18 |
| *** pradk has quit IRC | 18:19 | |
| clarkb | sdague: that looks like grenade is failing to setup the devstack repo on the subnode | 18:19 |
| mordred | jeblair: yes. we are waitingo n a fix to land | 18:19 |
| clarkb | but I don't see any logs | 18:19 |
| openstackgerrit | Merged openstack-infra/system-config: Use /opt not /etc as the system-config location https://review.openstack.org/269732 | 18:19 |
| mordred | yay! | 18:19 |
| mordred | there is patch number one (right on queue) | 18:19 |
| openstackgerrit | Sean Dague proposed openstack-infra/project-config: don't do multinode jobs in rax-iad https://review.openstack.org/269799 | 18:19 |
| jeblair | mordred: ah heh :) | 18:19 |
| AJaeger | mordred: how many do we need? ;) | 18:19 |
| *** ashishb has joined #openstack-infra | 18:19 | |
| mordred | one more | 18:20 |
| AJaeger | thanks, jeblair and mordred | 18:20 |
| AJaeger | tdurakov: see above, we first need to fix puppet | 18:20 |
| *** knikolla has joined #openstack-infra | 18:21 | |
| *** piet has joined #openstack-infra | 18:21 | |
| tdurakov | AJaeger, ok | 18:21 |
| jeblair | clarkb, sdague: because of this? http://logs.openstack.org/27/266827/1/gate/gate-grenade-dsvm-multinode/a453e43/console.html#_2016-01-19_09_00_45_039 | 18:22 |
| clarkb | sdague: http://logs.openstack.org/27/266827/1/gate/gate-grenade-dsvm-multinode/a453e43/logs/devstack-gate-setup-workspace-new.txt.gz | 18:22 |
| *** abregman has joined #openstack-infra | 18:22 | |
| clarkb | jeblair: yup that seems to be the genesis of the problem which leads to the thing I linked | 18:22 |
| *** qeas has quit IRC | 18:22 | |
| sdague | clarkb: that's before grenade touches anything | 18:23 |
| clarkb | sdague: yes | 18:23 |
| clarkb | I should s/grenade/the job/ | 18:23 |
| *** salv-orlando has quit IRC | 18:23 | |
| jeblair | sdague, clarkb: so perhaps some kind of intra-rax-iad network issue on the private network | 18:23 |
| sdague | jeblair: perhaps | 18:23 |
| jeblair | clarkb, sdague: is there a reason we use the private network? | 18:23 |
| sdague | it's been happening for a long time | 18:23 |
| sdague | and only this region | 18:24 |
| clarkb | jeblair: yes, public networks traverse NAT sometimes | 18:24 |
| clarkb | and NAT is terrible | 18:24 |
| jeblair | clarkb, sdague: nodepool verifies external connectivity to nodes via the public network... | 18:24 |
| jeblair | clarkb: i agree in principle -- however, has that caused us problems? | 18:24 |
| clarkb | jeblair: I believe they did, we intentionally switched at one point | 18:24 |
| jeblair | clarkb: okay. i ask because clearly the internal network is not particularly reliable. | 18:25 |
| clarkb | yup I agree, not disputing that | 18:25 |
| clarkb | and in this region using the public ip would be fine | 18:25 |
| jeblair | clarkb: was it on hpcloud or rax that the public network was causing us problems due to nat? | 18:25 |
| clarkb | would have been hpcloud | 18:25 |
| clarkb | since rax doesn't NAT aiui | 18:26 |
| jeblair | clarkb: was the nat affecting the tunnel that's built? | 18:26 |
| knikolla | clarkb: Thanks for the help. I’m able to login on review.openstack.org with my launchpad account and that pointed me to the ‘knikolla’ user. | 18:26 |
| *** pradk has joined #openstack-infra | 18:26 | |
| *** pradk has quit IRC | 18:26 | |
| sdague | anyway, if we could disable that region for now for this set of jobs, it would help reduce the failure rate, and provide time to explore options | 18:26 |
| clarkb | jeblair: I0c8cf8f5627d1c00f278827ab0cee708f7741e89 thats the change, apparently it was for debugging purposes | 18:27 |
| jeblair | clarkb: and were the nat problems consistent or intermittent? | 18:27 |
| *** unicell has joined #openstack-infra | 18:27 | |
| openstackgerrit | Merged openstack-infra/system-config: Don't rsync puppet on localhost https://review.openstack.org/269736 | 18:28 |
| knikolla | clarkb: And through git review -s also. All looks good. | 18:28 |
| clarkb | knikolla: excellent let us know if you run into any problems | 18:28 |
| jeblair | clarkb: i'm having trouble mapping that change onto a world where we have providers with only public ips | 18:29 |
| *** dizquierdo has quit IRC | 18:29 | |
| *** baoli has quit IRC | 18:29 | |
| *** knikolla has quit IRC | 18:29 | |
| clarkb | jeblair: if there is no explicit private IP nodepool writes the public IPs to the private IP list | 18:29 |
| jeblair | clarkb: so that change is just a noop in that case? | 18:29 |
| clarkb | yes | 18:30 |
| jeblair | clarkb: that makes its logic somewhat less compelling in my mind. | 18:30 |
| clarkb | ok | 18:30 |
| *** trown is now known as trown|lunch | 18:30 | |
| *** fhubik has quit IRC | 18:30 | |
| *** rossella_s has quit IRC | 18:30 | |
| clarkb | I am still spooling up stuff from a year ago to figure out what was done | 18:31 |
| jeblair | clarkb: *nod* | 18:31 |
| clarkb | there was definitely a thing with live migrations | 18:31 |
| *** rossella_s has joined #openstack-infra | 18:31 | |
| jeblair | sdague: do live migrations work on ovh? | 18:31 |
| clarkb | and unfortunately we don't have the parent child relationship in gerrit anymore so I can't easily look at the related changes | 18:32 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/release-tools: strip _ prefix from independent series https://review.openstack.org/269802 | 18:32 |
| mordred | jeblair, fungi: ok - the current puppet run looks like it's running puppet | 18:32 |
| sdague | jeblair: I do not know | 18:32 |
| mordred | jeblair, fungi: it has run puppet on the git nodes so far | 18:32 |
| fungi | clarkb: jeblair: sdague: if memory serves, there are multiple rfc-1918 backend nets in some rax regions and to be able to communicate between them you may need additional routes on eth1 | 18:32 |
| lucasagomes | hi all, quick question... there's anyway to disable that CTRL+F keybind from gerrit? I looked at the preferences but couldn't find a way to disable it | 18:33 |
| jeblair | fungi, clarkb: this is only making me more interested in only using public ips | 18:33 |
| lucasagomes | (I probably will have to do it at browser level) | 18:33 |
| mordred | jeblair: ++ | 18:33 |
| *** igorbelikov has quit IRC | 18:33 | |
| *** zhiyan has quit IRC | 18:33 | |
| *** maximov has quit IRC | 18:33 | |
| *** etoews has quit IRC | 18:33 | |
| *** edwarnicke has quit IRC | 18:33 | |
| *** odyssey4me has quit IRC | 18:33 | |
| lucasagomes | I just dislike the fact that gerrit is overwriting existing browser functionality I use all the time | 18:33 |
| jeblair | clarkb: I3afa113f731687be7bacf924e7f26a0635f95d46 may be relevant | 18:34 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API https://review.openstack.org/221964 | 18:34 |
| *** tpeoples has quit IRC | 18:34 | |
| *** kozhukalov has quit IRC | 18:34 | |
| mordred | jeblair, fungi: and the nodes have begun reporting to puppetdb as expected | 18:34 |
| *** pvaneck has joined #openstack-infra | 18:34 | |
| jeblair | mordred: yay! AJaeger ^ | 18:34 |
| *** woodster_ has quit IRC | 18:34 | |
| *** rguillebert has quit IRC | 18:34 | |
| *** jraim has quit IRC | 18:34 | |
| *** auggy has quit IRC | 18:34 | |
| *** Ng has quit IRC | 18:34 | |
| *** kozhukalov has joined #openstack-infra | 18:34 | |
| *** igorbelikov has joined #openstack-infra | 18:34 | |
| *** mwhahaha_ has joined #openstack-infra | 18:34 | |
| *** Ng_ has joined #openstack-infra | 18:34 | |
| fungi | mordred: thanks! | 18:34 |
| AJaeger | yah, thanks! | 18:34 |
| *** Guest95435 has quit IRC | 18:34 | |
| nibalizer | mordred: woot | 18:34 |
| sdague | jeblair: looks like yes - http://logs.openstack.org/15/255515/4/check/gate-tempest-dsvm-multinode-full/8db66f3/console.html#_2016-01-19_09_59_36_457 | 18:35 |
| clarkb | fuzzy memory said it had to do with libvirt not know what the public IPs are in a NAT/floating ip siutation | 18:35 |
| *** othermordred has quit IRC | 18:35 | |
| *** mwhahaha has quit IRC | 18:35 | |
| *** rackertom has quit IRC | 18:35 | |
| clarkb | so live migration would fail | 18:35 |
| jeblair | sdague: cool | 18:35 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Remove flavor creation https://review.openstack.org/251553 | 18:35 |
| *** mwhahaha_ is now known as mwhahaha | 18:35 | |
| jeblair | clarkb: aha | 18:35 |
| clarkb | its the whole "host doesn't know its actual IP problem" iirc | 18:35 |
| *** etoews has joined #openstack-infra | 18:35 | |
| jeblair | mordred: ^ wow | 18:35 |
| *** dham1 has quit IRC | 18:35 | |
| *** teran has quit IRC | 18:35 | |
| *** mhayden has quit IRC | 18:35 | |
| jeblair | mordred: it even affects openstack :) | 18:35 |
| clarkb | but I need to confirm that still | 18:35 |
| jeblair | mordred: but still, somehow, it's "not a problem" | 18:35 |
| sdague | clarkb: that would be true of floating ips | 18:36 |
| *** hockeynut has quit IRC | 18:36 | |
| *** odyssey4me has joined #openstack-infra | 18:36 | |
| clarkb | jeblair: that said it is easy enough to push a patch up to use only public IPs | 18:36 |
| clarkb | and see what happens | 18:36 |
| sdague | but if there is some public network that's not floating ips, then I'm not sure it's an issue | 18:36 |
| jeblair | clarkb: i am in favor of taht | 18:36 |
| clarkb | then recheck until you run somewhere with nat | 18:36 |
| *** tpeoples has joined #openstack-infra | 18:36 | |
| *** zhiyan has joined #openstack-infra | 18:36 | |
| *** maximov has joined #openstack-infra | 18:36 | |
| *** hockeynut has joined #openstack-infra | 18:36 | |
| *** bastelfreak has quit IRC | 18:37 | |
| *** othermordred has joined #openstack-infra | 18:37 | |
| *** jraim has joined #openstack-infra | 18:37 | |
| *** rguillebert has joined #openstack-infra | 18:37 | |
| *** dham1 has joined #openstack-infra | 18:37 | |
| *** teran has joined #openstack-infra | 18:37 | |
| *** auggy has joined #openstack-infra | 18:37 | |
| *** woodster_ has joined #openstack-infra | 18:37 | |
| clarkb | I will write that change | 18:37 |
| jeblair | clarkb, mordred: aiui, bluebox will soon be the only provider we use that has nat | 18:37 |
| mordred | jeblair: yup | 18:38 |
| *** annegentle has joined #openstack-infra | 18:38 | |
| *** bastelfreak has joined #openstack-infra | 18:38 | |
| *** mhayden has joined #openstack-infra | 18:38 | |
| *** angdraug has quit IRC | 18:38 | |
| sdague | can we do https://review.openstack.org/#/c/269799/ in parallel, as that would help relieve the 14 hr backlog in the gate | 18:38 |
| mordred | so - should we perhaps treat the problem as "we should use the network to which the vm is actually attached" ? | 18:38 |
| *** rackertom has joined #openstack-infra | 18:38 | |
| armax | regXboi: ^ | 18:39 |
| *** edwarnicke has joined #openstack-infra | 18:39 | |
| mordred | which is to say - on bluebox, since we are natted, we know that the network to which the vm is attached, the 'private' network, is a _real_ network that communication between vms can happen on | 18:39 |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Use public IPs for all comms https://review.openstack.org/269807 | 18:39 |
| zaro | morning | 18:39 |
| mordred | it is possible to know to use this from the metadata | 18:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Require non-NAT from providers https://review.openstack.org/269808 | 18:40 |
| armax | regXboi: it looks like bug 1535850 can be marked as duplicate | 18:40 |
| openstack | bug 1535850 in neutron "multinode check pipeline failing on Permission denied for scp" [Undecided,New] https://launchpad.net/bugs/1535850 | 18:40 |
| jeblair | mordred: for discussion, an alternate approach ^ | 18:40 |
| mordred | I mean - I hate NAT as much as the next guy | 18:40 |
| regXboi | armax: of which? | 18:40 |
| mordred | but I believe we can fix this by considering the metainfo we have currently in a different light | 18:40 |
| *** ybathia has joined #openstack-infra | 18:40 | |
| *** salv-orlando has joined #openstack-infra | 18:41 | |
| jeblair | mordred: but should we have to? it's about to become a minority configuration. | 18:41 |
| *** shashank_hegde has joined #openstack-infra | 18:41 | |
| armax | regXboi: bug 1531187 | 18:41 |
| openstack | bug 1531187 in OpenStack-Gate "subnode fails to be accessable over ssh" [Undecided,New] https://launchpad.net/bugs/1531187 | 18:41 |
| mordred | it's about to become a minority configuration in our current set of providers | 18:41 |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds fairy-slipper build jobs for api-site repo https://review.openstack.org/269809 | 18:41 |
| * regXboi reads | 18:41 | |
| mordred | but it's still a normal enough configuratoin, much as I hate that it is | 18:41 |
| mordred | that I tihnk us standing our ground and demanding that a cloud behave in a certian way does not fit with our stance up to this point which is "we will work around brokenness exhibited by clouds" | 18:42 |
| jeblair | mordred: i'm not sure we're in a position to demand anything of clouds :) | 18:42 |
| mordred | also, having talked with folks, the chances of us getting a direct public network in bluebox in the near term is none | 18:42 |
| regXboi | armax: ok we'll mark it as a dup for now and unmark it if this all doesn't fix things | 18:43 |
| zaro | anteaya: what's this deal with the sandbox repo? | 18:43 |
| armax | regXboi: I already did | 18:43 |
| *** jsavak has quit IRC | 18:44 | |
| * regXboi wonders if the ansible issues might be pegged to the same thing | 18:44 | |
| clarkb | regXboi: ansible issues? | 18:44 |
| clarkb | oh you mean the thing we are discussing, yes thats how we found it :) | 18:44 |
| *** jsavak has joined #openstack-infra | 18:44 | |
| *** sridhar_ram has quit IRC | 18:44 | |
| *** ari is now known as _ari_ | 18:45 | |
| regXboi | clarkb: I was seeing two different error signatures in the multinode job - one was ansible failing on the primary and the other was scp failing - I am now thinking the two have the same source | 18:45 |
| regXboi | er *in the multinode job I was looking at" | 18:45 |
| clarkb | jeblair: it is worth pointing out that ssh seems to work sometimes | 18:45 |
| clarkb | jeblair: so I don't think it is a route issue, but it is definitely unreliable | 18:46 |
| clarkb | you can see it failed on your link but it ran on mine and failed to find the file | 18:46 |
| clarkb | all in the same job | 18:46 |
| jeblair | clarkb: oh, that casts doubt on fungi's theory | 18:46 |
| fungi | yep, agreed | 18:46 |
| fungi | so this could be "backend network in iad is unreliable" | 18:47 |
| clarkb | or possibly even "network" in iad is unreliable | 18:47 |
| jeblair | mordred: okay, my concern is the amount of engineering that might go into what, for us, is a minority configuration | 18:47 |
| fungi | or possibly even "iad is unreliable" | 18:47 |
| clarkb | we just don't typically notice because jenkins and ndepool delete nodes with problems if it can find them | 18:47 |
| clarkb | but once they are attached they may be fine | 18:47 |
| mordred | jeblair: yah - I hear that - I think it's an almost trivial amount of work though | 18:48 |
| jeblair | mordred: but i can step off my soapbox for a moment and actually consider what that engineering should be :) | 18:48 |
| mordred | jeblair: so I will make a POC script that shows that | 18:48 |
| AJaeger | tdurakov: try again, zuul has been reconfigured and most jenkins are up to date | 18:48 |
| jeblair | mordred: okay cool | 18:48 |
| clarkb | we can collect host uuids and provide them back to rax and see if they can figure it out | 18:48 |
| *** ashishb has quit IRC | 18:49 | |
| *** pradk has joined #openstack-infra | 18:49 | |
| *** ashishb has joined #openstack-infra | 18:49 | |
| mordred | the logic goes something a little like this: "do you have a direct-attached public network? yes? use it. do you have a floating ip and only one direct attached network? use the direct attached. do you have a floating ip and more than one direct attached network - fail, because that's an unknown combination" | 18:49 |
| anteaya | zaro: https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 18:49 |
| anteaya | zaro: https://review.openstack.org/#/c/269387/ | 18:49 |
| mordred | jeblair: does that logic seem like it makes sense to you? | 18:49 |
| anteaya | zaro: gerrit says sandbox gives +2 to all registered users | 18:50 |
| anteaya | zaro: except it doesn't | 18:50 |
| jeblair | mordred: yes, and i think you can implement it by checking whether the nodepool reported public ip is on a local interface | 18:50 |
| *** Ramanjaneya has quit IRC | 18:50 | |
| mordred | jeblair: yes indeed | 18:50 |
| jeblair | mordred, clarkb: however, is it also potentially possible to solve this by configuration -- like "configure libvirt to know that its ip address is the nodepool supplied public ip, regardless of whether it appears locally?" | 18:51 |
| mordred | jeblair: in fact, if we do not care about being comprehensive - we could simply make it "is the public ip on your local interface? use it. else? use private" | 18:51 |
| jeblair | mordred, clarkb: that's essentially how you solve the problem with afs -- it's "i know you can't figure out your public ip because of nat, so here it is" | 18:51 |
| jeblair | mordred: yeah, i think that works | 18:52 |
| fungi | anteaya: zaro: i wonder if that's inheriting the -1..+1 range got registered users from the all-projects acl and so not getting overridden in the sandbox? | 18:52 |
| mordred | jeblair: that seems like a thing we can do entirely in the setup for this without any nodepool or ready script changes | 18:52 |
| fungi | s/got/for/ | 18:52 |
| mordred | since it's logic that works with the informatoin that is already written to the node anyway | 18:53 |
| tdurakov | AJaeger, works as expected, thank you | 18:53 |
| anteaya | fungi zaro oh perhaps that is it | 18:53 |
| anteaya | I wonder how we address that if that is what is happening | 18:53 |
| andreaf | clarkb: if you have a sec, I have another small puppet change on log_processor (add if ! defined to deps) https://review.openstack.org/#/c/269136/ | 18:53 |
| *** boris-42 has quit IRC | 18:53 | |
| fungi | anteaya: probably need to compare the ref pattern for that label in all-projects and see if maybe it's more specific than the one in the sandbox acl | 18:54 |
| fungi | if so, that would account for what we're seeing | 18:54 |
| *** salv-orlando has quit IRC | 18:54 | |
| anteaya | fungi zaro I will see if I can figure out how to do that (if I am able with my level of permissions) | 18:54 |
| fungi | anteaya: you can either look at it via gitweb, or just rely on what's in the gerrit configuration section of our system-config documentation (which should be accurate for that at least) | 18:55 |
| *** sdake_ has joined #openstack-infra | 18:55 | |
| anteaya | I'll start there | 18:55 |
| mordred | fungi: we're down to 10 unreported nodes | 18:55 |
| mordred | fungi: so I'll declare ansible/puppet fixed again | 18:56 |
| * anteaya still doesn't know what gitweb is, she thinks it is the gerrit ui | 18:56 | |
| pleia2 | anteaya: gitweb is the web interface that actually comes with git | 18:56 |
| fungi | anteaya: there's a gitweb interface served from review.openstack.org, and it's linked from the acl view pages in gerrit too | 18:56 |
| pleia2 | anteaya: it's not very beautiful, so everyone uses something else (like we use cgit) | 18:56 |
| pleia2 | for user interactions | 18:56 |
| *** arxcruz has joined #openstack-infra | 18:57 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds release notes jobs for fairy-slipper, app dev doc tool https://review.openstack.org/269813 | 18:57 |
| anteaya | pleia2: ah thank you | 18:57 |
| anteaya | I found this: https://review.openstack.org/gitweb?p=openstack-dev/sandbox.git;a=history;hb=refs/meta/config;f=project.config | 18:57 |
| fungi | however the gitweb on review.o.o has the up side that it serves repos/refs/branches that gerrit doesn't push to our mirrors (for example, the all-projects acl) | 18:57 |
| anteaya | fungi: thank you | 18:57 |
| *** sdake has quit IRC | 18:57 | |
| anteaya | fungi: ah ha, thank you | 18:58 |
| fungi | which is why our cgit doesn't have that | 18:58 |
| mordred | pleia2: at some point in the past ... | 18:58 |
| clarkb | jeblair: https://review.openstack.org/#/c/135768/ is the change I was hoping would shed more light on this but our logs hvae rolled off | 18:58 |
| mordred | oh | 18:58 |
| mordred | fungi answered the question I was about to ask | 18:58 |
| AJaeger | tdurakov: glad to hear! | 18:58 |
| mordred | which is why we didn't just use the cgit | 18:58 |
| pleia2 | yeah, fungi had usefulness/context data :) | 18:59 |
| *** acruz has quit IRC | 18:59 | |
| fungi | mordred: well, i'd still not be entirely opposed to switching gerrit's links to point to our cgit instead, even if we can't get it to still serve the all-projects acl | 18:59 |
| openstackgerrit | Andreas Jaeger proposed openstack-dev/hacking: Just a test... https://review.openstack.org/269814 | 19:00 |
| mordred | fungi: I think it would be niceish | 19:00 |
| fungi | meeting time... join us in #openstack-meeting | 19:00 |
| openstackgerrit | Sergey Lukjanov proposed openstack-infra/project-config: Rename sahara-scenario -> sahara-tests https://review.openstack.org/269815 | 19:00 |
| *** tiswanso has quit IRC | 19:01 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Run instack-test-overcloud https://review.openstack.org/269816 | 19:01 |
| *** tiswanso has joined #openstack-infra | 19:02 | |
| zaro | mordred: review change 249492 | 19:02 |
| *** jsavak has quit IRC | 19:03 | |
| *** alex_xu has quit IRC | 19:04 | |
| *** salv-orlando has joined #openstack-infra | 19:04 | |
| *** sabeen has joined #openstack-infra | 19:04 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 19:05 |
| *** acruz has joined #openstack-infra | 19:05 | |
| *** arxcruz has quit IRC | 19:06 | |
| *** sabeen2 has joined #openstack-infra | 19:06 | |
| *** zeih has quit IRC | 19:06 | |
| *** alex_xu has joined #openstack-infra | 19:07 | |
| *** jsavak has joined #openstack-infra | 19:08 | |
| *** sabeen has quit IRC | 19:09 | |
| *** eliqiao has quit IRC | 19:11 | |
| *** eliqiao has joined #openstack-infra | 19:13 | |
| *** jsavak has quit IRC | 19:14 | |
| *** tdurakov has quit IRC | 19:15 | |
| *** tdurakov has joined #openstack-infra | 19:16 | |
| *** angdraug has joined #openstack-infra | 19:17 | |
| *** acruz has quit IRC | 19:17 | |
| *** ihrachys has quit IRC | 19:18 | |
| *** jcoufal has quit IRC | 19:19 | |
| *** sridhar_ram has joined #openstack-infra | 19:23 | |
| openstackgerrit | Anne Gentle proposed openstack-infra/project-config: Adds release notes jobs for fairy-slipper, app dev doc tool https://review.openstack.org/269813 | 19:23 |
| *** erikwilson has joined #openstack-infra | 19:24 | |
| *** jaosorior has quit IRC | 19:25 | |
| *** jaosorior has joined #openstack-infra | 19:25 | |
| greghaynes | clarkb: did nodepool get a redeploy today? | 19:25 |
| clarkb | greghaynes: not yet | 19:25 |
| greghaynes | ok | 19:25 |
| clarkb | greghaynes: I think there is one or two changes we are waiting on | 19:26 |
| * greghaynes looks | 19:26 | |
| greghaynes | https://review.openstack.org/#/c/268867/ | 19:26 |
| greghaynes | has 2x+2 | 19:26 |
| greghaynes | also https://review.openstack.org/#/c/269266/2 | 19:30 |
| *** trown|lunch is now known as trown | 19:31 | |
| *** dkranz has quit IRC | 19:32 | |
| *** Ng_ is now known as Ng | 19:32 | |
| *** dkranz has joined #openstack-infra | 19:33 | |
| *** _nadya_ has joined #openstack-infra | 19:33 | |
| *** piet has quit IRC | 19:33 | |
| *** sridhar_ram has quit IRC | 19:34 | |
| *** sridhar_ram1 has joined #openstack-infra | 19:34 | |
| *** jsavak has joined #openstack-infra | 19:35 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: publish from openstack/releases to releases.openstack.org https://review.openstack.org/266515 | 19:39 |
| *** fitoduarte has joined #openstack-infra | 19:39 | |
| *** jsavak has quit IRC | 19:40 | |
| anteaya | dhellmann: you can just remove the whole target: line | 19:42 |
| *** jsavak has joined #openstack-infra | 19:42 | |
| anteaya | dhellmann: though setting the value to empty string will accomplish the same thing | 19:42 |
| anteaya | target: isn't a parameter that is required | 19:43 |
| dhellmann | anteaya : ah, sure | 19:43 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: publish from openstack/releases to releases.openstack.org https://review.openstack.org/266515 | 19:43 |
| *** jsavak has quit IRC | 19:43 | |
| *** acruz has joined #openstack-infra | 19:43 | |
| anteaya | dhellmann: so pretty | 19:44 |
| *** otter768 has joined #openstack-infra | 19:45 | |
| AJaeger | dhellmann: publishing to static.openstack.org is done differently than publishing to docs.o.o | 19:46 |
| dhellmann | AJaeger : did I edit the wrong file? | 19:47 |
| *** dslev has joined #openstack-infra | 19:47 | |
| dhellmann | oops, wrong patch, changing context... | 19:47 |
| zaro | pabelanger: not seeing apache::vhost::custom change merged. were you guys talking about a different change? | 19:47 |
| dhellmann | AJaeger : ok, I'm not sure what your comment is related to? | 19:47 |
| AJaeger | dhellmann: 266515, see my inline comment | 19:48 |
| greghaynes | clarkb: oh hey that requirements change merged for dib | 19:48 |
| AJaeger | dhellmann: I suggest you mark your changes as WIP until the spec is approved. | 19:48 |
| *** gyee has quit IRC | 19:49 | |
| AJaeger | or make it dependent on the spec ;) | 19:49 |
| dhellmann | AJaeger : ok, I'll take a look at it again tomorrow (meeting time starting soon) | 19:49 |
| *** rockyg has joined #openstack-infra | 19:49 | |
| *** santoshk has joined #openstack-infra | 19:49 | |
| *** krtaylor has quit IRC | 19:49 | |
| *** otter768 has quit IRC | 19:50 | |
| AJaeger | sure | 19:50 |
| *** Swanson has joined #openstack-infra | 19:50 | |
| dhellmann | AJaeger : thanks! | 19:50 |
| anteaya | Swanson: what is the situation please? | 19:50 |
| Swanson | I'm currently spamming the universe. For a CI I want to chop down the jobs section of zuul's layout.yaml file to just what I care about, right? | 19:50 |
| anteaya | what do you mean? | 19:50 |
| anteaya | please explain the details of spamming the universe | 19:51 |
| Swanson | I'm working on a new CI. I just got zuul up and running and I seem to be attaching myself to every possible review. | 19:51 |
| anteaya | what is the name of your ci system? | 19:51 |
| AJaeger | stevemar: there are 5 or so keystone patches that were started before the linters target was in and they need a recheck - you just rechecked the first one. They're still building... Everything newer than those is fine. | 19:51 |
| Swanson | DELL SC CI | 19:51 |
| anteaya | Swanson: have you tried shutting it off? | 19:51 |
| anteaya | if you haven't tried shutting it off, shut if off | 19:52 |
| Swanson | anteaya, In theory it is off. | 19:52 |
| *** eharney has quit IRC | 19:52 | |
| *** piet has joined #openstack-infra | 19:52 | |
| anteaya | does it have any running processes? | 19:52 |
| pabelanger | zaro: puppetlabs-apache | 19:53 |
| dougwig | Swanson: do you have a link to your zuul layout.yaml file? it's normal for it to *see* every review. | 19:53 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 19:53 |
| dougwig | Swanson: and a good starting point is here: https://github.com/openstack-infra/project-config-example | 19:53 |
| pabelanger | zaro: https://github.com/puppetlabs/puppetlabs-apache/blob/master/manifests/vhost/custom.pp | 19:53 |
| greghaynes | SpamapS: ^ I have some confidence that dib boot test patch will pass this time | 19:53 |
| SpamapS | greghaynes: yay | 19:53 |
| greghaynes | SpamapS: FYI since I really want that to merge after our bootloader breakage | 19:53 |
| Swanson | dougwig, It is probably the default. It didn't want to work and I followed someone elses suggestion and I think I got the default again. | 19:54 |
| anteaya | Swanson: I see that your ci system has reviewed 8 patches: https://review.openstack.org/#/q/reviewer:%22DELL+SC+CI+%253Copenstack-cinder-eql-ci%2540dell.com%253E%22 | 19:54 |
| *** jsavak has joined #openstack-infra | 19:54 | |
| *** kzaitsev_mb has joined #openstack-infra | 19:54 | |
| *** salv-orlando has quit IRC | 19:54 | |
| anteaya | dougwig: thank you | 19:54 |
| dougwig | Swanson: can you pastebin your layout file here? | 19:54 |
| krotscheck | AFS-supportive system-config patches for wheel mirror are up for review. https://review.openstack.org/#/q/branch:master+topic:wheel_mirror+status:open | 19:55 |
| *** ashishb has quit IRC | 19:56 | |
| *** PsionTheory has joined #openstack-infra | 19:56 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Just let shade and os-client-config handle clouds https://review.openstack.org/226758 | 19:56 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions https://review.openstack.org/226751 | 19:56 |
| *** rfolco has quit IRC | 19:56 | |
| AJaeger | mordred: one shade to rule them all? | 19:57 |
| Swanson | dougwig, anteaya It looks like it is the default. Let me see about pasting it. | 19:57 |
| *** sdake_ is now known as sdaake | 19:57 | |
| *** sdaake is now known as sdake | 19:57 | |
| anteaya | Swanson: paste.openstack.org is suggested unless you like something else | 19:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Log task name more succinctly https://review.openstack.org/247658 | 19:57 |
| Swanson | dougwig, the entire universe is in it ... http://paste.openstack.org/show/484330/ | 19:57 |
| mordred | AJaeger: :) | 19:58 |
| dougwig | Swanson: yegads, turn off your zuul right now. | 19:58 |
| Swanson | anteaya, dougwig the pipelines don't matter, right? It is the jobs, right? | 19:58 |
| dougwig | Swanson: start with this one, https://github.com/openstack-infra/project-config-example/blob/master/zuul/layout.yaml | 19:58 |
| dougwig | Swanson: and built it up. | 19:58 |
| Swanson | dougwig, Zuul should be off. That layout looks way closer to what I started with. | 19:59 |
| *** aysyd has quit IRC | 19:59 | |
| dougwig | Swanson: did you really intend to define that many jobs across that many projects? that is the beefiest 3rd party ci that i've ever seen. what are you trying to test? | 20:00 |
| *** megm has quit IRC | 20:00 | |
| *** jsavak has quit IRC | 20:00 | |
| Clint | jeblair: https://review.openstack.org/#/c/259705/ stole my lunch money | 20:00 |
| Swanson | dougwig, I didn't. I was debugging why zuul didn't start via this page ... https://etherpad.openstack.org/p/thirdpartyci | 20:01 |
| Swanson | dougwig, ended up here https://www.irccloud.com/pastebin/00tjSkvr/ | 20:01 |
| Swanson | dougwig, and while it DID start it sure started an awful lot of things. | 20:01 |
| jeblair | Clint: that sounds important | 20:01 |
| *** jsavak has joined #openstack-infra | 20:01 | |
| jeblair | Clint: wait, is it good if your lunch money is stolen or bad? | 20:02 |
| dougwig | Swanson: that layout file you posted has a huge number of jobs. i'd focus on the no-op job on ci-sandbox until it's all solid. | 20:02 |
| Clint | jeblair: gertty will get into a state where it keeps sending the same request with n=500&start=500 over and over multiple times a second | 20:02 |
| *** megm has joined #openstack-infra | 20:02 | |
| *** tonytan4ever has quit IRC | 20:02 | |
| Swanson | dougwig, That's what I thought. I wasn't sure if I was supposed to be defining all the jobs there and then limiting them else where or not. | 20:03 |
| jeblair | Clint: neat -- can you save a copy of your gertty.db real quick to make sure we don't lose the ability to reproduce? | 20:03 |
| *** lucasagomes is now known as lucas-pto | 20:03 | |
| dougwig | Swanson: zuul tells jenkins what to build. you don't want it doing that for anything but a test job yet, and in the silent queue at that. | 20:03 |
| *** austin81 has joined #openstack-infra | 20:03 | |
| *** krtaylor has joined #openstack-infra | 20:03 | |
| *** salv-orlando has joined #openstack-infra | 20:03 | |
| Swanson | dougwig, anteaya thank you for the help. Will replace my layout.yaml file with a more appropriate one. | 20:04 |
| anteaya | Swanson: perhaps attend some third party meetings? | 20:04 |
| gtmanfred | mordred: Shrews any chance yall can review this one? https://review.openstack.org/#/c/268404/ for shade | 20:04 |
| anteaya | Swanson: maybe get to know some other operators | 20:04 |
| jeblair | Clint: then can you run it with debug logging enabled, and send me the full output from startup till when it gets into that state? it will be huge, but it should have everything i need to debug and should be safe to share from a privacy pov. maybe a .bz2 via personal email. | 20:04 |
| Clint | jeblair: afraid not, i reverted that change yesterday and kept on using | 20:04 |
| anteaya | avoid the common issues | 20:04 |
| *** e0ne has quit IRC | 20:04 | |
| Clint | jeblair: i do have a debug log with the problem | 20:05 |
| anteaya | Swanson: and stick with the ci-sandbox for a while | 20:05 |
| zaro | pabelanger: i was actually curious about the change with WIP set and still merged. is there such a change? | 20:05 |
| Swanson | anteaya, That would be the plan. Thanks again. | 20:06 |
| fungi | zaro: "WIP" was in the change subject i think | 20:06 |
| anteaya | Swanson: thank you | 20:06 |
| jeblair | Clint: ok. be aware you are probably not getting all the data without that. | 20:06 |
| anteaya | Swanson: thanks for asking for help | 20:06 |
| anteaya | Swanson: perhaps the first thing to work on is confirming when you system is off | 20:06 |
| fungi | zaro: yeah, that's what it was https://review.openstack.org/244835 | 20:06 |
| anteaya | Swanson: so you can shut it off with confidence when you need to | 20:07 |
| Swanson | anteaya, It took me way too long to realize my email was blowing up or I would have shut it off sooner. | 20:07 |
| *** keedya has joined #openstack-infra | 20:07 | |
| anteaya | Swanson: okay but when I asked you if your system was off, you replied with "in theory" | 20:08 |
| anteaya | Swanson: I need you to have more confidence in your ability to know your system is off | 20:08 |
| *** jaosorior has quit IRC | 20:08 | |
| zaro | fungi: ahh, makes sense then. thought it might be a bug in gerrit. | 20:08 |
| anteaya | please consider that a priority | 20:08 |
| Swanson | anteaya, okay. | 20:09 |
| *** dprince has quit IRC | 20:09 | |
| *** salv-orlando has quit IRC | 20:09 | |
| fungi | zaro: nah, just someone overlooking the change subject, which was not in agreement with the workflow votes | 20:09 |
| clarkb | jeblair: https://jenkins04.openstack.org/job/gate-tempest-dsvm-multinode-full/2689/console is failing fwiw with the use public IPs change | 20:10 |
| *** kgiusti has quit IRC | 20:10 | |
| anteaya | Swanson: thank you | 20:10 |
| clarkb | jeblair: and it ran on bluebox with NAT so will have to do some digging into that | 20:10 |
| *** abitha has joined #openstack-infra | 20:11 | |
| *** ociuhandu has quit IRC | 20:11 | |
| Clint | jeblair: log sent, it's seeing that there's more to fetch but it's setting start=500 unconditionally when it should be going to 1000, i believe | 20:11 |
| AJaeger | anteaya: could you review https://review.openstack.org/#/c/269813/ , please? | 20:12 |
| anteaya | AJaeger: in a second | 20:12 |
| AJaeger | sure, thanks | 20:12 |
| anteaya | fungi zaro so I see sandbox has inherited from all projects: https://review.openstack.org/#/admin/projects/openstack-dev/sandbox,access | 20:12 |
| anteaya | so gerrit reports that all registered users have +2 except they don't | 20:12 |
| anteaya | I don't know where else to check or look or change | 20:13 |
| * anteaya hopes someone else also cares about getting sandbox permissions working | 20:13 | |
| clarkb | anteaya: I wsa thinking about that more and it is possible that the all projects acl sets registered users to +/-1 | 20:14 |
| anteaya | clarkb: yes it does | 20:14 |
| clarkb | anteaya: and our acl there isn't overriding. So to fix it we may attempt making that an exclusive group permission | 20:14 |
| anteaya | okay can we try that on -dev? | 20:14 |
| clarkb | anteaya: or maybe the refs don't match causing the ocnfusion and we can make them match | 20:14 |
| anteaya | oh that's interesting | 20:15 |
| anteaya | how might I check the refs don't match theory? | 20:15 |
| fungi | anteaya: zaro: specifically https://review.openstack.org/gitweb?p=All-Projects.git;a=blob;f=project.config;hb=HEAD#l21 is applied to refs/heads/* but so is the equivalent entry on the sandbox acl | 20:15 |
| *** tonytan4ever has joined #openstack-infra | 20:15 | |
| clarkb | looks to be the same ref | 20:15 |
| *** ldnunes has quit IRC | 20:15 | |
| anteaya | yes it is | 20:15 |
| anteaya | sorry I mis-understood your refs don't match point, clarkb | 20:16 |
| clarkb | so I am going to guess we need thta to be exclusive to override the all-projects and if that works file a bug wit hgerrit saying that acl inheritance should override | 20:16 |
| anteaya | yes they both reference refs/heads/* | 20:16 |
| anteaya | clarkb: do we have to create a new group to implement your approach? | 20:16 |
| clarkb | anteaya: no, we just have to update the acl to use exclusive permissions on code-review | 20:17 |
| anteaya | the acl for sandbox? | 20:17 |
| clarkb | yes | 20:17 |
| *** Sukhdev has joined #openstack-infra | 20:18 | |
| anteaya | I can take a try at that | 20:18 |
| *** tiswanso has quit IRC | 20:18 | |
| *** salv-orlando has joined #openstack-infra | 20:19 | |
| clarkb | jeblair: looks like metadata server may not be working in that setup | 20:19 |
| clarkb | and we can't config drive with mulitnode because config drive isn't migrateable | 20:19 |
| *** tiswanso has joined #openstack-infra | 20:19 | |
| *** e0ne has joined #openstack-infra | 20:22 | |
| *** claudiub has quit IRC | 20:22 | |
| openstackgerrit | Merged openstack-infra/project-config: Adds release notes jobs for fairy-slipper, app dev doc tool https://review.openstack.org/269813 | 20:22 |
| *** e0ne has quit IRC | 20:22 | |
| *** tdurakov has quit IRC | 20:23 | |
| openstackgerrit | gordon chung proposed openstack-infra/project-config: fix ceilometer integration job to handle slow gate https://review.openstack.org/269846 | 20:23 |
| *** flwang has joined #openstack-infra | 20:24 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Run instack-test-overcloud https://review.openstack.org/269816 | 20:24 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Deploy with --libvirt-type qemu https://review.openstack.org/269848 | 20:24 |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Add exclusiveGroupPermissions to sandbox's acl https://review.openstack.org/269849 | 20:24 |
| anteaya | clarkb fungi zaro ^^ | 20:25 |
| *** MarkAtwood1 has quit IRC | 20:25 | |
| *** kzaitsev_mb has quit IRC | 20:26 | |
| *** binbincong has quit IRC | 20:27 | |
| anteaya | clarkb: thanks | 20:28 |
| pabelanger | zaro: this was the WIP patch, https://github.com/openstack-infra/system-config/commit/7910a841977a39a20a9b35889216d6bf8d6450bd | 20:29 |
| pabelanger | zaro: this is the new change using puppetlabs-apache https://review.openstack.org/#/c/205596/ | 20:29 |
| zaro | pabelanger: thanks, fungi explained it to me. wip was only on title. | 20:29 |
| *** kzaitsev_mb has joined #openstack-infra | 20:30 | |
| *** binbincong has joined #openstack-infra | 20:30 | |
| *** salv-orlando has quit IRC | 20:32 | |
| *** sabeen2 has quit IRC | 20:33 | |
| *** ZZelle_ has joined #openstack-infra | 20:34 | |
| anteaya | fungi: thanks | 20:35 |
| *** electrofelix has quit IRC | 20:35 | |
| *** denisra_ has quit IRC | 20:36 | |
| *** denisra_ has joined #openstack-infra | 20:36 | |
| *** tdurakov has joined #openstack-infra | 20:36 | |
| *** rguillebert has quit IRC | 20:37 | |
| *** dtardivel has quit IRC | 20:37 | |
| *** gildub has joined #openstack-infra | 20:39 | |
| *** xyang1 has joined #openstack-infra | 20:39 | |
| *** kzaitsev_mb has quit IRC | 20:40 | |
| zaro | anteaya: docs seems to indicate that it should work. | 20:41 |
| anteaya | zaro: what is "it" in your above assertion? | 20:42 |
| anteaya | my change, or the original configuration that doesn't work | 20:42 |
| *** boris-42 has joined #openstack-infra | 20:42 | |
| openstackgerrit | Mathieu Gagné proposed openstack-infra/project-config: Increase internap max servers from 36 to 72 https://review.openstack.org/269857 | 20:42 |
| anteaya | mgagne: thank you | 20:43 |
| mgagne | anteaya =) | 20:43 |
| zaro | anteaya: your change, https://gerrit-review.googlesource.com/Documentation/access-control.html | 20:44 |
| anteaya | :) | 20:44 |
| fungi | zaro: yeah, we're mainly concerned that a project-specific acl is not overriding an equivalent ref section from the all-projects acl | 20:44 |
| mgagne | I find it curious that some jenkins masters are getting more jobs/slaves scheduled to them than others | 20:44 |
| anteaya | mgagne: what are you seeing? | 20:45 |
| fungi | zaro: i had always assumed that project-specific acls override the inherited all-projects acl when there were equivalent sections defined? | 20:45 |
| mgagne | curious lines in graphs: http://grafana.openstack.org/dashboard/db/nodepool | 20:45 |
| *** bardia has quit IRC | 20:45 | |
| *** bardia has joined #openstack-infra | 20:46 | |
| fungi | mgagne: that indicates that jenkins02 is having issues and is probably not responding to nodepool's attempts to add or remove slaves | 20:46 |
| openstackgerrit | greghaynes proposed openstack/diskimage-builder: Perform a booting test for our images https://review.openstack.org/204639 | 20:46 |
| anteaya | mgagne: I'm going with jenkins02 is off doing its own thing for some reason | 20:46 |
| fungi | mgagne: likely the thread leak we battle continually in jenkins (at this point we have a cron job which reboots all our jenkins masters once a week just to work around it) | 20:46 |
| mgagne | we just have to wait for graphs to form a skull shape :D | 20:46 |
| clarkb | fungi: those restarts seem to help but may not be aggressive enough, maybe we should do it every other day | 20:47 |
| anteaya | mgagne: ha ha ha | 20:47 |
| greghaynes | ianw: SpamapS can I get a +3 on https://review.openstack.org/#/c/268318/1 | 20:48 |
| *** nmagnezi has quit IRC | 20:49 | |
| *** moravec has quit IRC | 20:51 | |
| prometheanfire | sure gating is alright, been waiting 8 hours, some are up to 16 | 20:51 |
| zaro | fungi: it's unclear in the doc, but i think it's an addition too what was specified in the parent to narrow the set of groups. although i'm not sure so might be worth testing and making the docs more clear. | 20:51 |
| clarkb | prometheanfire: the infra systems themselves seem to be operating properly | 20:52 |
| clarkb | running >1k jobs per hour | 20:52 |
| clarkb | the jobs themselves are failing at a relatively high rate which some people are addressing | 20:53 |
| prometheanfire | ah, k | 20:53 |
| clarkb | prometheanfire: when jobs fial in the gate everything behind that change must be restarted without that change | 20:53 |
| prometheanfire | that is painful | 20:54 |
| fungi | basically we end up with a serialized testing scenario within a set of projects sharing common integration test jobs | 20:54 |
| *** MarkAtwood1 has joined #openstack-infra | 20:54 | |
| prometheanfire | ya | 20:54 |
| fungi | rather than being able to effectively test them in parallel any longer | 20:54 |
| *** dslev has quit IRC | 20:55 | |
| openstackgerrit | Merged openstack-infra/project-config: Add exclusiveGroupPermissions to sandbox's acl https://review.openstack.org/269849 | 20:56 |
| *** vgridnev has quit IRC | 20:56 | |
| anteaya | madorn lbragstad so we are hoping that 269849 gives sandbox users +2 +A, it will take about 30 minutes to update (I believe) | 20:58 |
| lbragstad | anteaya sweet, thanks for the update! | 20:58 |
| openstackgerrit | Andre keedy proposed openstack-infra/project-config: Add new project(shovel) to openstack https://review.openstack.org/269411 | 20:58 |
| anteaya | lbragstad: thanks for bringing it to our attention | 20:59 |
| *** daemontool_ has quit IRC | 20:59 | |
| anteaya | lbragstad: let's see what actually happens in 30 minutes but I am hopeful | 20:59 |
| *** pc_m has quit IRC | 21:00 | |
| *** annegentle has quit IRC | 21:00 | |
| *** sridhar_ram1 has quit IRC | 21:01 | |
| *** _nadya_ has quit IRC | 21:01 | |
| ianw | clarkb: mind if i re +w the nodepool import patches that rebased? ... i'd dearly love to get a build and some logs for f23! | 21:02 |
| clarkb | ianw: go for it | 21:03 |
| *** kgiusti has joined #openstack-infra | 21:04 | |
| *** kgiusti has quit IRC | 21:06 | |
| openstackgerrit | Merged openstack-infra/project-config: Increase internap max servers from 36 to 72 https://review.openstack.org/269857 | 21:07 |
| *** bpokorny_ has joined #openstack-infra | 21:07 | |
| *** vgridnev has joined #openstack-infra | 21:08 | |
| *** adduarte has joined #openstack-infra | 21:08 | |
| *** dslevin has joined #openstack-infra | 21:09 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Normalize some log lines in builder https://review.openstack.org/269784 | 21:09 |
| *** kzaitsev_mb has joined #openstack-infra | 21:09 | |
| *** pc_m has joined #openstack-infra | 21:10 | |
| *** bpokorny has quit IRC | 21:11 | |
| *** bpokorny_ has quit IRC | 21:11 | |
| mordred | clarkb: turns out that option is only for git format-patch | 21:11 |
| mordred | clarkb: not for git commit | 21:11 |
| *** fitoduarte has quit IRC | 21:11 | |
| mordred | clarkb: so it's totally unuseful | 21:12 |
| mordred | lifeless: ^^ | 21:12 |
| *** bpokorny has joined #openstack-infra | 21:12 | |
| lifeless | mordred: yay git. | 21:13 |
| lifeless | with the experience I had from my last patch I sent there, I'm disinclined to offer one for this | 21:13 |
| jeblair | mordred: that sort of makes a weird sort of kind of sense from the "only add this when you are _sending_ the patch" | 21:14 |
| jeblair | mordred, lifeless, fungi: which does sort of suggest that git-review is a not inappropriate place for a similar change | 21:14 |
| mordred | jeblair: it does | 21:14 |
| *** sputnik13 has quit IRC | 21:14 | |
| mordred | but yeah - I think git review | 21:14 |
| mordred | and I think a git review config option to opt-in to it | 21:14 |
| mordred | so that it has to be a conscious decision | 21:14 |
| mordred | would be nice | 21:14 |
| jeblair | right; should probably _not_ be a .gitreview file | 21:14 |
| mordred | ++ | 21:14 |
| *** tdurakov has quit IRC | 21:15 | |
| mordred | that way if you wanted to do git config --global gitreview.signoff - so be it - that's your choice | 21:15 |
| mordred | or you could do --local | 21:15 |
| jeblair | yep | 21:15 |
| zaro | fungi: i think gerritlib release is safe. i don't see any changes in functionality, all changes are additive. | 21:15 |
| jeblair | those both sound good to me. and it might be worth a code comment about why that particular option should never be handled in .gitreview. | 21:15 |
| openstackgerrit | Merged openstack-infra/nodepool: Revert "Import nodepool.builder late for statsd" https://review.openstack.org/269266 | 21:15 |
| *** jpr has quit IRC | 21:15 | |
| anteaya | oh general infra announcement, magnum sprint is Feb 18-19, they announced in their meeting today, location to be determined | 21:16 |
| openstackgerrit | Merged openstack-infra/nodepool: Remove workaround for early paramiko import https://review.openstack.org/269303 | 21:16 |
| anteaya | I was hoping someone from infra could attend as they seem to really have difficulty with their testing | 21:16 |
| anteaya | how ever I can't make those dates | 21:16 |
| *** amrith is now known as _amrith_ | 21:16 | |
| anteaya | as I'll be in the uk | 21:16 |
| jeblair | ttx, dhellmann: https://review.openstack.org/269862 sorry for the delay | 21:16 |
| anteaya | oh sorry location is sunnyvale | 21:16 |
| *** sdake has quit IRC | 21:16 | |
| anteaya | hoping someone else feels like attending and perhaps offering some testing setup guidance | 21:17 |
| *** gildub has quit IRC | 21:17 | |
| *** tdurakov has joined #openstack-infra | 21:17 | |
| *** gyee has joined #openstack-infra | 21:17 | |
| *** tdurakov has quit IRC | 21:17 | |
| *** eharney has joined #openstack-infra | 21:18 | |
| fungi | zaro: thanks, i'll try to work a gerritlib tag in before or between upcoming server reboots | 21:18 |
| mordred | anteaya: I also cannot attend | 21:18 |
| anteaya | mordred: thanks for considering it | 21:18 |
| anteaya | or at least reading what I wrote | 21:18 |
| *** vgridnev has quit IRC | 21:19 | |
| *** vgridnev has joined #openstack-infra | 21:19 | |
| *** jtomasek has quit IRC | 21:20 | |
| *** vgridnev has quit IRC | 21:20 | |
| pabelanger | jeblair: care to review: https://review.openstack.org/#/c/264994/ templating support for nodepool rackspace | 21:20 |
| openstackgerrit | Khai Do proposed openstack-infra/nodepool: Remove duplicate python-jenkins code from nodepool https://review.openstack.org/259157 | 21:21 |
| *** tdurakov has joined #openstack-infra | 21:23 | |
| *** e0ne has joined #openstack-infra | 21:23 | |
| *** Sukhdev has quit IRC | 21:25 | |
| *** moravec has joined #openstack-infra | 21:26 | |
| *** rguillebert has joined #openstack-infra | 21:26 | |
| openstackgerrit | Merged openstack-infra/project-config: don't do multinode jobs in rax-iad https://review.openstack.org/269799 | 21:27 |
| anteaya | my goodness it is going to be warm in bristol next week | 21:27 |
| *** sputnik13 has joined #openstack-infra | 21:28 | |
| ianw | clarkb / greghaynes: do we need https://review.openstack.org/#/c/268867/3 (Handle dib build failures) before we try doing a restart? | 21:28 |
| greghaynes | ianw: it would be nice to have | 21:29 |
| greghaynes | it is a definite bug, it does still work without it though | 21:29 |
| *** mriedem has quit IRC | 21:29 | |
| *** moravec1 has joined #openstack-infra | 21:29 | |
| SpamapS | so.. | 21:29 |
| ianw | greghaynes: so then also https://review.openstack.org/#/c/269774 ? | 21:29 |
| SpamapS | with the new HORRIBLE gerrit UI, how can I see previous CI results? | 21:29 |
| SpamapS | I don't have a toggle CI button anymore | 21:30 |
| fungi | SpamapS: i don't think that's changed? just click the toggle ci button at the bottom of the page and look in the review comments | 21:30 |
| mtreinish | SpamapS: it's there, in the bottom left of the page | 21:30 |
| greghaynes | SpamapS: there is a patch sets menu on the top right | 21:30 |
| SpamapS | oh its just well hidden | 21:30 |
| greghaynes | also the toggle ci should still be there | 21:30 |
| SpamapS | VERY well hidden | 21:30 |
| fungi | SpamapS: was it cowering in your blind spot? | 21:31 |
| SpamapS | fungi: yeah, used to be a prominent feature | 21:31 |
| SpamapS | now it's as tucked away as possible. | 21:31 |
| *** gordc has quit IRC | 21:31 | |
| SpamapS | I'm probably not giving the UI enough of a chance though | 21:31 |
| SpamapS | too busy puking on it | 21:31 |
| fungi | painted pink with an improbability field erected around it? | 21:31 |
| fungi | improbability field | 21:31 |
| fungi | such a useful tool | 21:31 |
| ianw | greghaynes: ok, not familiar enough with 268867 to merge but maybe clarkb can be convinced | 21:31 |
| greghaynes | heh, I think most folks are not entirely fond of the new UI but not a whole lot we can do :( | 21:31 |
| greghaynes | aside from use gertty | 21:32 |
| greghaynes | :p | 21:32 |
| SpamapS | Oh and I missed ianw's already +A'd that | 21:32 |
| SpamapS | so meh | 21:32 |
| fungi | or join the effort to write the replacement webui | 21:32 |
| SpamapS | greghaynes: I use gertty | 21:32 |
| SpamapS | exclusively | 21:32 |
| *** moravec has quit IRC | 21:33 | |
| SpamapS | I used to casually sometimes use the web UI | 21:33 |
| SpamapS | until this upgrade | 21:33 |
| SpamapS | fungi: you definitely _do not_ want me writing UI's | 21:33 |
| greghaynes | ya, working on the replacement UI has been too low on my todo list for way too long | 21:33 |
| SpamapS | I mean, unless you want code review to look a lot like Zork | 21:33 |
| fungi | also the t key in gertty's default keymap will toggle ci comments | 21:33 |
| fungi | SpamapS: surprisingly, i do, yes! | 21:33 |
| *** e0ne has quit IRC | 21:33 | |
| SpamapS | You are in a python file, it has 6000 lines. You see a diff at line 4509. | 21:33 |
| greghaynes | turn around | 21:34 |
| sdague | SpamapS: I thought toggle ci was hidden the same way as before | 21:34 |
| SpamapS | sdague: I may be just raging on the UI in general and assuming _everything_ was always better before. ;) | 21:34 |
| SpamapS | This new UI needs to get off my lawn. | 21:34 |
| sdague | heh | 21:34 |
| * fungi shakes cane vigorously | 21:34 | |
| *** kzaitsev_mb has quit IRC | 21:35 | |
| *** ihrachys has joined #openstack-infra | 21:35 | |
| *** sridhar_ram has joined #openstack-infra | 21:35 | |
| ianw | SpamapS: it annoys me when it gets less information dense. showing all the merged changes by default (three ticks) is really useless info because they're done and dusted. at least there isn't acres of white-space on either side. anyway, let's not all whinge in here :) | 21:36 |
| *** sdake has joined #openstack-infra | 21:37 | |
| openstackgerrit | Merged openstack/diskimage-builder: Add pkg-map for redhat https://review.openstack.org/268318 | 21:37 |
| SpamapS | No I did not want this to be a UI whambulance. Just needed some details. | 21:37 |
| anteaya | clarkb fungi zaro lbragstad madorn so this is fun: https://review.openstack.org/#/c/269387/ I have successfully removed any permissions for registered users on sandbox | 21:37 |
| anteaya | yay me | 21:37 |
| *** tiswanso has quit IRC | 21:38 | |
| *** dslevin has quit IRC | 21:38 | |
| *** baoli has joined #openstack-infra | 21:38 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Revert "Add exclusiveGroupPermissions to sandbox's acl" https://review.openstack.org/269873 | 21:38 |
| *** ksantoshk has joined #openstack-infra | 21:39 | |
| *** baoli_ has joined #openstack-infra | 21:39 | |
| fungi | anteaya: wow. even as an administrator i can no longer vote on those changes | 21:39 |
| anteaya | the power | 21:40 |
| anteaya | so now we know what not to do | 21:40 |
| anteaya | let me fix the commit message on that | 21:41 |
| *** acruz has quit IRC | 21:41 | |
| fungi | looking at the current acl applied in gerrit for that repo, it seems to have done the things we expected (checked the exclusive checkbox on those permissions) | 21:42 |
| *** tdurakov has quit IRC | 21:42 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Revert "Add exclusiveGroupPermissions to sandbox's acl" https://review.openstack.org/269873 | 21:42 |
| *** santoshk has quit IRC | 21:42 | |
| fungi | but that has obviously not had the desired effect, and implies there's something odd about the registered users group in general | 21:42 |
| *** bpokorny has quit IRC | 21:42 | |
| anteaya | fungi: for some reason it is reading the acl for that file and then just making stuff up | 21:42 |
| *** baoli has quit IRC | 21:43 | |
| fungi | oh... | 21:43 |
| anteaya | fungi: and thanks for checking that effect, as I had not | 21:43 |
| fungi | i see what's going on | 21:43 |
| anteaya | ah good | 21:43 |
| fungi | https://review.openstack.org/#/admin/groups/1230,members | 21:43 |
| *** bpokorny has joined #openstack-infra | 21:43 | |
| anteaya | what is that? | 21:43 |
| anteaya | well obviously a group called Registered Users, but how did it get created? | 21:44 |
| fungi | the empty "Registered Users" group manage-projects just created | 21:44 |
| anteaya | oh I bet we have an acl file derp somewhere | 21:44 |
| anteaya | was that created in response to my patch edit? | 21:44 |
| anteaya | my acl file derp | 21:44 |
| *** dslev has joined #openstack-infra | 21:45 | |
| fungi | i'm not sure | 21:45 |
| *** otter768 has joined #openstack-infra | 21:46 | |
| *** ihrachys_ has joined #openstack-infra | 21:46 | |
| fungi | no, i don't think so, it was created before sahara-scenario-core, the next group number in sequence | 21:46 |
| jhesketh | Morning | 21:46 |
| anteaya | okay so at least it wasn't me, but perhaps its existance is the reason sandbox permissions are... interesting | 21:46 |
| anteaya | morning jhesketh | 21:46 |
| *** ihrachys has quit IRC | 21:46 | |
| fungi | created by https://review.openstack.org/259373 which merged Mon, 21 Dec 2015 10:01:18 +0000 | 21:47 |
| anteaya | jhesketh: when you have coffeed up take a peruse of the meeting logs | 21:47 |
| anteaya | jhesketh: we had some chat about swift logs and want to hear your thoughts | 21:47 |
| jhesketh | anteaya: okay, will do, thanks | 21:47 |
| fungi | anteaya: zaro: it looks like the first change to an acl after our gerrit 2.11 upgrade caused manage-projects to create an empty Registered Users group | 21:47 |
| anteaya | fungi: ah | 21:48 |
| anteaya | jhesketh: thank you | 21:48 |
| *** jsavak has quit IRC | 21:48 | |
| anteaya | fungi: yeah, cause that acl file looks as I would expect | 21:48 |
| *** jsavak has joined #openstack-infra | 21:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 21:49 | |
| *** baoli_ has quit IRC | 21:49 | |
| *** baoli has joined #openstack-infra | 21:49 | |
| *** baoli has quit IRC | 21:50 | |
| *** otter768 has quit IRC | 21:50 | |
| clarkb | ianw: if you review https://review.openstack.org/#/c/269784/2 I can go through and push the whole stack through | 21:50 |
| *** baoli has joined #openstack-infra | 21:50 | |
| zaro | very odd. not sure what's going on. but definately bug somewhere. either on override or exclusive. maybe we can test a little on review-dev | 21:50 |
| clarkb | then I can do a service restart once we get those installed | 21:50 |
| fungi | zaro: see my comment about the empty registered users group. it seems that acl is referencing an actual group rather than the global meta-group for registered users | 21:51 |
| anteaya | madorn: hopefully we get https://review.openstack.org/#/c/269873/ merged shortly, then share the common topic for the patches and hopefully someone can approve and merge them | 21:52 |
| fungi | which explains all the weirdness we're seeing | 21:52 |
| zaro | fungi: yeah, saw that. not sure what's up with that. | 21:52 |
| anteaya | madorn: you seem to have helped us uncover a bug, thank you | 21:52 |
| keedya | hi there | 21:52 |
| keedya | i have a review request for a new project https://review.openstack.org/#/c/269411/ | 21:52 |
| anteaya | fungi clarkb can we get 269873 in the process? | 21:53 |
| keedya | Do I need to add reviewers? | 21:53 |
| *** _ari_ has quit IRC | 21:53 | |
| anteaya | with my gratitude | 21:53 |
| fungi | zaro: so something is 1. allowing us to create a regular group which shadows a built-in group name, and 2. causing the acl to refer to that new group rather than the builtin of the same name | 21:53 |
| clarkb | fungi: oh! | 21:53 |
| clarkb | I bet we have a bug in jeepyb then | 21:53 |
| clarkb | :/ | 21:54 |
| fungi | clarkb: entirely possible | 21:54 |
| anteaya | clarkb: thanks | 21:54 |
| fungi | clarkb: could be a behavior change in gerrit, for example no longer reporting the builting groups in a group listing query or something | 21:54 |
| fungi | s/builting/builtin/ | 21:54 |
| clarkb | fungi: right well it treats them as system groups | 21:54 |
| clarkb | I thought we had addressed that but apparently not | 21:55 |
| anteaya | zaro dougwig thank you | 21:55 |
| fungi | which then caused manage-projects to think it needed to create it | 21:55 |
| clarkb | and since all-projects isn't jeepyb managed it likely uses the correct uuid in its group file so the default +-1 works | 21:55 |
| *** angdraug has quit IRC | 21:55 | |
| fungi | yep | 21:55 |
| *** mrmartin has joined #openstack-infra | 21:56 | |
| clarkb | greghaynes: want to review https://review.openstack.org/#/c/269774/1 before I approve it? | 21:56 |
| *** cburgess has quit IRC | 21:56 | |
| fungi | also if you view the all-projects acl in gerrit you can see it's using the correct group reference for registered users (it's not a clickable url like the one in the sandbox acl) | 21:56 |
| greghaynes | clarkb: oh, I missed that one | 21:56 |
| *** cburgess has joined #openstack-infra | 21:57 | |
| *** eil397 has quit IRC | 21:57 | |
| greghaynes | clarkb: lgtm | 21:58 |
| *** salv-orlando has joined #openstack-infra | 21:58 | |
| *** aeng has joined #openstack-infra | 21:59 | |
| openstackgerrit | Merged openstack-infra/nodepool: Handle dib build failures https://review.openstack.org/268867 | 21:59 |
| *** aeng has quit IRC | 22:00 | |
| clarkb | fungi: anteaya so the quick ninja fix would be to update the groups file but I think jeepyb will overwrite that the next time it runs | 22:00 |
| *** Guest51217 is now known as med_ | 22:00 | |
| *** med_ has quit IRC | 22:00 | |
| *** med_ has joined #openstack-infra | 22:00 | |
| *** peristeri has quit IRC | 22:00 | |
| *** baoli has quit IRC | 22:01 | |
| clarkb | mordred: we just need to trick gerrit into accepting format patch output | 22:01 |
| *** jpr1 has joined #openstack-infra | 22:01 | |
| *** abregman has quit IRC | 22:01 | |
| zaro | lets just fix jeepyb | 22:01 |
| anteaya | fungi: thanks now I see the clickable url, missed that difference before | 22:01 |
| *** baoli has joined #openstack-infra | 22:01 | |
| anteaya | I'm for helping madorn with tonight's class by someone with gerrit admin approving the patches associated with the common-gerrit topic the class is using | 22:02 |
| anteaya | and fixing jeepyb | 22:02 |
| *** jpr1 has quit IRC | 22:02 | |
| openstackgerrit | Merged openstack-infra/nodepool: Also handle work exceptions in builders https://review.openstack.org/269774 | 22:02 |
| *** erikwilson has quit IRC | 22:03 | |
| clarkb | zaro: are you volunteering to do that? | 22:03 |
| anteaya | madorn: when you have that common gerrit topic for your patches, do post it | 22:03 |
| zaro | if anteaya can wait for it | 22:03 |
| anteaya | I can wait sure | 22:03 |
| zaro | ok. i'll take a look.. | 22:04 |
| anteaya | since we have a workaround to help madorn if there is a gerrit admin willing to assist | 22:04 |
| anteaya | if we get the gerrit topic | 22:04 |
| *** sabeen1 has joined #openstack-infra | 22:05 | |
| *** jsavak has quit IRC | 22:06 | |
| *** sabeen3 has joined #openstack-infra | 22:07 | |
| clarkb | https://review.openstack.org/#/c/269784/ is the last nodepool change that needs review before I can restart it | 22:09 |
| * jhesketh finishes reading the logs | 22:09 | |
| jhesketh | It was quite a meeting I missed! | 22:10 |
| *** sabeen1 has quit IRC | 22:10 | |
| *** Sukhdev has joined #openstack-infra | 22:11 | |
| *** ybathia has quit IRC | 22:12 | |
| *** eliqiao has quit IRC | 22:12 | |
| *** baoli has quit IRC | 22:13 | |
| *** eliqiao has joined #openstack-infra | 22:13 | |
| *** baoli has joined #openstack-infra | 22:14 | |
| *** ybathia has joined #openstack-infra | 22:15 | |
| *** sridhar_ram has quit IRC | 22:15 | |
| pleia2 | mordred: noted inline the fix you need for https://review.openstack.org/#/c/249492/ | 22:15 |
| pleia2 | mordred: naturally, it's a single character :) wooo sphinx | 22:16 |
| ianw | clarkb: will do, just waiting for the jenkins | 22:16 |
| clarkb | ianw: roger | 22:16 |
| *** gildub has joined #openstack-infra | 22:17 | |
| anteaya | jhesketh: yes we missed you | 22:17 |
| *** sridhar_ram has joined #openstack-infra | 22:17 | |
| *** tiswanso has joined #openstack-infra | 22:17 | |
| *** erikwilson has joined #openstack-infra | 22:18 | |
| *** erikwilson has quit IRC | 22:18 | |
| *** tiswanso has quit IRC | 22:18 | |
| anteaya | fungi pleia2 jhesketh any one feel like helping by giving this a push? I broke sandbox, this fixes it: https://review.openstack.org/#/c/269873/ | 22:18 |
| *** abregman has joined #openstack-infra | 22:18 | |
| *** mrmartin has quit IRC | 22:18 | |
| anteaya | well to the extend it worked before that patch was merged | 22:18 |
| *** tiswanso has joined #openstack-infra | 22:18 | |
| *** ybathia has quit IRC | 22:19 | |
| pleia2 | anteaya: done | 22:19 |
| *** thorst has quit IRC | 22:19 | |
| anteaya | and I have to offline for dinner and a meeting, will check back after that | 22:19 |
| anteaya | pleia2: thank you | 22:19 |
| anteaya | jhesketh: I do want to hear your thoughts but can't ask now as I don't have time to properly listen | 22:19 |
| fungi | doubly approved for double the approval | 22:20 |
| anteaya | fungi: thank you | 22:20 |
| fungi | sorry, my head is mostly in -incident channel for the moment | 22:20 |
| anteaya | fungi: I understand | 22:20 |
| *** sridhar_ram has quit IRC | 22:20 | |
| anteaya | fungi: I hope to be back by the time you have to restart servers so I can answer the channel but I don't know | 22:20 |
| *** rcernin has quit IRC | 22:21 | |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Run instack-test-overcloud https://review.openstack.org/269816 | 22:21 |
| fungi | anteaya: thanks! | 22:21 |
| *** thorst has joined #openstack-infra | 22:22 | |
| *** sabeen3 has quit IRC | 22:22 | |
| nibalizer | mordred: hi | 22:22 |
| nibalizer | im around now | 22:22 |
| nibalizer | how did puppet apply go | 22:22 |
| *** sabeen has joined #openstack-infra | 22:22 | |
| *** HeOS has joined #openstack-infra | 22:22 | |
| *** Sukhdev has quit IRC | 22:22 | |
| nibalizer | fungi: pabelanger jeblair yes we should move to the new puppetlabs-apache, I think we are waiting on them to release though | 22:23 |
| *** Sukhdev has joined #openstack-infra | 22:23 | |
| *** akshai has quit IRC | 22:24 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/devstack-gate: WIP: Use devstack subunit stream in output https://review.openstack.org/234447 | 22:24 |
| *** trown is now known as trown|outttypeww | 22:25 | |
| *** baoli has quit IRC | 22:25 | |
| *** baoli has joined #openstack-infra | 22:26 | |
| *** thorst has quit IRC | 22:26 | |
| nibalizer | greghaynes: SpamapS the SAS vpn is a different vpn right? | 22:27 |
| *** fitoduarte has joined #openstack-infra | 22:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Add exclusiveGroupPermissions to sandbox's acl" https://review.openstack.org/269873 | 22:27 |
| *** tlian has quit IRC | 22:27 | |
| *** abregman has quit IRC | 22:28 | |
| *** adduarte has quit IRC | 22:29 | |
| *** sdake has quit IRC | 22:30 | |
| *** rossella_s has quit IRC | 22:31 | |
| *** baoli has quit IRC | 22:31 | |
| dmsimard | hey so I have a question about zuul-cloner | 22:31 |
| *** rossella_s has joined #openstack-infra | 22:31 | |
| dmsimard | when run from within a gate job, will it magically clone and checkout the current review patchset or what ? | 22:31 |
| clarkb | dmsimard: yes, though I think you still have to tell it which projects to do it for | 22:32 |
| dmsimard | clarkb: I love that kind of magic. Thanks. | 22:32 |
| *** aeng has joined #openstack-infra | 22:33 | |
| openstackgerrit | Merged openstack-infra/nodepool: Log task name more succinctly https://review.openstack.org/247658 | 22:33 |
| *** jamielennox|away is now known as jamielennox | 22:35 | |
| *** akshai has joined #openstack-infra | 22:35 | |
| *** baoli has joined #openstack-infra | 22:36 | |
| *** davidlenwell has quit IRC | 22:37 | |
| *** tonytan4ever has quit IRC | 22:38 | |
| *** davidlenwell has joined #openstack-infra | 22:38 | |
| *** tiswanso has quit IRC | 22:40 | |
| *** thorst has joined #openstack-infra | 22:40 | |
| openstackgerrit | Brant Knudson proposed openstack-infra/project-config: Keystone run py34 unit tests only on master https://review.openstack.org/269896 | 22:41 |
| *** tiswanso has joined #openstack-infra | 22:41 | |
| *** eil397 has joined #openstack-infra | 22:42 | |
| *** ybathia has joined #openstack-infra | 22:42 | |
| mordred | nibalizer: I have not rolled apply live yet | 22:43 |
| mordred | nibalizer: but it works | 22:43 |
| jeblair | krotscheck: did you get a volunteer to push through the mirror stuff yet? | 22:43 |
| nibalizer | mordred: ok cool | 22:44 |
| *** amitgandhinz has quit IRC | 22:45 | |
| *** thorst has quit IRC | 22:45 | |
| *** bastelfreak is now known as lieblingsrasi | 22:45 | |
| *** lieblingsrasi is now known as bastelfreak | 22:45 | |
| *** baoli_ has joined #openstack-infra | 22:46 | |
| *** dkranz has quit IRC | 22:46 | |
| *** baoli_ has quit IRC | 22:48 | |
| *** baoli_ has joined #openstack-infra | 22:49 | |
| *** baoli has quit IRC | 22:49 | |
| *** austin81 has quit IRC | 22:52 | |
| *** austin81 has joined #openstack-infra | 22:53 | |
| *** mudassirlatif has quit IRC | 22:53 | |
| *** moravec1 has quit IRC | 22:55 | |
| *** moravec has joined #openstack-infra | 22:57 | |
| openstackgerrit | Boris Pavlovic proposed openstack-infra/project-config: Add lib-forward-testing for OSprofiler https://review.openstack.org/269905 | 22:57 |
| *** baoli_ has quit IRC | 23:00 | |
| *** baoli has joined #openstack-infra | 23:01 | |
| *** rkukura_ has joined #openstack-infra | 23:01 | |
| *** mudassirlatif has joined #openstack-infra | 23:01 | |
| clarkb | ianw: https://review.openstack.org/#/c/269784/ has test results | 23:02 |
| ianw | clarkb: ok, just put it in ... that's the last one right? | 23:02 |
| *** rkukura has quit IRC | 23:03 | |
| *** rkukura_ is now known as rkukura | 23:03 | |
| clarkb | ianw: yup | 23:03 |
| clarkb | I will likely restart that service as part of kernel patching reboots | 23:03 |
| *** rockyg has quit IRC | 23:04 | |
| *** sputnik13 has quit IRC | 23:05 | |
| ianw | ok ... i would really love to see some build logs! :) and a fedora back in the gate | 23:06 |
| *** moravec has quit IRC | 23:06 | |
| craige | o/ | 23:06 |
| clarkb | ianw: soon! | 23:06 |
| openstackgerrit | Merged openstack-infra/nodepool: Normalize some log lines in builder https://review.openstack.org/269784 | 23:06 |
| *** dingyichen has joined #openstack-infra | 23:06 | |
| *** sdake has joined #openstack-infra | 23:06 | |
| *** dslev has quit IRC | 23:07 | |
| fungi | jeblair: i said i would try to look at it, but that was before i realized we have kernel updates | 23:08 |
| jeblair | fungi: i want to update my bandersnatch thing and was thinking i might be able to get pypi in afs working in conjunction | 23:09 |
| clarkb | ianw: any idea when centos7 will receive kernel updates for recent cve? | 23:09 |
| clarkb | ianw: best I can tell the latest kernel is from the 6th of januarar | 23:10 |
| fungi | jeblair: sounds worth a shot | 23:10 |
| clarkb | y | 23:10 |
| *** annegentle has joined #openstack-infra | 23:10 | |
| ianw | clarkb: hmm, not off the top of my head | 23:10 |
| jeblair | fungi: i expect to work on bandersnatch today and be ready to push on mirrors for real tomorrow | 23:11 |
| ianw | CVE-2016-0728 ? | 23:11 |
| fungi | zaro: can puppet agent be reenabled on review-dev or were you testing something there that puppet might revert? | 23:11 |
| clarkb | ianw: yes that is the one | 23:11 |
| fungi | ianw: yeah, we're working through it in #openstack-infra-incident but have to wait to do the centos servers | 23:12 |
| *** baoli has quit IRC | 23:12 | |
| jeblair | fungi: oh i missed the red-alert :) | 23:12 |
| *** dimtruck is now known as zz_dimtruck | 23:12 | |
| fungi | heh. it's not really an all-hands-on-deck emergency (yet anyway) | 23:13 |
| fungi | still getting package updates applied everywhere | 23:13 |
| clarkb | ianw: oh there are reports that selinux may be sufficient to protect us (need to check on that more but we do have selinux enabled) | 23:13 |
| zaro | fungi: you can reenable. | 23:13 |
| *** ihrachys has joined #openstack-infra | 23:14 | |
| *** sputnik13 has joined #openstack-infra | 23:14 | |
| fungi | zaro: thanks! | 23:14 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: puppet/spec_helper: run the script from project dir https://review.openstack.org/269914 | 23:15 |
| *** baoli has joined #openstack-infra | 23:15 | |
| *** regXboi has quit IRC | 23:16 | |
| *** ihrachys_ has quit IRC | 23:16 | |
| ianw | clarkb / fungi : so there's the systemtap workaround in https://bugzilla.redhat.com/show_bug.cgi?id=1297475 | 23:18 |
| openstack | bugzilla.redhat.com bug 1297475 in vulnerability "CVE-2016-0728 kernel: Possible use-after-free vulnerability in keyring facility" [High,New] - Assigned to security-response-team | 23:18 |
| *** sputnik13 has quit IRC | 23:19 | |
| *** sputnik13 has joined #openstack-infra | 23:19 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: WIP: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 23:20 |
| *** ihrachys has quit IRC | 23:20 | |
| fungi | interesting that red hat knew about the bug for more than a week but didn't actually have a new package brewing to put out once disclosed | 23:21 |
| ianw | fungi: i think there must be more too it than i can see, the bug is restricted. what i can see from internal responses to important customers is basically a "we're working on it" message. so i guess the answer is "we're working on it" :) | 23:23 |
| openstackgerrit | Clint Adams proposed openstack-infra/project-config: Add periodic job to autogenerate devstack plugins list https://review.openstack.org/269921 | 23:23 |
| fungi | ianw: i figured they were working on it, but thanks for the confirmation! | 23:24 |
| *** baoli has quit IRC | 23:25 | |
| *** yamamoto_ has joined #openstack-infra | 23:27 | |
| *** thorst has joined #openstack-infra | 23:28 | |
| ianw | http://www.gizmodo.com.au/2016/01/theres-a-pop-up-in-n-out-burger-store-in-sydney-today/ | 23:29 |
| * ianw strongly considers less worrying about nodepool logs and more eating | 23:29 | |
| *** alex_xu has quit IRC | 23:30 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/project-config: Add Packstack integration test gate jobs https://review.openstack.org/269918 | 23:30 |
| jhesketh | jeblair, clarkb: so, re the swift logs from this mornings meeting. I thought we discussed this during the last summit and decided we did want to continue. We acknowledged that we're largely reimplementing a posix filesystem but since we were rather close to being there (or at least it seems we're close to having the features devs need) then we should continue | 23:31 |
| *** alex_xu has joined #openstack-infra | 23:31 | |
| openstackgerrit | Daniel Wallace proposed openstack-infra/shade: add the ability to get an object back from swift https://review.openstack.org/269925 | 23:32 |
| *** Swami has quit IRC | 23:32 | |
| fungi | jhesketh: for me the current worry is how we shore this up against malicious changes subverting the write access for the metadata container. do we have an easy way to recover if something scribbles garbage over top of what's there? | 23:32 |
| *** Swami has joined #openstack-infra | 23:32 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:33 | |
| jhesketh | fungi: we could create a script to recalcaulate the metadata in the worst case | 23:33 |
| clarkb | jhesketh: yes I am ok with continuing, but I am apprehensive simply because its been so much work in general (I am still reviewing those changes though) | 23:33 |
| fungi | or am i misreading and the workers don't actually get write access to the metadata? | 23:33 |
| jeblair | jhesketh: hrm, i thought it was more along the lines of "we need to write a routine to query swift to get the last modified time" -- the new container/metadata stuff seems much more complicated. but maybe i'm just mis-remembering. | 23:33 |
| jhesketh | fungi: by looking at the object change times | 23:33 |
| jhesketh | fungi: the workers get write access, yes | 23:33 |
| jhesketh | jeblair: that is discussed as an alternative option in the spec | 23:34 |
| fungi | i also think it's worth making sure we evaluate the effort involved and not fall into a sunk costs fallacy. pressing forward may still make the most sense, but i want to be sure it's not primarily because we've already got so much effort invested | 23:34 |
| jhesketh | the spec has been for review for a while, I've been trying to get attention on it to make sure it was heading in the right way and it seemed like this was what we wanted | 23:34 |
| jeblair | jhesketh: yeah, i just thought we could query things a little better than "traverse the entire system" | 23:34 |
| jeblair | jhesketh: i'm really glad for the spec, because i didn't understand what the zuul changes were about until i started reading it | 23:35 |
| *** watanabe_isao has joined #openstack-infra | 23:35 | |
| openstackgerrit | James E. Blair proposed openstack-infra/infra-specs: Add AFS logs spec https://review.openstack.org/269928 | 23:36 |
| jhesketh | jeblair: I'm not sure how we'd do it accurately without traversing the entire system.. we could make assumptions and take the first object in a path and set the pseudo folders to that, but I don't like it | 23:36 |
| *** xyang1 has quit IRC | 23:37 | |
| jeblair | jhesketh, clarkb, fungi: it at least made me want to step back and ask 'would i want to do swift logs knowing what i know now?' | 23:37 |
| jeblair | so i wrote what i think the afs alternative looks like in https://review.openstack.org/269928 | 23:37 |
| *** ybathia has quit IRC | 23:38 | |
| jhesketh | I probably should know this, but what do we currently have that is using afs? | 23:39 |
| *** austin81 has left #openstack-infra | 23:39 | |
| jeblair | and while i think we can do that pretty quickly -- i also know we've been saying 'afs' a lot but not doing 'afs' a lot, so maybe we should pause this for a week, see if we can get pypi afs mirrors going, and then go from there | 23:39 |
| *** doug-fish has quit IRC | 23:39 | |
| jeblair | jhesketh: nothing seriously yet, i think they have all been lower-than-back-burner priority projects so far | 23:40 |
| clarkb | I am not sure that afs is the right answer fwiw either | 23:40 |
| clarkb | when I tested it the performance was really abd | 23:40 |
| clarkb | that could've been my local connection or poor caching settings though | 23:41 |
| jeblair | it's not exactly iscsi, no | 23:41 |
| *** thorst has quit IRC | 23:41 | |
| clarkb | (also NAT) | 23:41 |
| jhesketh | so the original problem of disk space seems to have been less of an issue anyway right? ie throwing disks and lowing retention has largely solved the problem | 23:41 |
| *** thorst has joined #openstack-infra | 23:42 | |
| jhesketh | maybe we're just wasting effort altogether here and we should focus on more important things | 23:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent https://review.openstack.org/249492 | 23:42 |
| clarkb | jhesketh: I think it has largely solved it if we are ok with 4 momths of logs | 23:42 |
| clarkb | I don't know that we are actually happy with that number | 23:42 |
| jeblair | clarkb: but i do think it maps well to both log serving and pypi mirroring -- both can benefit from caching | 23:42 |
| jeblair | clarkb: i would like us to be able to get back to 6; i don't think i'd push for more than that. | 23:42 |
| jeblair | but yeah, right now we are 'only' using 6TB. | 23:43 |
| jhesketh | clarkb: right, but what number of months are we happy with and how much more disk do we need for that | 23:43 |
| jhesketh | I haven't heard anybody complain about 4 months fwiw | 23:43 |
| jeblair | i think we might even be able to bump back up to 6months now. | 23:43 |
| clarkb | jhesketh: I ran into it today (granted it was 10 months I needed) | 23:43 |
| clarkb | and was easy enough to just make change and get new logs :) | 23:44 |
| jhesketh | clarkb: sure it'll happen.. just thinking through our options | 23:44 |
| *** aeng has quit IRC | 23:44 | |
| *** sflanigan has quit IRC | 23:44 | |
| *** dingyichen has quit IRC | 23:44 | |
| jhesketh | clarkb: right, there's an argument for how useful are the 10month old logs really when you can re-run jobs | 23:44 |
| jhesketh | ie even if you have the 10month old logs you may still be better re-running | 23:44 |
| clarkb | yes especailly if you want ot address an issue today | 23:44 |
| clarkb | since deps change and all that | 23:44 |
| jeblair | jhesketh: the older the job, the less likely you'll be able to re-run it :) | 23:44 |
| jhesketh | sure, but the problem may have shifted | 23:45 |
| jhesketh | I do agree, there is a use for older logs, but exactly how useful I'm less sure on | 23:45 |
| *** mudassirlatif has quit IRC | 23:45 | |
| *** thorst has quit IRC | 23:46 | |
| jeblair | probably not enough to justify an extra 8TB of space for 1 year retention | 23:46 |
| *** otter768 has joined #openstack-infra | 23:47 | |
| clarkb | jeblair: with afs and mirrors I expect that the http server can also act as a cache and keep things very warm. With logs I worry more because we have very large directories and tend to not access them in patterns that work well for caching | 23:47 |
| jeblair | clarkb: there may be a caching benefit for 'recently created logs' the old ones will certainly have to be pulled in over the network | 23:47 |
| mordred | that, of course, is similar to rolling things into colder-storage, fwiw | 23:48 |
| jeblair | (in that people are probably more likely to visit and re-visit recent logs) | 23:48 |
| *** piet has quit IRC | 23:49 | |
| clarkb | we are still talking almost half a TB of logs per week though | 23:49 |
| *** doug-fish has joined #openstack-infra | 23:49 | |
| clarkb | and afs seems to cache on the order of half a gigabyte | 23:49 |
| jeblair | clarkb: .375TB ? | 23:50 |
| mordred | clarkb: afs caches exactly how much you tell it to | 23:50 |
| clarkb | jeblair: ya | 23:50 |
| mordred | clarkb: it's a config setting | 23:50 |
| clarkb | jeblair: with probably some jitter for things that get compressed on the backend | 23:50 |
| jeblair | right, so we can configure like a 50G cache or something if we want | 23:50 |
| mordred | clarkb: in /etc/openafs/cacheinfo ... | 23:50 |
| mordred | clarkb: my laptop has /afs:/var/cache/openafs:500000 set | 23:50 |
| clarkb | mordred: and that doesn't ramp up the cpu/io requirements for the cluster to expire things properly? | 23:51 |
| mordred | if we had web server machines set up to server AFS - I would expect we'd want to dedicate pretty much their entire disk to AFS cache | 23:51 |
| *** Qiming has joined #openstack-infra | 23:51 | |
| clarkb | (I am not sure what is involved in pruning the cache) | 23:51 |
| *** otter768 has quit IRC | 23:51 | |
| mordred | jeblair: ^^ do you know the answer to that? my protocol memory is fading with the lovely Nikka Coffey Grain I just poured | 23:52 |
| jhesketh | so coming back to swift... what metrics should we use to determine if the work should stop | 23:53 |
| fungi | well, also the current logs.o.o giant lvm o' doom has reliability drawbacks | 23:53 |
| mordred | fungi: next year for halloween ... | 23:53 |
| jhesketh | perhaps we first need to decide on how many logs we want to store (ie retention) | 23:53 |
| mordred | fungi: PLEASE go as a giant lvm o' doom | 23:53 |
| fungi | we basically have 14x the chance of the filesystem getting marked offline because we're aggregating a bunch of block devices, each of which is a single point of failure | 23:53 |
| clarkb | jhesketh: I think 6 months seems to be generally agree upon | 23:53 |
| jeblair | callbacks can be discarded; it's possible there may be an lru thing or something to limit them. but even if a callback is dropped, it can be re-established without needing to re-fetch the data (ie, it's just a check to see if it's still up to date) | 23:54 |
| jeblair | clarkb, mordred: ^ | 23:54 |
| *** mudassirlatif has joined #openstack-infra | 23:54 | |
| jeblair | clarkb, mordred: that is a vague answer based on my limited knowledge of the protocol but it's the best i got right now | 23:54 |
| *** bpokorny_ has joined #openstack-infra | 23:54 | |
| fungi | we could at our current utilization push back up to 6 months now according to http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all | 23:54 |
| *** doug-fish has quit IRC | 23:54 | |
| fungi | though that's ignoring things that are stored only on swift i guess? | 23:54 |
| clarkb | fungi: yes | 23:54 |
| fungi | no idea how statistically significant that is | 23:55 |
| jeblair | how much is swift-only? | 23:55 |
| *** bpokorny_ has quit IRC | 23:55 | |
| mordred | I thought we were double-writing everything still | 23:55 |
| clarkb | mordred: we are not, some things are swift only | 23:55 |
| clarkb | mostly infra jobs for dog fooding | 23:55 |
| jeblair | (if it's just our jobs, then.. meh..) | 23:55 |
| fungi | after the qa team took a hatchet to a lot of the logs, the filesystem utilization dwindled dramatically | 23:55 |
| clarkb | jeblair: yes I think that it is basically us. THere may be one or two other jobs that don't double write | 23:55 |
| *** piet has joined #openstack-infra | 23:55 | |
| mordred | fungi: yay qa team | 23:55 |
| fungi | but that aside, i'm still uneasy about having what amounts to a linear array of block devices as a 14-way point of failure | 23:56 |
| fungi | it's already bitten us at least a couple times | 23:56 |
| *** bpokorny has quit IRC | 23:57 | |
| fungi | one iscsi hiccough in the provider's backend causing one of those to get switched read-only and it's a day of lost time rebuilding the logical volume and repairing the filesystem (we've been lucky so far that we haven't just lost the whole shebang) | 23:58 |
| *** yamamoto_ has quit IRC | 23:59 | |
| clarkb | according to http://csg.sph.umich.edu/docs/unix/afs/ performance is proportional to number of files and number of bytes granted I have no idea how up to date that is and what their setup tuning was like | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
