Wednesday, 2014-09-24

*** marcoemorais has quit IRC00:00
*** marcoemorais has joined #openstack-infra00:00
lifelessclarkb: hmm? it should be doing most of its work in ram00:00
lifelessclarkb: we just use one of the default openstack flavors00:00
clarkblifeless: I end up with 4GB of stuff in /tmp and each image itself takes up a ton of data00:01
clarkbone image build is worth like 8GB of disk00:01
lifelessclarkb: there's something wrong there00:01
clarkbwe can't tmpfs because we do/did things that broke in tmpfs00:01
lifelessclarkb: unless you're relaly low ram ?00:01
lifelessoh00:01
lifelessso transient use of disk space could be quite big then00:01
lifelessbut it shouldn't be leaking stuff post build00:01
clarkblifeless: it leaks whenyou run out of disk00:01
clarkblifeless: whcih only makes the problem worse00:02
clarkbI think we might be able to go back to tmpfs'ing though. I will test that in the future (probably not on the nodepool node though because ENORAM)00:02
*** sarob_away is now known as sarob00:03
*** unicell has joined #openstack-infra00:07
*** wenlock has joined #openstack-infra00:08
*** dimtruck is now known as zz_dimtruck00:09
openstackgerritA change was merged to openstack-infra/config: Set the nodepool user and key on dib images  https://review.openstack.org/12360300:11
*** wenlock has quit IRC00:12
clarkbok I have confirmed that /etc/nodepool and /opt/nodepool-scripts are created by the changes I have made00:15
clarkbfungi: ^ any opposition to me self approving? you +2'd the first (it has two +2's) but not the second00:16
*** salv-orlando has quit IRC00:16
fungiback, looking00:19
clarkbjhesketh: http://logs.openstack.org/09/123609/1/check/gate-config-layout/e553dc3/ will probably interest you00:19
*** jergerber has quit IRC00:19
clarkbjhesketh: I am not sure why that happened but it did happen00:20
*** dmsimard_away is now known as dmsimard00:20
jheskethclarkb: hmm, https://jenkins06.openstack.org/job/gate-config-layout/1505/consoleText works for me00:21
*** mwagner_lap has quit IRC00:22
*** JayJ has joined #openstack-infra00:22
* jhesketh watches the recheck00:22
clarkbjhesketh: ya  Iwonder if it is a transient thing with jenkins derping and apache doesn't know how to handle it so it 502s00:22
jheskeththat'd be my guess00:23
fungiclarkb: yeah, go for it. i only +2'd an didn't approve because you were still testing that00:23
clarkbfungi: and the child change as well?00:24
*** dims has quit IRC00:24
fungier, or i'm misreading you... which is the second change?00:24
*** SumitNaiksatam has quit IRC00:24
*** nosnos has joined #openstack-infra00:24
*** dims has joined #openstack-infra00:24
clarkbfungi: https://review.openstack.org/#/c/123611/ child of that first one00:25
fungii think gertty isn't showing it on the other change. thanks00:25
*** otherwiseguy has quit IRC00:26
*** JayJ has quit IRC00:27
fungiapproved00:27
*** JayJ has joined #openstack-infra00:27
clarkbty00:27
clarkbI will approve the first00:28
*** dims has quit IRC00:29
*** ChuckC_ has joined #openstack-infra00:29
*** emagana has quit IRC00:29
*** hemna has quit IRC00:30
*** hdd has joined #openstack-infra00:31
*** JayJ has quit IRC00:32
*** JayJ has joined #openstack-infra00:33
*** ChuckC has joined #openstack-infra00:33
*** ChuckC_ has quit IRC00:33
mikalfungi: you around?00:35
fungimikal: barely00:35
mikalfungi: got time for a quick question in a PM?00:35
fungisure00:36
*** JayJ has quit IRC00:37
ianwlifeless: what is the release policy for d-i-b?  (sorry if this is written down somewhere obvious ... i didn't see it)00:37
*** JayJ has joined #openstack-infra00:38
lifelessianw: we release each week if there are unreleased changes in it00:39
*** todd_dsm has joined #openstack-infra00:39
lifelessianw: more often if there is an urgent thing00:40
*** gyee has quit IRC00:41
morganfainberganteaya, zuul tv is live and just as exciting as last time! :)00:42
morganfainberg(maybe more exciting)00:42
*** JayJ has quit IRC00:42
ianwlifeless: thanks00:42
*** Margaretta_Gorcz has quit IRC00:42
*** JayJ has joined #openstack-infra00:43
* anteaya checks00:44
*** gokrokve_ has quit IRC00:44
anteayamerge-check is a whole new adventure00:44
morganfainbergi know!00:45
*** sarob is now known as sarob_away00:46
*** marcoemorais has quit IRC00:47
*** JayJ has quit IRC00:47
*** dmsimard is now known as dmsimard_away00:47
*** marcoemorais has joined #openstack-infra00:48
*** marcoemorais has quit IRC00:48
*** marcoemorais has joined #openstack-infra00:48
*** marcoemorais has quit IRC00:48
*** marcoemorais has joined #openstack-infra00:49
*** gokrokve has joined #openstack-infra00:49
*** JayJ_ has joined #openstack-infra00:49
*** bhuvan has quit IRC00:51
ianwlifeless: while on d-i-b, without having to look at all 148 files or whatever it is, do you have any general issue with the approach as in the changelog in https://review.openstack.org/#/c/119023/ (https://review.openstack.org/#/c/119023/)?00:51
*** cnesa15 has quit IRC00:52
clarkbI have reenabled puppet on nodepool.o.o00:52
lifelessianw: I have a concern, I've put it in the review00:53
*** JayJ_ has quit IRC00:54
*** mfer has joined #openstack-infra00:54
*** Angie_Rath21 has joined #openstack-infra00:55
*** JayJ_ has joined #openstack-infra00:55
*** r-daneel_ has quit IRC00:56
*** mfer has quit IRC00:59
*** JayJ_ has quit IRC01:00
*** JayJ_ has joined #openstack-infra01:00
*** arnaud has joined #openstack-infra01:01
*** todd_dsm has quit IRC01:02
*** hdd has quit IRC01:02
*** Longgeek has joined #openstack-infra01:03
*** VijayTripathi has quit IRC01:05
*** JayJ_ has quit IRC01:05
*** mmaglana has quit IRC01:05
*** gokrokve has quit IRC01:07
*** mwagner_lap has joined #openstack-infra01:10
*** camunoz has quit IRC01:11
*** boden has joined #openstack-infra01:11
fungiclarkb: thanks01:13
*** lcheng has joined #openstack-infra01:13
*** shashankhegde has quit IRC01:13
*** hdd has joined #openstack-infra01:14
*** boden has quit IRC01:16
*** SumitNaiksatam has joined #openstack-infra01:16
*** changbl_ has joined #openstack-infra01:16
*** david-lyle has joined #openstack-infra01:17
david-lylels01:18
clarkb. .. foo bar foobar test01:19
*** JayJ_ has joined #openstack-infra01:19
david-lyleslow, but accurate :)01:19
*** todd_dsm has joined #openstack-infra01:20
*** yaguang has joined #openstack-infra01:22
*** emagana has joined #openstack-infra01:22
*** zz_dimtruck is now known as dimtruck01:25
*** signed8bit has joined #openstack-infra01:25
*** Angie_Rath21 has quit IRC01:25
*** JayJ_ has quit IRC01:25
*** praneshp has quit IRC01:26
*** JayJ_ has joined #openstack-infra01:26
*** camunoz has joined #openstack-infra01:27
*** VijayTripathi has joined #openstack-infra01:28
*** hdd has quit IRC01:29
*** zns has joined #openstack-infra01:30
*** JayJ_ has quit IRC01:31
*** JayJ_ has joined #openstack-infra01:31
*** ttx has quit IRC01:33
*** jyuso has joined #openstack-infra01:34
*** JayJ_ has quit IRC01:34
openstackgerritA change was merged to openstack-infra/config: Rename nodepool exta-data.d to extra-data.d  https://review.openstack.org/12360901:36
*** JayJ_ has joined #openstack-infra01:38
*** ttx has joined #openstack-infra01:38
*** ttx has quit IRC01:38
*** ttx has joined #openstack-infra01:38
openstackgerritA change was merged to openstack-infra/config: Add a nodepool dib script to create /etc/nodepool  https://review.openstack.org/12361101:39
*** Aurore50 has joined #openstack-infra01:40
*** todd_dsm has quit IRC01:42
*** groknix has quit IRC01:43
*** KanagarajM has quit IRC01:43
*** baoli has joined #openstack-infra01:44
*** cristov_mac has joined #openstack-infra01:44
*** cristov_mac has quit IRC01:44
*** sarob_away is now known as sarob01:47
*** cristov_mac has joined #openstack-infra01:47
*** amotoki has joined #openstack-infra01:47
*** cristov_mac has quit IRC01:47
*** acfleury has quit IRC01:48
*** cristov_mac has joined #openstack-infra01:49
*** cristov_mac has quit IRC01:49
*** cristov_mac has joined #openstack-infra01:50
*** otherwiseguy has joined #openstack-infra01:52
*** hdd has joined #openstack-infra01:52
*** armax has quit IRC01:56
*** sarob is now known as sarob_away01:56
*** ildikov has joined #openstack-infra01:57
*** mmedvede has quit IRC01:59
*** zns has quit IRC01:59
*** ildikov_ has quit IRC02:00
*** ildikov has quit IRC02:01
*** todd_dsm has joined #openstack-infra02:01
*** dims has joined #openstack-infra02:02
*** todd_dsm has quit IRC02:02
*** marcoemorais has quit IRC02:02
*** patrickeast has quit IRC02:04
*** david-lyle has quit IRC02:05
*** david-lyle has joined #openstack-infra02:06
*** pcrews has quit IRC02:06
*** zz_jgrimm is now known as jgrimm02:08
*** gokrokve has joined #openstack-infra02:09
*** david-lyle has quit IRC02:10
*** todd_dsm has joined #openstack-infra02:11
*** lcheng has quit IRC02:12
*** Ryan_Lane has quit IRC02:12
*** camunoz has quit IRC02:13
*** baoli has quit IRC02:15
*** baoli has joined #openstack-infra02:15
openstackgerritIan Wienand proposed a change to openstack-infra/config: Add centos7 image to nodepool d-i-b config  https://review.openstack.org/12359402:16
openstackgerritIan Wienand proposed a change to openstack-infra/config: Initial centos7 support for build-image.sh  https://review.openstack.org/12334302:16
ryanpetrelloanybody know if this is something beyond just a fluke?02:16
ryanpetrellohttp://screens.objects.dreamhost.com/09-23-2014-22-16-36.png02:16
ryanpetrellotest failing to upload to Connecting to static.openstack.org02:16
*** emagana has quit IRC02:17
*** emagana has joined #openstack-infra02:18
*** otherwiseguy has quit IRC02:18
clarkbryanpetrello: possible network problem. can you link to the log?02:18
*** otherwiseguy has joined #openstack-infra02:18
ryanpetrelloclarkb: https://jenkins03.openstack.org/job/gate-pecan-tox-scaffolds-26/4/console02:19
*** lcheng has joined #openstack-infra02:19
clarkbya  Ithink that must be another manifestation of the hpcloud to rax networking trouble02:20
clarkbactually no, scp is master to static.o.o02:20
clarkbso internal rax network trouble? http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=139&rra_id=all doesn't make it look like a persistent problem02:22
*** emagana has quit IRC02:22
*** praneshp has joined #openstack-infra02:22
*** yamahata has joined #openstack-infra02:23
*** todd_dsm has quit IRC02:24
ryanpetrellookay, I'll just try again02:24
ryanpetrellonot a big deal02:24
*** camunoz has joined #openstack-infra02:30
*** praneshp_ has joined #openstack-infra02:30
*** gokrokve has quit IRC02:31
*** gokrokve has joined #openstack-infra02:32
*** praneshp has quit IRC02:32
*** praneshp_ is now known as praneshp02:32
*** dims has quit IRC02:35
*** JadnG has quit IRC02:35
*** sdake_ has joined #openstack-infra02:35
*** dims has joined #openstack-infra02:35
*** gokrokve has quit IRC02:36
*** harlowja is now known as harlowja_away02:36
*** Aurore50 has quit IRC02:38
*** dims has quit IRC02:39
*** SumitNaiksatam has quit IRC02:41
clarkbmorganfainberg: WARNING keystoneclient.httpclient: Failed to retrieve management_url from token is a really interesting thing to log02:43
*** rmcall has quit IRC02:43
*** sarob_away is now known as sarob02:47
*** JadnG has joined #openstack-infra02:47
*** Ryan_Lane has joined #openstack-infra02:50
*** Amie74 has joined #openstack-infra02:51
clarkbit appears that we can build our dib image with tmpfs if we want to do that. though it does't make the process go any quicker and may require a bigger nodepool.o.o node02:54
clarkblifeless: how big of a tmpfs does dib expect?02:54
lifelessclarkb: see the docs02:55
openstackgerritA change was merged to openstack-infra/infra-manual: Changed width of the image at the top of the page  https://review.openstack.org/10886702:55
clarkblifeless: :P02:55
*** maxbit has quit IRC02:55
lifelessI don't often play the RTFM card, but when I do, dos equis02:56
*** maxbit has joined #openstack-infra02:56
morganfainbergclarkb, oh joy02:56
morganfainbergclarkb, haha02:56
clarkbmorganfainberg: I mean its a thing but I didn't expect a management url in the first place02:57
clarkbnot sure why I am being warned02:57
morganfainbergclarkb, yeah super useful thing to warn about02:57
*** sarob is now known as sarob_away02:57
morganfainbergclarkb, maybe "HEY i thought i'd have a management url but it's missing... what do it do with this token?" might be better02:57
morganfainbergclarkb, actually i don't know why that would be logged.02:57
clarkbmorganfainberg: or maybe if I am not trying to do anything managementy it should care?02:57
morganfainbergclarkb, i'd need to go look - but it seems weirdly out of context02:57
clarkb*shouldn't02:58
*** flaviof is now known as flaviof_zzz02:58
morganfainbergclarkb, perhaps it wont mater.02:58
clarkboh comcast02:58
clarkbsome of my packaets currently go through new york to hit google02:58
clarkbgoogle is about 60 miles east of here02:59
morganfainbergclarkb, oh it's just that it can't find the endpoint in the token for identoty02:59
*** armax has joined #openstack-infra02:59
clarkbmorganfainberg: right but why would it expect that02:59
clarkbyou want 99.99% of your users to never get a managemtn url02:59
*** JayJ_ has quit IRC02:59
morganfainbergclarkb, oh this looks like a region thing...03:00
clarkblifeless: ok I have rtfm'd. The docs imply that nodepool will only tmpfs if there is 4GB of ram free?03:00
morganfainbergclarkb, and it may not actually matter03:01
*** homeless has quit IRC03:01
clarkblifeless: my grasp of english at this hour is poor. is that a correctreading?03:01
clarkbmorganfainberg: my operations succeeded03:02
clarkbmorganfainberg: but its spat out those warnings to the console which is silly imo03:02
morganfainbergclarkb, then it's because you don't have a catalog and the warning is the wrong log level03:02
morganfainbergclarkb, in short, totally silly :)03:02
morganfainbergclarkb, and we should fix that.03:02
*** sdake_ has quit IRC03:05
*** jvrbanac has quit IRC03:05
clarkbianw: jeblair fungi mordred nodepool just set its first devstack-trusty-dib node to ready state \o/03:06
*** dimtruck has quit IRC03:06
clarkbthings that I have noted. `sudo nodepool image-build devstack-trusty-dib` makes the dib files in /opt/nodepool_dib owned by root. I have manually chown -R'd to nodepool:nodepool so that nodepool can delete them when the time comes03:06
clarkbnot sure if there is a better way to solve that problem than to say we should use sudo -u nodepool03:07
*** ttx has quit IRC03:07
*** jlk has quit IRC03:07
*** comstud has quit IRC03:07
*** wendar has quit IRC03:07
*** cyeoh has quit IRC03:07
*** vishy has quit IRC03:08
clarkbif we set up workers to do it like we talked about I think it becomes a non issue, because the command whould just hand a job off to a worker whcih would do it with the correct perms03:08
*** wendar has joined #openstack-infra03:08
*** otherwiseguy has quit IRC03:09
openstackgerritA change was merged to openstack-infra/jenkins-job-builder: doc update: add references to yaml local include files  https://review.openstack.org/11147803:10
clarkbwoo my packets between home and irc box are hitting almost 50% packet loss crossing L303:11
clarkbon the link between comcast and L3. This must be primetime netflix derpage03:11
lifelessclarkb: I believe so03:11
clarkblifeless: ty03:12
clarkbin that case I think it is safe for us to at least attempt to tmpfs rather than disale it completely03:12
*** otherwiseguy has joined #openstack-infra03:13
clarkblifeless: previously we are setting the immutable bit on a file but we gave up on that03:13
clarkbwhen we were doing that tmpfs said lolno because its tmpfs. but now that we are not doing that tmpfs should be fine03:13
*** melwitt has quit IRC03:15
*** jgrimm is now known as zz_jgrimm03:16
*** ttx has joined #openstack-infra03:16
*** ttx has quit IRC03:16
*** ttx has joined #openstack-infra03:16
*** armax has quit IRC03:17
*** yaguang has quit IRC03:18
*** radez_g0n3 is now known as radez03:19
lifelessmordred: ping03:19
*** Longgeek_ has joined #openstack-infra03:20
*** rmcall has joined #openstack-infra03:21
*** Longgeek has quit IRC03:23
*** signed8bit is now known as signed8bit_ZZZzz03:24
*** radez is now known as radez_g0n303:25
openstackgerritChaoyi Huang proposed a change to openstack-infra/config: Remove Jenkins jobs of tricircle project  https://review.openstack.org/12362203:30
*** koolhead17 has joined #openstack-infra03:31
*** daya_k has joined #openstack-infra03:32
*** mmaglana has joined #openstack-infra03:33
ianwclarkb: yay!03:35
*** miqui has joined #openstack-infra03:35
*** Amie74 has quit IRC03:39
*** Alford_Cummings has joined #openstack-infra03:40
*** mmaglana has quit IRC03:45
*** mmaglana has joined #openstack-infra03:45
*** stevemar has joined #openstack-infra03:46
*** arnaud has quit IRC03:48
*** mmaglana has quit IRC03:50
*** signed8bit_ZZZzz has quit IRC03:50
*** marcoemorais has joined #openstack-infra03:55
*** marcoemorais1 has joined #openstack-infra03:57
*** yaguang has joined #openstack-infra03:59
*** marcoemorais has quit IRC04:00
*** Ryan_Lane has quit IRC04:03
*** liusheng has quit IRC04:03
*** terryw has joined #openstack-infra04:03
*** liusheng has joined #openstack-infra04:03
*** otherwiseguy has quit IRC04:06
*** garyh has quit IRC04:09
*** jaypipes has joined #openstack-infra04:10
*** markmcclain has quit IRC04:11
*** jpich has quit IRC04:11
*** jpich has joined #openstack-infra04:12
*** Alford_Cummings has quit IRC04:18
*** reed has joined #openstack-infra04:21
*** comstud has joined #openstack-infra04:21
*** zz_dimtruck has joined #openstack-infra04:21
*** jvrbanac has joined #openstack-infra04:21
*** zz_dimtruck is now known as dimtruck04:21
*** vishy has joined #openstack-infra04:22
*** rushiagr_away is now known as rushiagr04:23
*** arnaud has joined #openstack-infra04:24
*** rmcall has quit IRC04:25
*** reed has quit IRC04:25
*** rmcall has joined #openstack-infra04:27
*** dimtruck has quit IRC04:34
*** vishy has quit IRC04:34
*** jvrbanac has quit IRC04:34
*** baoli has quit IRC04:35
*** comstud has quit IRC04:35
*** zz_dimtruck has joined #openstack-infra04:38
*** comstud has joined #openstack-infra04:38
*** jvrbanac has joined #openstack-infra04:38
*** zz_dimtruck is now known as dimtruck04:38
*** vishy has joined #openstack-infra04:39
*** harlowja_at_home has joined #openstack-infra04:40
*** garyh has joined #openstack-infra04:40
*** jvrbanac has quit IRC04:44
*** dimtruck has quit IRC04:44
*** vishy has quit IRC04:44
*** comstud has quit IRC04:45
*** hdd has quit IRC04:46
*** sarob_away is now known as sarob04:47
mordredlifeless: pppppppooooooonnnnnngggggg04:53
*** stevemar has quit IRC04:54
openstackgerritAdam Gandelman proposed a change to openstack-infra/devstack-gate: Bump BUILD_TIMEOUT for both Ironic drivers  https://review.openstack.org/12362904:54
*** stevemar has joined #openstack-infra04:54
*** arnaud has quit IRC04:56
*** sarob is now known as sarob_away04:57
*** rmcall has quit IRC04:58
*** hdd has joined #openstack-infra04:59
*** gokrokve has joined #openstack-infra05:00
*** Longgeek_ has quit IRC05:02
*** e0ne has joined #openstack-infra05:04
*** e0ne has quit IRC05:07
*** KanagarajM has joined #openstack-infra05:10
*** e0ne has joined #openstack-infra05:11
*** boden has joined #openstack-infra05:14
lifelessmordred: so hai05:15
*** shashankhegde has joined #openstack-infra05:16
lifelessmordred: um,I replied to your blog; it may be incoherent. I think I have won over namespaces, see my post to the -dev list (I'd like to unblock zzeeek!)05:16
*** Longgeek has joined #openstack-infra05:17
*** jvrbanac has joined #openstack-infra05:17
*** zz_dimtruck has joined #openstack-infra05:18
*** comstud has joined #openstack-infra05:18
*** zz_dimtruck is now known as dimtruck05:18
*** vishy has joined #openstack-infra05:18
*** boden has quit IRC05:19
*** harlowja_at_home has quit IRC05:19
*** e0ne has quit IRC05:22
*** xyang0 has quit IRC05:26
*** e0ne has joined #openstack-infra05:26
*** e0ne has quit IRC05:27
*** changbl_ has quit IRC05:28
*** mrmartin has joined #openstack-infra05:30
*** stevemar has quit IRC05:31
*** afazekas has joined #openstack-infra05:35
*** lcheng has quit IRC05:46
*** sc68cal has quit IRC05:51
*** arnaud has joined #openstack-infra05:51
*** miqui has quit IRC05:52
*** Longgeek has quit IRC05:52
*** gokrokve_ has joined #openstack-infra05:58
*** shashankhegde has quit IRC05:59
*** gokrokve has quit IRC06:01
*** jcoufal has joined #openstack-infra06:01
*** gokrokve_ has quit IRC06:02
*** YorikSar has quit IRC06:06
*** HeOS has quit IRC06:08
*** YorikSar has joined #openstack-infra06:08
*** k4n0 has joined #openstack-infra06:09
openstackgerritIan Wienand proposed a change to openstack-infra/nodepool: Add meta option for passing meta-data  https://review.openstack.org/12363906:13
ianwclarkb / modred  : ^ first-cut at meta-data setting as discussed for RAX images ... hopefully that is somewhat similar to what you're thinking of :)06:14
*** amcrn has quit IRC06:15
*** boden has joined #openstack-infra06:15
*** flaper87|afk is now known as flaper8706:17
*** lcheng has joined #openstack-infra06:17
*** Longgeek has joined #openstack-infra06:18
*** ayoung has quit IRC06:19
*** boden has quit IRC06:19
*** sdake_ has joined #openstack-infra06:20
*** sdake_ has quit IRC06:21
*** sdake_ has joined #openstack-infra06:21
*** KanagarajM has quit IRC06:21
*** ppai has joined #openstack-infra06:23
*** yamahata has quit IRC06:27
*** yamahata has joined #openstack-infra06:27
*** gokrokve has joined #openstack-infra06:28
*** lcheng has quit IRC06:29
*** gokrokve has quit IRC06:29
*** gokrokve has joined #openstack-infra06:30
*** imcsk8 has quit IRC06:31
*** imcsk8 has joined #openstack-infra06:31
*** gokrokve has quit IRC06:35
*** berendt has joined #openstack-infra06:37
*** nadya has joined #openstack-infra06:37
*** nadya is now known as Guest1615106:38
*** ayoung has joined #openstack-infra06:40
*** berendt has quit IRC06:43
*** jistr has joined #openstack-infra06:46
*** ildikov has joined #openstack-infra06:46
*** vigneshvar has joined #openstack-infra06:48
*** hdd has quit IRC06:52
*** unicell has quit IRC06:53
AJaegerjhesketh, is there a special reason that you did not approve https://review.openstack.org/#/c/122946/ ? Would be great to get that one in withhttps://review.openstack.org/#/c/122076, please06:54
jheskethAJaeger: I think I just wasn't going to hang around at the time to make sure it went thorugh06:55
jhesketh*through06:55
jheskethAJaeger: if you're online, I'll approve it now so you can help debug if it goes wrong06:56
*** Guest16151 has quit IRC06:56
AJaegerjhesketh, I'm around...06:56
*** arnaud has quit IRC06:56
AJaegercould you review 122076 as well, please? 122946 depends on it...06:56
*** ihrachyshka has joined #openstack-infra06:56
*** yfried is now known as yfried|afk06:57
*** ashaeron has joined #openstack-infra06:58
*** berendt has joined #openstack-infra06:58
*** mpaolino has joined #openstack-infra06:59
*** ihrachyshka has quit IRC06:59
*** sdake_ has quit IRC06:59
*** zns has joined #openstack-infra07:00
*** ihrachyshka has joined #openstack-infra07:01
*** ihrachyshka has quit IRC07:02
jheskethAJaeger: 122076 approved07:05
AJaegerjhesketh, thanks a lot!07:05
*** arxcruz has joined #openstack-infra07:06
*** GheRivero_ has joined #openstack-infra07:11
*** GheRivero has quit IRC07:14
*** GheRivero_ is now known as GheRivero07:15
openstackgerritA change was merged to openstack-infra/config: Use python-jobs and publish docs for manila  https://review.openstack.org/12207607:15
openstackgerritA change was merged to openstack-infra/config: Publish python-manilaclient on pypi  https://review.openstack.org/12294607:15
AJaegerjhesketh, Do you have time for one more reivew that would be nice but is not critical, please?  https://review.openstack.org/#/c/121633/07:19
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Use a job-template for the tripleo jobs  https://review.openstack.org/12364907:19
openstackgerritDavid Pursehouse proposed a change to stackforge/gertty: Rename gerrit-gertty.yaml to googlesource-gertty.yaml  https://review.openstack.org/12365007:19
*** denis_makogon has joined #openstack-infra07:20
*** VijayTripathi has quit IRC07:28
*** gokrokve has joined #openstack-infra07:31
*** Longgeek has quit IRC07:31
*** HeOS has joined #openstack-infra07:31
*** jp_at_hp has joined #openstack-infra07:35
*** gokrokve has quit IRC07:35
*** cnesa15 has joined #openstack-infra07:36
*** zns has quit IRC07:37
*** zns has joined #openstack-infra07:38
jheskethAJaeger: Approved :-)07:40
*** amuller has joined #openstack-infra07:40
*** ihrachyshka has joined #openstack-infra07:41
*** MaxV has joined #openstack-infra07:48
openstackgerritA change was merged to openstack-infra/config: Update jobs for zaqar  https://review.openstack.org/12163307:50
*** yfried|afk is now known as yfried07:51
*** Hal has joined #openstack-infra07:52
*** e0ne has joined #openstack-infra07:53
*** Hal is now known as Guest6851007:53
*** boden has joined #openstack-infra07:56
*** Ryan_Lane has joined #openstack-infra07:56
*** Ryan_Lane has quit IRC07:56
*** zns has quit IRC07:56
*** jlibosva has joined #openstack-infra07:58
*** mancdaz has joined #openstack-infra07:59
*** zns has joined #openstack-infra08:00
*** yfried is now known as yfried|afk08:04
*** hashar has joined #openstack-infra08:04
*** pblaho has joined #openstack-infra08:07
*** nadya has joined #openstack-infra08:08
*** nadya is now known as Guest8899208:08
*** hashar_ has joined #openstack-infra08:09
*** mrmartin has quit IRC08:09
*** e0ne has quit IRC08:10
*** hashar has quit IRC08:11
*** praneshp has quit IRC08:11
*** mwhudson has joined #openstack-infra08:12
*** Longgeek has joined #openstack-infra08:12
*** ttx has quit IRC08:16
*** mrmartin has joined #openstack-infra08:16
*** ttx has joined #openstack-infra08:17
*** ttx has quit IRC08:17
*** ttx has joined #openstack-infra08:17
*** lcheng has joined #openstack-infra08:25
*** gokrokve has joined #openstack-infra08:28
*** gokrokve has quit IRC08:33
*** mpaolino has quit IRC08:36
*** zz_johnthetubagu is now known as johnthetubaguy08:36
*** yfried|afk is now known as yfried08:38
*** cnesa has joined #openstack-infra08:39
*** marcoemorais1 has quit IRC08:41
*** bauzas has joined #openstack-infra08:42
bauzasSergeyLukjanov: hi08:42
bauzasSergeyLukjanov: https://review.openstack.org/118651 is already having 2 +2s, could you please +W it ? Thanks !08:43
DinaBelovabauzas, SergeyLukjanov is in the US right now, he's sleeping I guess :)08:44
*** jistr has quit IRC08:45
*** lcheng has quit IRC08:46
bauzasDinaBelova: oooooh :)08:46
bauzasDinaBelova: then no worries :)08:47
DinaBelovabauzas :)08:47
bauzasany infra-core could then bless this patch ? :)08:47
*** andreaf_ is now known as andreaf08:47
*** yfried is now known as yfried|afk08:48
*** e0ne has joined #openstack-infra08:50
openstackgerritA change was merged to openstack/requirements: Bump websockify to >=0.6.0 which fixes bug causing zombie processes  https://review.openstack.org/11475708:52
DinaBelovabauzas, I suppose jhesketh may be somewhere around08:55
bauzasDinaBelova: ok, no worries, will wait for a few hours08:55
jheskethbauzas: I'll take a look08:56
openstackgerritDerek Higgins proposed a change to openstack-infra/config: Use a job-template for the tripleo jobs  https://review.openstack.org/12364908:57
jheskethbauzas, DinaBelova: approved08:58
DinaBelovajhesketh, thank you sir!08:59
*** YorikSar has quit IRC08:59
*** Longgeek has quit IRC09:00
*** YorikSar has joined #openstack-infra09:01
*** BobBall_Away is now known as BobBall09:03
*** jpich has quit IRC09:04
hashar_jhesketh: good sometime zone :]  I got Zuul cloner enabled on Wikimedia production!09:06
jheskethhashar_: woot :-)09:07
hashar_jhesketh: monty/james told me they were going to switch Openstack jobs to it as well. So I guess it is probably safe enough for turbo hipster09:07
*** ociuhandu has quit IRC09:08
*** jistr has joined #openstack-infra09:08
*** derekh has joined #openstack-infra09:09
bauzasjhesketh: awesome, thanks09:09
*** salv-orlando has joined #openstack-infra09:10
*** zns has quit IRC09:12
*** zns has joined #openstack-infra09:12
openstackgerritA change was merged to openstack-infra/config: Enable IRC channel logging for #openstack-fr  https://review.openstack.org/11865109:14
*** andreykurilin_ has joined #openstack-infra09:16
*** loki184 has joined #openstack-infra09:20
*** openstack has joined #openstack-infra09:24
*** gokrokve has joined #openstack-infra09:28
*** gokrokve has quit IRC09:29
*** gokrokve has joined #openstack-infra09:30
*** toni_v has joined #openstack-infra09:32
toni_vHi09:32
toni_vI can not install devstack over icehouse Igot this error: ERROR: openstack Internal Server Error (HTTP 500)09:32
*** gokrokve has quit IRC09:35
AJaegerjhesketh, thanks!09:39
*** miarmak has quit IRC09:39
*** zns has quit IRC09:39
*** dizquierdo has joined #openstack-infra09:39
*** Longgeek has joined #openstack-infra09:41
*** zns has joined #openstack-infra09:42
*** denis_makogon has quit IRC09:43
*** ihrachyshka has quit IRC09:46
*** ildikov has quit IRC09:48
*** Guest88992 is now known as _nadya_09:50
*** rlandy has joined #openstack-infra09:51
*** esker has joined #openstack-infra09:52
*** e0ne has quit IRC09:54
*** hashar_ is now known as hashar09:57
*** unicell has joined #openstack-infra10:00
*** flaviof_zzz is now known as flaviof10:04
*** yfried|afk is now known as yfried10:07
*** esker has quit IRC10:08
*** e0ne has joined #openstack-infra10:08
*** pcm_ has joined #openstack-infra10:12
*** Guest68510 has quit IRC10:14
*** pcm_ has quit IRC10:15
*** pcm_ has joined #openstack-infra10:15
*** ociuhandu has joined #openstack-infra10:17
*** ildikov has joined #openstack-infra10:17
*** cnesa15 has quit IRC10:19
*** cbkyeoh has joined #openstack-infra10:20
*** ihrachyshka has joined #openstack-infra10:20
*** dizquierdo has quit IRC10:23
*** yfried is now known as yfried|afk10:23
*** yfried|afk is now known as yfried10:23
*** cbkyeoh is now known as cyeoh10:24
*** andreykurilin_ has quit IRC10:28
*** gokrokve has joined #openstack-infra10:28
*** gokrokve has quit IRC10:33
*** hashar has quit IRC10:36
*** dizquierdo has joined #openstack-infra10:37
*** sandywalsh has joined #openstack-infra10:39
*** rushiagr is now known as rushiagr_away10:40
*** zns has quit IRC10:40
*** rushiagr_away is now known as rushiagr10:43
*** cnesa has quit IRC10:45
*** dims has joined #openstack-infra10:48
*** dmsimard_away is now known as dmsimard11:00
openstackgerritA change was merged to openstack/requirements: Bump python-cinderclient version to 1.1.0.  https://review.openstack.org/12162411:00
*** ildikov_ has joined #openstack-infra11:02
*** toni_v has quit IRC11:02
*** ppai has quit IRC11:04
*** ildikov has quit IRC11:04
*** esker has joined #openstack-infra11:04
*** esker has quit IRC11:04
*** mwagner_lap has quit IRC11:05
openstackgerritPaul Bourke proposed a change to openstack-infra/jeepyb: Allow expire_old_reviews to be used with suexec  https://review.openstack.org/12369311:08
*** ociuhandu has quit IRC11:08
*** ociuhandu has joined #openstack-infra11:10
*** e0ne has quit IRC11:12
openstackgerritPaul Bourke proposed a change to openstack-infra/jeepyb: Allow expire_old_reviews to be used with suexec  https://review.openstack.org/12369311:13
*** ppai has joined #openstack-infra11:18
*** yaguang has quit IRC11:19
*** koolhead17 has quit IRC11:24
dstufftclarkb: fungi mordred If you upgrade bandersnatch to 1.6.1 you'll be ready when the next version of pip comes out, Christian merged my name normalization patch and released it11:28
*** gokrokve has joined #openstack-infra11:28
*** ociuhandu has quit IRC11:28
*** yfried is now known as yfried|afk11:29
*** gokrokve has quit IRC11:33
*** marcusvrn has joined #openstack-infra11:33
*** yfried|afk is now known as yfried11:33
*** yamahata has quit IRC11:35
*** HeOS has quit IRC11:41
*** e0ne has joined #openstack-infra11:44
*** weshay has joined #openstack-infra11:46
*** dkliban_afk is now known as dkliban11:47
sdagueso something is pretty wrong with zuul at the moment, merge checks are backed up 45 minutes, which I thought was suposed to be super cheap11:48
*** HeOS has joined #openstack-infra11:49
sdagueand we've got a ton of stuff waiting for nodes... but aren't using our nodes11:50
*** mbacchi has joined #openstack-infra11:53
*** baoli has joined #openstack-infra11:55
*** dmakogon_ is now known as denis_makogon11:56
*** p5ntangle has quit IRC12:00
*** p5ntangle has joined #openstack-infra12:01
*** berendt has quit IRC12:02
*** loki184 has quit IRC12:04
*** dims has quit IRC12:05
*** dims has joined #openstack-infra12:05
*** p5ntangle has quit IRC12:06
*** HenryG_afk is now known as HenryG12:09
*** yfried is now known as yfried|afk12:10
*** mrmartin has quit IRC12:10
*** julim has joined #openstack-infra12:11
*** JayJ_ has joined #openstack-infra12:12
*** bswartz has quit IRC12:14
*** adalbas has joined #openstack-infra12:21
*** ppai has quit IRC12:21
*** signed8bit has joined #openstack-infra12:23
*** yfried|afk is now known as yfried12:23
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Allow projects to use oslo.middleware  https://review.openstack.org/12371212:25
*** aysyanne has joined #openstack-infra12:26
*** mrmartin has joined #openstack-infra12:28
*** gokrokve has joined #openstack-infra12:28
openstackgerritDavanum Srinivas (dims) proposed a change to openstack/requirements: Bump requests to 2.2.0  https://review.openstack.org/12371412:28
*** doug-fish has joined #openstack-infra12:28
*** mmedvede has joined #openstack-infra12:29
*** maxbit_ has joined #openstack-infra12:31
*** hdd has joined #openstack-infra12:32
*** hashar has joined #openstack-infra12:32
*** gokrokve has quit IRC12:33
*** maxbit has quit IRC12:33
sdaguefungi: let me know when you get around, the gate is quite... odd12:33
HenryGHi. In openstack-infra/config there is slave_db.pp to install mysql and pg backends with 'openstack_citest' user. I would like this to be done for the machines running the neutron-dsvm-functional job. I am not familiar with puppet and would appreciate some hints.12:34
*** ppai has joined #openstack-infra12:35
HenryGNot sure if that would cause issues for devstack on the node?12:35
*** mwagner_lap has joined #openstack-infra12:36
*** ppai has quit IRC12:37
*** signed8bit has quit IRC12:38
*** hashar has quit IRC12:39
*** Guest68510 has joined #openstack-infra12:39
*** hashar has joined #openstack-infra12:42
openstackgerritThomas Bechtold proposed a change to openstack-infra/config: install_puppet.sh: Initial openSUSE support  https://review.openstack.org/12371712:46
*** eharney has joined #openstack-infra12:47
*** ociuhandu has joined #openstack-infra12:49
*** jyuso has quit IRC12:52
*** pblaho has quit IRC12:52
*** jgallard has joined #openstack-infra12:54
jrollg'morning12:55
*** ayoung has quit IRC12:57
*** mfer has joined #openstack-infra12:57
*** signed8bit has joined #openstack-infra12:58
*** flaviof is now known as flaviof_zzz12:59
*** viktors has joined #openstack-infra12:59
*** dkranz has joined #openstack-infra13:00
*** _nadya_ has quit IRC13:02
*** amuller has quit IRC13:06
*** _nadya_ has joined #openstack-infra13:07
*** bswartz has joined #openstack-infra13:07
*** changbl_ has joined #openstack-infra13:08
*** radez_g0n3 is now known as radez13:11
*** maxbit_ has quit IRC13:11
*** miqui has joined #openstack-infra13:11
*** mjturek has joined #openstack-infra13:17
*** hdd has quit IRC13:18
*** bauzas has left #openstack-infra13:18
*** dkranz has quit IRC13:18
fungisdague: thanks, looking13:18
*** dane_leblanc has joined #openstack-infra13:19
*** mriedem has joined #openstack-infra13:21
fungisdague: so... the periods of jobs not using ready nodes are generally zuul dealing with the merge-check pipeline. there was some talk about making it lower priority than other pipelines (except maybe experimental and silent pipelines) but i don't think anybody' shad time to tweak that yet13:21
*** hdd has joined #openstack-infra13:21
*** yfried is now known as yfried|afk13:21
*** hdd has quit IRC13:21
sdaguefungi: I thought it was super cheap?13:21
*** yamahata has joined #openstack-infra13:21
*** xyang0 has joined #openstack-infra13:21
fungisuper-cheap-ish compared to using a node. calculating a nova merge is still a couple minutes i think13:22
*** zns has joined #openstack-infra13:22
sdagueok, so I saw those at 45 min backup this morning13:23
fungithough also i think we have a few nodes which got left in ready during nodepool restarts last night (picked up jobs in the moments when the daemon wasn't watching events) so i'm clearing those now13:23
*** yamahata has quit IRC13:24
*** yamahata_ has joined #openstack-infra13:24
*** yfried|afk is now known as yfried13:24
fungihrm... we did merge a requirements change ~2 hours ago... so the spike there may have been when it proposed reqs updates13:26
*** markmcclain has joined #openstack-infra13:26
*** _buzz has joined #openstack-infra13:27
*** nosnos has quit IRC13:27
*** nosnos has joined #openstack-infra13:27
*** _nadya_ has quit IRC13:28
*** gokrokve has joined #openstack-infra13:28
*** markmcclain1 has joined #openstack-infra13:28
*** _buzz_ has quit IRC13:29
*** che-arne has quit IRC13:30
*** bknudson has joined #openstack-infra13:30
*** markmcclain has quit IRC13:31
*** dprince has joined #openstack-infra13:31
*** nosnos has quit IRC13:32
*** gokrokve has quit IRC13:33
*** _nadya_ has joined #openstack-infra13:33
sdaguemaybe it would be nice if we rate limitted those, so we do that once a week, and not as a post job13:33
*** yfried is now known as yfried|afk13:34
*** dkranz has joined #openstack-infra13:34
*** skolekonov has quit IRC13:35
sdagueI feel like 10hr gate backup is really high for the state of the world right now, given that logstash is really delayed, it's hard to figure out what's causing it13:35
sdaguefor instance yesterday's rax outage only just showed up13:35
fungiwell, if zuul did merge checks when its mergers weren't busy calculating merges for jobs in higher-priority pipelines, then it wouldn't lead to a lull like that... but also i don't think that's likely to be causing the gate to be running so far behind13:37
fungiand it's certainly not the cause of the logstash backlog13:38
fungiif anything it would be helping logstash catch up by causing us not to run jobs as quickly13:38
fungikeeping in mind that those lulls from bursts in merge-check are letting nodepool catch back up and also letting providers catch up on delete requests, the actual impact to throughput may be minimal13:39
*** koolhead17 has joined #openstack-infra13:39
*** koolhead17 has joined #openstack-infra13:39
fungiand i think clarkb has resolved at this point that the solution to the logstash backlog is either lower log volume or faster disk13:40
*** homeless has joined #openstack-infra13:42
fungialso, we seem to continue to accumulate disassociated floating ips in our hpcloud tenant, causing us to overrun our quota there. not sure if this morning's batch is in any way related to our nodepool restarts last night, but i'll keep an eye on it throughout the day after this cleanup pass13:43
fungi#status log removed 79 disassociated floating ips in hpcloud13:43
openstackstatusfungi: finished logging13:43
*** juzuluag has joined #openstack-infra13:44
fungiand bandersnatch has released a new version which may help with our occasional staleness from hanging at teardown13:45
fungihopefully that doesn't break other things in the process13:45
fungiour puppet manifest just ensures "present" for that, so we could in theory upgrade it on pypi.ord.o.o since it serves the smallest volume of workers and then proceed with the others if that doesn't expose any issues13:49
*** zz_gondoi is now known as gondoi13:53
*** lcheng has joined #openstack-infra13:54
*** flaviof_zzz is now known as flaviof13:54
*** _nadya_ has joined #openstack-infra13:56
*** dangers_away is now known as dangers13:56
pabelangerfungi, yes, I don't keep any nodes ready.  Mostly because of the costs associated with keeping them running 24/713:58
*** Sincler has joined #openstack-infra13:58
openstackgerritVictor Sergeyev proposed a change to openstack/requirements: Add PyMySQL to global requirements  https://review.openstack.org/12373713:58
*** r-daneel_ has joined #openstack-infra13:59
*** hdd has joined #openstack-infra13:59
fungiviktors: ^ i'm very much in favor of that addition, but you may want to consider keeping it at wip status until after the reqs freeze is lifted14:00
viktorsfungi: done14:01
*** vhoward has joined #openstack-infra14:02
viktorsfungi: by the way, when K master will be open?14:02
*** yamahata_ has quit IRC14:02
*** gokrokve has joined #openstack-infra14:03
*** yamahata_ has joined #openstack-infra14:03
*** gokrokve has quit IRC14:03
*** gokrokve has joined #openstack-infra14:03
*** gokrokve has quit IRC14:03
*** gokrokve has joined #openstack-infra14:04
*** yamahata_ has quit IRC14:04
fungiviktors: once rc1 has been tagged for all integrated servers14:04
fungiviktors: i believe that's the point at which we branch off openstack/requirements for the upcoming release14:05
*** yamahata_ has joined #openstack-infra14:05
viktorsfungi: ok, thanks.14:06
*** mmedvede has quit IRC14:06
*** nelsnelson has joined #openstack-infra14:07
*** sabeen has quit IRC14:08
fungiyikes... "CVE-2014-6271: remote code execution through bash" (code execution via envvars inherited over remote login sessions, e.g. ssh)14:09
uvirtbotfungi: ** RESERVED ** This candidate has been reserved by an organization or individual that will use it when announcing a new security problem.  When the candidate has been publicized, the details for this candidate will be provided. (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-6271)14:09
*** gondoi is now known as zz_gondoi14:10
*** maxbit has joined #openstack-infra14:10
*** mmedvede has joined #openstack-infra14:15
BobBallsounds very nasty fungi14:16
fungimostly nasty if you're filtering commands via .ssh/authorized_keys or in your sshd_config but not completely sanitizing the environment too14:18
*** MindfulMonk has joined #openstack-infra14:18
*** sabeen1 has joined #openstack-infra14:18
*** hdd has quit IRC14:21
BobBallWell surely it also allows people to 'piggy back' on someoens SSH session from a compromised box in order to compromise another box?14:21
*** asettle has joined #openstack-infra14:21
*** daya_k has quit IRC14:21
fungisure, though there were already ways to do that14:21
fungi(tweak their path to call a trojaned ssh client/agent for example)14:22
fungithough yes, this could be an arguably slightly more trivial means to the same end14:23
*** lcheng has quit IRC14:24
*** hdd has joined #openstack-infra14:24
fungieither method is generally thwarted by never logging into a higher-security system from a lower-security system14:24
sdaguefungi: https://review.openstack.org/#/c/123742/ - closes one of the top gate bugs by deleting the problematic test14:25
sdagueso a promote might be in order14:25
fungisdague: lgtm. i'll bring it forward14:26
sdagueit looks like if we wait 5 minutes we might land 4 patches first14:26
fungii plan to14:26
*** arnaud__ has joined #openstack-infra14:27
*** zz_jgrimm is now known as jgrimm14:27
sdaguecoolio14:27
sdaguethank you sir14:27
*** rlandy has quit IRC14:27
*** zz_gondoi is now known as gondoi14:27
nibalizerclarkb: you were saying memory use on the puppetmaster under 3.7 is super high right?14:27
funginibalizer: leaking, it seemed14:28
*** ociuhandu_ has joined #openstack-infra14:31
*** ociuhandu has quit IRC14:31
*** ociuhandu_ is now known as ociuhandu14:31
*** david-lyle has joined #openstack-infra14:31
*** david-lyle has quit IRC14:31
*** lcheng has joined #openstack-infra14:31
*** lcheng has quit IRC14:31
*** david-lyle has joined #openstack-infra14:32
funginibalizer: right now, we have 5 ruby processes running as the puppet user consuming between 125 and 500 mib resident, with very little at all shared, and the server only has 2gib ram14:32
fungiand the numbers are slowly climbing14:32
fungisince upgrading all the systems to point at the new puppetmaster, we've had it run out of memory and need restarting twice14:33
*** david-lyle has quit IRC14:33
*** david-lyle has joined #openstack-infra14:34
*** hashar_ has joined #openstack-infra14:34
*** david-lyle has quit IRC14:34
*** david-lyle has joined #openstack-infra14:35
*** mrmartin has quit IRC14:35
*** tty57 has joined #openstack-infra14:36
*** andreaf_ has joined #openstack-infra14:36
*** andreaf has quit IRC14:36
*** andreaf_ is now known as andreaf14:36
*** andreaf_ has joined #openstack-infra14:36
*** hashar has quit IRC14:37
nibalizerokay14:39
nibalizersomeone in #puppet is in about the same boat14:39
*** jp_at_hp has quit IRC14:39
fungiwould tweaking apache's max requests per thread cause passenger to free up accumulated virtual memory?14:39
*** gargola has joined #openstack-infra14:40
*** stevemar has joined #openstack-infra14:40
*** emagana has joined #openstack-infra14:41
jrollfungi, nibalizer, I'm sure it's not on your radar yet but you might be interested in http://puppetlabs.com/blog/puppet-server-bringing-soa-to-a-puppet-master-near-you14:43
*** yfried|afk is now known as yfried14:43
*** pballand has quit IRC14:43
*** andreaf has quit IRC14:43
*** lcheng has joined #openstack-infra14:45
nibalizerjroll: yea, but we want to wait for it to be real software first14:46
nibalizerright now its 0.2.0 or something and they make no promises about the stability of the api14:46
jroll:P14:46
jrolltouché14:46
nibalizeron the other hand14:46
clarkbmeh I still contend the server isnt the issue. The issue is expecting one central thing to compile graphs for everyone when you have a botnet14:47
*** pcrews has joined #openstack-infra14:47
*** jp_at_hp has joined #openstack-infra14:47
fungisdague: promoted that nova change to the front now that anything which wasn't at least an hour out has reported14:47
*** jp_at_hp has quit IRC14:48
clarkbfungi did you see I got the dib slave up last night?14:49
fungiclarkb: i did. i just deleted one from ready and am watching it build again14:49
funginot the image, but just the node14:49
*** mmaglana has joined #openstack-infra14:49
clarkbwould like to get https://review.openstack.org/#/c/123245/ in soon to run tests on those nodes14:50
*** hemnass has quit IRC14:51
*** jp_at_hp has joined #openstack-infra14:51
mtreinishclarkb: do you know if there is a better way to do?: https://review.openstack.org/#/c/123589/14:51
EmilienMcould I have a review on https://review.openstack.org/#/c/120285/ ? It has been a while, and I already have +2, +1 reviews. Thanks!14:51
EmilienMit's about "Introduce tripleo-puppet-elements into Stackforge"14:51
fungialso currently trying to figure out why puppetmaster.o.o never shows up in the cacti index and has no rrd files... the puppet log notes (/Stage[main]/Openstack_project::Cacti/Openstack_project::Cacti_device[puppetmaster.openstack.org]/Exec[cacti_create_puppetmaster.openstack.org]/returns) executed successfully14:51
*** hashar_ has quit IRC14:52
*** yfried is now known as yfried|afk14:53
*** maxbit has quit IRC14:53
clarkbmtreinish: I do not. maybe use two tox.ini files?14:53
mtreinishclarkb: heh, yeah that's what I was thinking too, but I think that would probably be more confusing14:54
*** yfried|afk is now known as yfried14:54
*** emagana has quit IRC14:55
*** datsun180b has joined #openstack-infra14:55
nibalizerfungi: so if im looking at this right... we don't cacti graph on puppetmaster.o.o ?14:55
*** emagana has joined #openstack-infra14:55
funginibalizer: we should, but for some reason it's not working. i'm digging into why14:55
openstackgerritA change was merged to openstack-infra/release-tools: Skip tagging if tag exists  https://review.openstack.org/12243814:56
*** hashar has joined #openstack-infra14:57
nibalizerhrm yea, puppetmaster.o.o is in the big list of machines14:57
*** andreaf has joined #openstack-infra14:57
*** groknix has joined #openstack-infra14:57
fungiAHA! elementary my dear nibalizer...14:58
fungi"This host already exists in the database (puppetmaster.openstack.org) device-id:14:58
fungi (85)"14:58
nibalizero.O14:58
fungiwe used to have a puppetmaster.o.o in the beforetime, in the long, long ago14:58
*** yfried has quit IRC14:59
fungithe fun bit is that /usr/local/bin/create_graphs.sh puppetmaster.openstack.org exits 0 after erroring in this fashion, so puppet reports it as having succeeded14:59
*** emagana has quit IRC14:59
nibalizerah cool14:59
*** afazekas has quit IRC15:00
sdaguefungi: thanks15:01
*** juzuluag has quit IRC15:02
*** juzuluag has joined #openstack-infra15:03
*** hashar_ has joined #openstack-infra15:03
*** signed8bit has quit IRC15:04
*** signed8bit has joined #openstack-infra15:04
*** hashar has quit IRC15:04
*** sabeen1 has quit IRC15:05
*** berendt has joined #openstack-infra15:06
*** juzuluag has quit IRC15:07
*** ZZelle has quit IRC15:08
*** ZZelle has joined #openstack-infra15:08
*** juzuluag has joined #openstack-infra15:09
nibalizerfungi: so it doesn't look like plabs is tracking/aware of the memory leak issue15:13
nibalizerbut someone else in #puppet has this problem on centos so its likely a real proble15:13
nibalizerwould be awesome to have some graphs of the memory use15:14
yolandaclarkb, got something working for dib and debian, how are things with dib in trusty? they work fine?15:14
*** che-arne has joined #openstack-infra15:15
clarkbyolanda yes it is working now after a few more patches15:15
yolandapatches for nodepool or for config?15:15
clarkbneeded to update the elements to match nodepools expectations15:15
openstackgerritA change was merged to openstack-infra/subunit2sql: Add get_failing_from_run db api method  https://review.openstack.org/11786015:15
clarkbthe config15:15
yolandaclarkb, ok, i'll take a look to update mine as well15:15
*** james_li has joined #openstack-infra15:15
*** terryw has quit IRC15:16
clarkbwe had to create /etc/nodepool and we had a misnamed run parts dir15:16
yolandabtw, i filed the debootstrap install to debian element, but got the right answer, debootstrap is not needed in the element but in the host running dib, so if we want to build Debian images in nodepool we need debootstrap package15:16
clarkbI see15:17
fungiright, debootstrap runs from outside the chroot15:17
fungiit creates a chroot with a base debian packageset installed in it15:17
fungiand minimal necessary configuration in /etc (hostname, passwd/groups, et cetera)15:18
yolandaso if we need to build debian images we need it as a req15:18
fungiits operation is sort of a subset of dib's15:19
fungialso note that there is at least one debootstrap competitor, cdebootstrap (in typical fashion they tend to leapfrog one another so i can't remember which one is recommended these days)15:20
yolandaoh, didn't know about cdebootstrap15:20
yolandaalways been using debootstrap package and i installed that into my setup as well15:20
*** VijayTripathi has joined #openstack-infra15:20
*** jistr has quit IRC15:21
fungigiven that i spend a lot of time cross-building base filesystem images for embedded devices from other systems, i've tended to use both over time15:21
*** dizquierdo has quit IRC15:21
*** IAX has quit IRC15:22
nibalizerfungi: how many vcpus does the master have?15:22
fungimost recent upload of cdebootstrap into sid was a little under two months ago, so it's still under active development15:22
funginibalizer: 215:23
yolandaclarkb, fungi, are we interested in debian images for nodepool upstream?15:23
nibalizerfungi: so i think we just want more cpus and more rams, is that a problem? can we make it 4cpu and 4Gig?15:23
yolandai saw some work for Centos but nothing for Debian atm, right¿15:24
funginibalizer: it's doable, though "upgrading" means building another instance and copying a lot of stuff over to it. could be a useful exercise to see what gaps we still have in puppet master deployment automation, but it will not be a light undertaking15:24
clarkbyolanda: right now we havent been asked to test debian. so unless that changes probably not15:25
nibalizeroh :/15:25
clarkbnibalizer we shouldnt need to...15:25
clarkbwe have < 80 nodes15:25
yolandaclarkb,ok, so i'll keep that as a local change then15:25
*** david-ly_ has joined #openstack-infra15:25
fungiyolanda: the "project we" is not particularly interested because we don't have current demand for it. the "royal we" (me) is of course quite interested to see debian gain better representation15:26
fungibut i'm unfortunately lacking in time to help make that come about15:26
*** jergerber has joined #openstack-infra15:26
jeblairis the change just to install debbootstrap?  that probably won't kill us... :)15:26
yolandajeblair, for debian wheezy there weren't more issues15:27
*** david-lyle has quit IRC15:27
*** ashaeron has quit IRC15:27
yolandai sent some patches in the past for debian as well15:27
fungiif that's the only effective cost to adding an experimental debian dib flavor in our nodepool, then seems worth the trivial effort15:27
*** sabeen has joined #openstack-infra15:28
clarkbya I wouldnt be opposed to it15:28
yolandafungi, i need to double check that i don't have any other custom things, maybe some puppet issues15:29
fungiout of curiosity, what tool makes the base ubuntu chroot for dib? is there some ubuntu equivalent to (c)debootstrap or is this a reinvented wheel in dib itself?15:29
fungii mean, i've used debootstrap to create ubuntu chroots, so i know it does work15:29
*** marun has joined #openstack-infra15:29
*** rushiagr is now known as rushiagr_away15:29
*** jgallard has quit IRC15:29
*** loki184 has joined #openstack-infra15:30
fungibut if we don't have it installed on our nodepool server then presumably that's not what it's doing15:30
clarkbubuntu publishes base images15:30
clarkbso you chroot into that15:30
fungioh, dib starts from an official ubuntu cloud image?15:30
clarkbyes15:30
fungiwe could presumably so the same with debian's then15:30
*** mmaglana has quit IRC15:31
yolandawe grab an ubuntu image, and the only extra install i see there is install linux-image-generic package15:31
*** mmaglana has joined #openstack-infra15:31
yolandaReadme of debian element says Use of this element will also require the tool 'debootstrap' to be available on your system. It should be available on Ubuntu, Debian, and Fedora.15:32
fungirightscale has been hosting these on behalf of the debian cloud team... http://support.rightscale.com/Community/RightScale_OSS#Debian_GNU.2FLinux_MultiCloud_Images15:33
*** dizquierdo has joined #openstack-infra15:34
*** maxbit has joined #openstack-infra15:34
fungithough those are pretty ancient15:34
*** james_li has quit IRC15:34
fungiyeah, bootstrap or one of the cloud image build tools in debian are probably the best bet for a backend implementation for now it looks like15:35
*** ociuhandu has quit IRC15:35
*** ihrachyshka has quit IRC15:36
*** mmaglana has quit IRC15:36
*** rushiagr_away is now known as rushiagr15:37
fungiaiui there's a plan to start publishing images generated with the build-debian-cloud tool on the mirrors alongside cd images15:39
tristanCHello folks, I wonder if this bash bug (http://seclists.org/oss-sec/2014/q3/651) is exploitable through gerrit...15:40
*** Sukhdev has joined #openstack-infra15:40
fungitristanC: unlikely since you don't ssh into a shell when you connect to the gerrit "ssh" api15:41
fungitristanC: it uses a java-based ssh service to provide the protocol implementation and feeds that directly into some custom command parser15:41
clarkband I dont think there is any env interpretation15:42
tristanCfungi: clarkb: Good news then :)15:42
fungitristanC: the one place we do currently need to keep an eye on and think about is our ansible implementation, but that would still require someone compromising our puppetmaster first and then using that to gain slighthly elevated privileges through our authorized_keys command filters15:42
*** mrmartin has joined #openstack-infra15:42
clarkbfungi which is easier via puppet/ansible directly :)15:43
fungiclarkb: indeed15:43
*** emagana has joined #openstack-infra15:45
rcarrillocruzhey folks15:45
rcarrillocruzjjb question15:45
rcarrillocruzis it possible to invoke a job template from a job template15:46
rcarrillocruzuse case: i have job template foo that does something15:46
*** ildikov_ has quit IRC15:46
rcarrillocruzbut i want a specialized version that does the same thing but install a package on the node before it runs15:46
rcarrillocruzso is it possible to create a new job template with a shell builder that does the package install and then invokes the generic job?15:47
*** denis_makogon has quit IRC15:47
*** sabeen3 has joined #openstack-infra15:47
rcarrillocruzi thought about a job group, but me thinks the jobs contained run on different nodes15:47
*** viktors is now known as viktors|afk15:47
clarkbyou can chain jobs with zuul or a jenkins plugin. or you can have two templates. or you can pass a parameter to a single template that optionally enables additional behavior15:48
rcarrillocruzclarkb: and the chain jobs assures they run on the same node?15:48
rcarrillocruz- job15:48
*** sabeen has quit IRC15:49
jeblairor you can make the builder a macro15:49
rcarrillocruz - job15:49
rcarrillocruz?15:49
clarkbno it doesnt ensure that15:49
rcarrillocruzjeblair: right, but then i'd put that macro in the general job-template15:50
*** jlk_clone has joined #openstack-infra15:50
*** arnaud__ has quit IRC15:51
*** marcusvrn has quit IRC15:51
rcarrillocruzoh, you mean making a macro the builder from the general job-template15:52
rcarrillocruzso it can be called from both job templates, generic and specialized15:52
jeblairyep, so both are very simple definitions.  i'd probably either do that or clarkb's template-with-option idea.15:53
fungifun. someone leaked the prenotification for the embargoed bash vulnerability and it ended up getting quoted in a news article15:53
*** vigneshvar has quit IRC15:54
rcarrillocruzk, thx guys :-)15:55
*** arxcruz has quit IRC15:57
*** rfolco has quit IRC15:58
*** emagana has quit IRC15:58
*** emagana has joined #openstack-infra15:59
jeblairclarkb, fungi, mordred, SergeyLukjanov, jhesketh: i'm going through the config review queue looking for changes that would be affected by the freeze that could be merged (eg, not WIP, and not Verify-1)15:59
zaromorning15:59
jeblairclarkb, fungi, mordred, SergeyLukjanov, jhesketh: i'm recording them here: https://etherpad.openstack.org/p/config-queue-flush16:00
SergeyLukjanovjeblair, ack16:00
fungijeblair: good idea. i was starting to do that in an ad-hoc manner but coordination will help a bunch16:00
fungihitlist style makes great sense for this16:00
jeblairi'm also just approving the ones that already have >=1 +2 (and not etherpadding them)16:00
fungiwfm16:01
*** emagana has quit IRC16:01
*** emagana has joined #openstack-infra16:01
*** JayJ_ has quit IRC16:01
fungijeblair: did you use a filename pattern search to start working through them? i should have thought of that before now16:01
*** JayJ_ has joined #openstack-infra16:01
*** hashar_ is now known as hashar16:01
jeblairfungi: no; i'm just going by subject names.  that's a fine idea though.  :)16:02
jeblair(_somebody_ hasn't gotten around to figuring out how to do filename searches in gertty...)16:02
*** hashar is now known as hasharConfCall16:03
*** k4n0 has quit IRC16:03
openstackgerritA change was merged to openstack-infra/config: add experimental job for nova with f20 virt-preview  https://review.openstack.org/11335016:04
*** otherwiseguy has joined #openstack-infra16:04
*** clayg has quit IRC16:04
*** jlk_clone has quit IRC16:05
*** mmaglana has joined #openstack-infra16:05
*** jlk_clone has joined #openstack-infra16:05
jeblairmordred, dhellmann: i'd like someone else to ack that https://review.openstack.org/#/c/120269/ is desired16:05
jeblairmordred, dhellmann: who should that person be?16:05
*** jlk_clone is now known as jlk16:07
*** jlk has joined #openstack-infra16:07
*** jlibosva has quit IRC16:08
dimsjeblair: zzzeek would be my guess16:10
jeblairdims: i'm pretty sure he doesn't want anything to do with that project :)16:11
dimslol ok :)16:11
jeblairfungi, clarkb: i'm not sure i love the direct reference to an openstack core team from a stackforge project in 12028516:12
*** liusheng has quit IRC16:12
*** liusheng has joined #openstack-infra16:12
*** patrickeast has joined #openstack-infra16:13
mordredjeblair: looking16:13
clarkbmaybe tripleo wants it? if not then ya I think I agree16:14
*** rfolco has joined #openstack-infra16:15
jeblairEmilienM: i gave 120285 a -1; if you're around, let's try to discuss and resolve that today so we can get it merged16:16
*** markmcclain1 has quit IRC16:16
*** markmcclain has joined #openstack-infra16:16
*** pballand has joined #openstack-infra16:17
*** VijayTripathi has quit IRC16:18
*** james_li has joined #openstack-infra16:18
fungiagreed. either it's part of openstack managed by official teams, or it's in stackforge and has its own groups which just happen to contain the same reviewers16:18
*** stevemar has quit IRC16:20
mordredjeblair: unsure ... I'm fine with it, but the core team there is mushy16:20
zaroclarkb: jenkins has verified,  i got a report from someone that could not sign into storyboard when their username on gerrit is 'c-launchpader'.  that username was assigned by launchpad from the16:20
zaroclarkb: opps. ignore that.16:21
fungijeblair: on the other hand, in the name of prior art, note that stackforge/windows-diskimage-builder has acl-config: /home/gerrit2/acls/openstack/diskimage-builder.config16:21
zaroclarkb: jenkins has verified, https://review.openstack.org/#/c/11634716:21
fungigranted, that project was fairly contentious as well, so i think we probably need to figure out an appropriate stance on the "blending" of stackforge and openstack projects16:22
jeblairfungi: drat, i thought i checked that16:22
jeblairmy grep must have been off16:22
*** flaviof is now known as flaviof_zzz16:22
jeblairhappy to fix that though :)16:22
fungii wholeheartedly support making it use its own separate acl definition with separate group names which just happen to contain the same gerrit users16:23
*** MaxV has quit IRC16:25
fungifor a counterexample, the stackforge/xstatic-.* projects _do_ have their own separate acl and group names, even though they're presumably mostly the same horizon reviewers in the end16:26
openstackgerritDavid Lenwell proposed a change to openstack-infra/infra-manual: Add an introduction paragraph to the developer guide  https://review.openstack.org/10702416:29
*** hasharConfCall has quit IRC16:32
*** r-daneel_ has quit IRC16:33
*** afazekas has joined #openstack-infra16:33
*** IAX has joined #openstack-infra16:34
*** radez is now known as radez_g0n316:34
*** openstackgerrit has quit IRC16:35
fungifwiw i queried gerrit for "status:open project:openstack-infra/config file:^modules/openstack_project/files/.*" for a quick and dirty list, and am working my way up from oldest-modified first while i gnaw on some lunch. any which i +2 but don't have an existing +2 already i'll add to the etherpad16:38
*** finch has joined #openstack-infra16:40
*** Guest68510 has quit IRC16:40
finchclarkb: so puppet 3.7.1 huh?16:40
*** bswartz has quit IRC16:40
nibalizerclarkb: fungi ^^ tell finch the tale of the puppet memory leaking16:40
nibalizerhe is puppet core developer16:40
finchclarkb: if possible could you downgrade to 3.7.0 to help pin down what may have introduced the bloat?16:41
*** daya_k has joined #openstack-infra16:41
sdaguetox question for folks16:42
finchI'm at puppetconf right now so I'll be randomly around16:42
*** ramashri has joined #openstack-infra16:42
*** arnaud has joined #openstack-infra16:43
*** bhuvan has joined #openstack-infra16:45
funginibalizer: finch it's 3.7.0-1puppetlabs1 technically16:45
zarojeblair: i think hashar wanted to review this one.  https://review.openstack.org/#/c/8960316:45
*** jistr has joined #openstack-infra16:45
funginibalizer: finch: i don't believe we've tried 3.7.1 yet16:45
nibalizeroh i thought we were on 3.7.116:45
nibalizeroh weird16:46
nibalizerhttp://puppetboard.openstack.org/fact/puppetversion16:46
jeblairzaro: +3 thx16:46
funginibalizer: yeah, seems the master hasn't upgraded16:47
finchfungi: aaah, interesting. so did you just jump to 3.7.0?16:47
*** afazekas has quit IRC16:47
fungifinch: from 2.7, yeah16:47
jgriffithfungi: great summary on that sec issue, thanks!16:47
fungijgriffith: yw16:48
nibalizeri would have thought unatended upgrades would have uped it16:48
*** ihrachyshka has joined #openstack-infra16:48
funginibalizer: me too16:48
*** ihrachyshka has quit IRC16:48
jeblairmorganfainberg: is keystoneclient-federation going in the identity program in governance?16:48
vishyjgriffith, sdague: have you seen cinder hang on create in the gate before? http://logs.openstack.org/22/119522/9/check/check-tempest-dsvm-postgres-full/46a5f8c/logs/screen-c-vol.txt.gz16:48
finchfungi: oh word, that's a pretty hefty jump. for things like this I would like to know which version introduced the issue, would you be able to walk through Puppet versions to nail down which version is the culprit?16:48
morganfainbergjeblair, it should be, let me post a change for it.16:48
finchfungi: also you should make clarkb do it because his suffering nourishes me :)16:48
sdaguevishy: can you like the line in question?16:48
nibalizerfungi: if you do this https://docs.puppetlabs.com/guides/rest_api.html#status you can authoritatively tell what the master is running16:49
vishysdague: it is the very last line16:49
sdaguevishy: click on the date16:49
vishysdague: it never finishes the last create in the file16:49
vishyand apparently cinder-volume stops reporting its data at that point16:49
*** marun has quit IRC16:49
fungifinch: anyway, we have puppet-passenger running under apache on ubuntu trusty, and see ruby processes owned by the puppet user steadily growing in memory usage... this is on a vm with 2g memory and 2x vcpus16:49
vishybecause all future creates fail to schedule16:49
sdaguehmmmm16:50
sdague55 fails is usually glance is dead16:50
jeblairfungi, clarkb: i just realized i should add changes i aprv to the etherpad too, so we can track merge failures.16:50
funginibalizer: Forbidden request: localhost(127.0.0.1) access to /status/puppetclient [find] at :11916:51
clarkbsorry I stepped away to finish booting my morning16:52
*** yamahata_ has quit IRC16:52
sdaguevishy: it looks like iscsi dies in apparently interesting ways - http://logs.openstack.org/22/119522/9/check/check-tempest-dsvm-postgres-full/46a5f8c/logs/syslog.txt.gz16:52
*** shashankhegde has joined #openstack-infra16:52
clarkbfinch: I don't think it is likely that we would walk through versions16:53
finchfungi: passenger should regularly terminate and restart ruby workers, how have you tuned passenger?16:53
finchclarkb!16:53
clarkbit should be easily reproduceable. install puppet via puppetlabs repo. <- thats it really :)16:53
nibalizerfungi: yea you have to do the --cert `puppet config print hostcert` --key `puppet config print hostprivkey` -k thing16:53
*** bswartz has joined #openstack-infra16:53
clarkbfungi: we haven't tuned it. we installed your puppet passenger package16:53
*** jamielenz has joined #openstack-infra16:53
jgriffithvishy: no, that's a new one for me16:53
*** jamielenz is now known as jamielennox16:53
*** shashankhegde has quit IRC16:53
morganfainbergjeblair, https://review.openstack.org/12378616:54
finchthose settings _should_ work. clarkb how long lived are the ruby processes that passenger is launching?16:54
*** dtroyer has quit IRC16:54
*** kylo has joined #openstack-infra16:54
clarkbfungi: one is as old as yesterday16:54
clarkbone from ~10 hours ago16:55
*** _nadya_ has quit IRC16:55
vishysdague: smoking gun: http://logs.openstack.org/22/119522/9/check/check-tempest-dsvm-postgres-full/46a5f8c/logs/syslog.txt.gz#_Sep_22_21_19_0116:55
*** arborism has joined #openstack-infra16:55
sdaguevishy: yeh, that lines up with the fails16:55
vishyjgriffith: looks like an lvm bug16:55
jeblairfungi: thoughts on anteaya's comment in 122220?16:56
sdaguerealistically the c-vol log runs way past where the fails happen16:56
finchclarkb: that definitely indicates that passenger is misbehaving16:56
jgriffithvishy: yeah, it's an lvm hang *again*16:56
*** dtroyer has joined #openstack-infra16:56
jgriffithvishy: I'm trying to see if there are many more of these around16:56
*** harlowja_away is now known as harlowja16:56
*** derekh has quit IRC16:56
vishywe should do a logstash search for errors like that in syslog16:56
jgriffithYeah... trying that now :)16:57
jgriffithbtw: https://bugs.launchpad.net/cinder/+bug/136612516:57
finchclarkb: fungi: could you pastebin the output of `passenger-memory-stats` ?16:57
uvirtbotLaunchpad bug 1366125 in cinder "LVM hang in volume creation" [Undecided,New]16:57
*** dannywilson has joined #openstack-infra16:59
funginibalizer: http://paste.openstack.org/show/115010/16:59
clarkbfinch: one sec16:59
jeblairgah! 122122!17:00
fungifinch: http://paste.openstack.org/show/115011/17:00
finchfungi: thanks17:00
clarkbfinch: http://paste.openstack.org/show/115012/17:00
clarkbfungi wins17:00
fungifinch: that command needs tty detection so it can turn off obnoxious ansi escapes automatically17:00
nibalizerfungi: yea its a giver :P17:01
nibalizerfungi: try that same curl but hit /production/status/no_key17:01
*** jp_at_hp has quit IRC17:01
jgriffithvishy: yeah, got a few more17:01
clarkbjeblair: 122122?17:01
clarkblooks like a tripleo change?17:01
jgriffithcrud17:02
*** dizquierdo has quit IRC17:02
jeblairclarkb: yeah, and it's a bad one17:02
clarkboh /me looks closer17:02
jeblairclarkb: see counterpart in 12250417:02
*** stevemar has joined #openstack-infra17:02
funginibalizer: Forbidden request: puppetmaster.openstack.org(127.0.0.1) access to /status/no_key [find] authenticated  at :11917:03
clarkboh fun17:03
jeblairjust proposed https://review.openstack.org/#/c/123788/17:03
nibalizerfungi: okay i give up :P17:03
*** rwsu has quit IRC17:03
sdaguesomething up with the log server?17:03
sdagueit's taking minutes to pull this syslog file17:03
*** _buzz has quit IRC17:03
sdaguewhich reports as 441k17:04
funginibalizer: i can compare the last started time on the puppet processes to when 3.7.0 was installed if you're concerned it's running something older than its reporting to puppetdb17:04
jgriffithvishy: well crap, where are all these rbd results coming from17:04
finchfungi: this is a bit of a shot in the dark but you might want to turn off http keepalives: https://docs.puppetlabs.com/references/latest/configuration.html#httpkeepalivetimeout17:04
*** dimtruck is now known as zz_dimtruck17:04
clarkbsdague: it doesn't look particularly unhappy. have a link to the file?17:04
sdagueclarkb: http://logs.openstack.org/22/119522/9/check/check-tempest-dsvm-postgres-full/46a5f8c/logs/syslog.txt.gz#_Sep_22_21_19_0117:04
funginibalizer: is there some reason not to trust the entry on the puppetboard page for it, which looks to me like it agrees with the package version?17:05
*** arborism has quit IRC17:05
sdaguejgriffith: http://logstash.openstack.org/#eyJzZWFyY2giOiJcImJsb2NrZWQgZm9yIG1vcmUgdGhhbiAxMjAgc2Vjb25kc1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDExNTc4Mjc1ODA3fQ==17:05
*** amcrn has joined #openstack-infra17:05
sdaguethat seems to be basically fails17:05
jgriffithsdague: nice!17:06
*** rwsu has joined #openstack-infra17:06
*** homeless has quit IRC17:06
jgriffithsdague: way better than my lame ass multipart querie17:06
*** _buzz has joined #openstack-infra17:06
sdagueheh17:06
jgriffithwe were close though in terms of results17:06
fungifinch: you think that a 4s keepalive is too long?17:06
sdagueyeh, I expect that 10% pass rate is fails during tempest test cleanups17:06
*** homeless has joined #openstack-infra17:06
clarkbit came right up for me. nothing for that file in the error log17:07
finchfungi: how many nodes do you have? and a colleague suggested it, but off the top of my head I have no idea17:07
clarkbhrm I do notice that when you paste a link here all the irssis and other irc clients seem to want to head the file17:07
finchthere's been a looot of changes between Puppet 2.7 and 3.717:07
clarkbfinch: 70-8017:07
fungifinch: i think roughly 80 but i'll confirm17:07
finchfungi: oh that's tiny. huh.17:07
clarkband we only hit the master 10 at a time17:07
clarkbsdague: I was getting terrible packet loss between home and rax last night but that seemed to be related to netflix hour and went away after 10pm17:08
clarkbsdague: so probably not related17:08
*** unicell has quit IRC17:08
*** mfer has quit IRC17:08
*** rushiagr is now known as rushiagr_away17:08
*** todd_dsm has joined #openstack-infra17:10
clarkbsdague: I do note (and this is based on my probably wrong infering) that you got fewer bytes than I did17:10
clarkbaccording to the apache log17:10
clarkbmaybe you asked for >DEBUG17:10
*** flaviof_zzz is now known as flaviof17:11
*** mfer has joined #openstack-infra17:11
finchclarkb: fungi: passenger_max_pool_size might be the trick: https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#PassengerMaxPoolSize17:11
*** dannywilson has quit IRC17:13
*** Ryan_Lane has joined #openstack-infra17:13
*** tty57 has quit IRC17:14
*** Sukhdev has quit IRC17:14
*** otherwiseguy has quit IRC17:14
*** dannywilson has joined #openstack-infra17:14
fungifinch: would that control the number of ruby processes we see in the system process table, or something more subtle?17:15
*** zz_dimtruck is now known as dimtruck17:16
clarkbjeblair: the commit message formatting on the revert is funky. otherwise +1 oh you just fixed it even better17:17
*** andreaf has quit IRC17:17
jeblairalso it had a conflict so i tried to remove just what was necessary17:18
*** dannywilson has quit IRC17:18
*** andreaf has joined #openstack-infra17:18
dhellmannjeblair: I'm not sure we care about python 3 for that lib. We want people to stop using it.17:18
dhellmannmordred: ^^17:18
clarkbjeblair: I take it we are ok merging 98426 as is to get it into the move then we can add the job to layout.yaml post move?17:19
finchfungi: that should cap the number of ruby processes running based on memory, so if one process is devouring memory then passenger won't launch more17:20
clarkbfinch: shouldn't it be killing these processes every 5 minutes or so though?17:20
*** shashankhegde has joined #openstack-infra17:20
jeblairclarkb: that was my thought17:20
*** reed has joined #openstack-infra17:22
finchclarkb: that's what I thought as well, hence me asking about the process lifetime. still looking17:22
fungiit seems we've lost openstackgerrit. retrieving17:23
*** armax has joined #openstack-infra17:24
*** armax has quit IRC17:24
*** armax has joined #openstack-infra17:24
jeblairAJaeger: should we proceed with 122620 now, or do you want to wait?17:24
*** openstackgerrit has joined #openstack-infra17:24
finchclarkb: another idea: passenger_pool_idle_time - set that to a low value so master processes are killed sooner17:25
*** dustins has joined #openstack-infra17:26
*** harlowja has quit IRC17:27
*** harlowja has joined #openstack-infra17:27
*** emagana has quit IRC17:28
DuncanTHi. I'm finding log urls in logstash to be 404ing... e.g. the one in http://logstash.openstack.org/#eyJzZWFyY2giOiJcImJsb2NrZWQgZm9yIG1vcmUgdGhhbiAxMjAgc2Vjb25kc1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDExNTc4Mjc1ODA3fQ==    - Any hints please?17:29
jeblairAJaeger: actually, i think we need a bit more discussion on that.  a quick graphite check suggests that will send 15-30 messages to the qa list every day.17:29
*** emagana has joined #openstack-infra17:29
EmilienMjeblair: I'm here17:31
*** masayukig has quit IRC17:31
EmilienMjeblair: just read your comment also...17:31
*** asettle has quit IRC17:32
*** lcheng has quit IRC17:32
*** emagana has quit IRC17:33
vishysdague, rbd: bug 137351317:34
uvirtbotLaunchpad bug 1373513 in cinder "Lvm hang during tempest tests" [High,Confirmed] https://launchpad.net/bugs/137351317:34
openstackgerritA change was merged to openstack-infra/config: Create repo for keystoneclient-federation  https://review.openstack.org/12067017:35
*** marcoemorais has joined #openstack-infra17:35
vishyah i see you found it already17:35
openstackgerritA change was merged to openstack-infra/config: Fix exit code in rally-cli job  https://review.openstack.org/12081517:35
mordreddhellmann: good by me17:36
openstackgerritA change was merged to openstack-infra/config: Simport - Simple python import library (not entry-point based)  https://review.openstack.org/12110817:36
openstackgerritA change was merged to openstack-infra/config: Remove unnecessary metadata check  https://review.openstack.org/12195817:36
*** masayukig has joined #openstack-infra17:36
jeblairmordred, dhellmann: can one or both of you please -1 120269 then? :)17:36
openstackgerritA change was merged to stackforge/python-jenkins: Support node startup with the JNLP Launcher  https://review.openstack.org/8960317:36
dhellmannjeblair: on it17:37
*** AJaeger_ has joined #openstack-infra17:37
openstackgerritSandy Walsh proposed a change to openstack-infra/config: notification_utils: utility library for OpenStack notifications.  https://review.openstack.org/12110917:37
dhellmannjeblair: to be clear, that lib is not an oslo thing and I don't actually have any permissions in that repo because I'm not on the maintenance team17:38
*** asselin has quit IRC17:38
jeblairdhellmann: yeah, i'm not really sure who is :)17:38
*** praneshp has joined #openstack-infra17:39
dhellmannjeblair: ditto17:39
*** asselin has joined #openstack-infra17:40
*** HeOS has quit IRC17:40
jeblairsandywalsh: because i know you're on a new-project spree, in case you missed it, see: http://lists.openstack.org/pipermail/openstack-dev/2014-September/046904.html17:41
*** dannywilson has joined #openstack-infra17:41
jogoclarkb: did something happen to ElasticSearch?17:41
sandywalshjeblair: eeek!17:41
jeblairsandywalsh: we're trying to get all the outstanding changes merged today so they don't need to be reproposed17:41
clarkbjogo: ?17:41
sandywalshjeblair: ok, will get my dependent branches fixed up shortly. Have a quick question about that17:42
sandywalshjeblair: I've been following https://wiki.openstack.org/wiki/Gerrit_Workflow#Add_dependency17:42
jeblairsandywalsh: so basically, we're happy to iterate quickly with you to get them in17:42
jogoclarkb: most recent timestamp I can find in ES is from Sept 23 20:3117:42
clarkbjogo: it is behind17:42
*** markmcclain has quit IRC17:42
jogoclarkb: way behind http://status.openstack.org/elastic-recheck/17:42
clarkbnothing "happened" to it as much as status quo17:42
*** ramashri has quit IRC17:42
*** newbie|3 has joined #openstack-infra17:42
jogoclarkb: any insight into how I can see if a log blew up recently?17:43
sandywalshjeblair: but I was wondering if it's possible to define the parent dependency at 'git review' time (vs. having to git review -r <change> beforehand?17:43
clarkbjogo: no not really17:43
fungisandywalsh: gerrit determines dependencies based on the history of the commit, so you generally end up needing to address merge conflicts anyway as that would imply some manner of rebase to order the commits as needed17:43
sandywalshjeblair: that way, I can cherry pick from my existing branches (ideally)17:44
jeblairsdague: can you review 123028 please?17:44
jeblairsandywalsh: if you already have the change locally, you can just add a new commit without using git review -d17:45
clarkbthings that do not mix. oil and water, gerrit and firefox17:45
EmilienMjeblair: about #120285 : I have no idea if that should be part of the program or not.17:45
sandywalshjeblair: hmm, can you elaborate?17:45
*** emagana has joined #openstack-infra17:45
jeblairsandywalsh: eg, if you are making a bunch of new projects, you can just create a series of commits and run 'git review' and it will push all of them up and they will have their dependencies set correctly in gerrit17:45
jogoclarkb: hmm I don't see any log with massive file sizes17:46
jogoat least not in tempest jobs17:46
*** e0ne has quit IRC17:46
*** gokrokve has quit IRC17:46
jeblairsandywalsh: and if you need to go back to edit something in the middle, you can use 'git rebase -i' to edit a particular commit, but keep the whole chain intact17:46
sandywalshjeblair: ah, so should I abandon the current branches and remove the change_id from the new branch?17:46
*** pballand has quit IRC17:47
jeblairsandywalsh: oh no, never do that -- keep the changeid in the commit message so that the existing gerrit review gets updated.  i may not fully understand what you're asking :)17:47
*** atiwari has joined #openstack-infra17:48
newbie|3Hi I have a question about a tempest test case that is testing a new feature that is in Juno, I defined an orchestration_feature_enabled section and set a flag there for this feature. This flag will set to true by default. If a user want to test Icehouse or older version will need to change this flag to false in the tempest.conf.17:49
sandywalshjeblair: ah, so if I keep the existing change-id's in each commit, it'll get resolved at review time?17:49
openstackgerritA change was merged to openstack-infra/config: Give oslo-release team permission to tag incubator  https://review.openstack.org/12245917:49
jeblairnewbie|3: you may want to ask that question in the #openstack-qa channel17:49
newbie|3thanks17:50
*** newbie|3 has left #openstack-infra17:50
jogoclarkb: any logs that show what files logstash is spending most of the time on or things like that?17:50
openstackgerritA change was merged to openstack-infra/config: Add functional test gate job for openstackclient  https://review.openstack.org/12259917:50
*** jcoufal has quit IRC17:51
jeblairsandywalsh: the change id is how gerrit determines that a given commit is related to a change in its database.  because each time a commit is changed (amended, rebased, etc), there's actually nothing in common between it and the old commit, so when you push it up for review, gerrit would have no way to know that it's a new version of a change it already knows about.17:51
*** openstackgerrit has quit IRC17:51
jeblairsandywalsh: so the commit-id header says "if you already have a change with this change id, consider this to be a new patchset for that change"17:51
jogoclarkb: having logstash so far behind makes it really hard to see what bugs we are hitting etc17:51
clarkbjogo: no not really. I don't have logstash feeding into itself for turtle reasons17:51
clarkbjogo: I understand...17:51
sandywalshjeblair: ok, I think I got it. Will try ... cross fingers :)17:51
clarkbjogo: so this is one of those things that isn't working and without better cloud probably isn't oing to get much better17:52
clarkbeither that or someone that knows how to elasticsearch will need to teach us17:52
jogoclarkb: hehe that would be funny logstash feeding itself17:52
*** openstackgerrit has joined #openstack-infra17:52
*** MaxV has joined #openstack-infra17:52
jogoclarkb: where is the main bottleneck?17:52
jeblairclarkb: oh, it's es that's the bottleneck?  i thought it was the log processors17:52
clarkbjeblair: no I think it is es17:52
jeblairclarkb: i thought crm114 was a bottleneck17:53
*** arnaud has quit IRC17:53
clarkbjeblair: I don't think so. When  Iturned off crm114 that one time it didn't increase throughput17:53
clarkbjeblair: because it is pipelined the crm114 slowness just gets "averaged" or whatever the term is17:54
openstackgerritBarnaby Court proposed a change to openstack-infra/jenkins-job-builder: Adds credentials-binding plugin support to wrappers  https://review.openstack.org/12018417:54
clarkbjeblair: the crm114 subprocesses was leaking file descriptors which lead to bitbucketing of data which masked the issue for a while17:54
clarkbbecause we were indexing much less data17:55
jeblairclarkb: we have 65 workers online17:56
clarkbits also possible that upgraded ES would help. I should look through their release notes17:56
clarkbjeblair: huh should only be 60. I wonder if that is a gear bug17:56
jeblairclarkb: don't we have more than that?17:56
clarkbjeblair: no its 20 nodes X 3 workers per node17:57
jeblairoh we dropped from 4 to 317:57
jogowow that is a beefy system17:57
clarkbjeblair: ya 4 led to OOMing17:57
jeblairclarkb: i think some of them are broken17:57
jeblairall the workers on 01 have 0 byte logfiles17:57
openstackgerritA change was merged to openstack-infra/config: For requirements, run job to verify that requirements overlap  https://review.openstack.org/9842617:58
jogojeblair: that may explain the backlog that I am seeing17:58
openstackgerritA change was merged to openstack-infra/config: Enable functional testing for openstackclient  https://review.openstack.org/12276517:58
*** emagana has quit IRC17:58
openstackgerritA change was merged to openstack-infra/config: Add integration tests to the Horizon gate  https://review.openstack.org/11500417:58
openstackgerritA change was merged to openstack-infra/config: Add keystone-formula project  https://review.openstack.org/12032417:58
openstackgerritA change was merged to openstack-infra/config: Add glance-formula project  https://review.openstack.org/12204917:58
*** daya_k has quit IRC17:58
*** emagana has joined #openstack-infra17:58
*** mfer has quit IRC17:59
clarkbjeblair: gaierror: [Errno -3] Temporary failure in name resolution17:59
clarkbjeblair: so this could be the leaky fd thing again17:59
sdaguejeblair: +1 on https://review.openstack.org/#/c/123028/17:59
jeblairsdague: thx; aprvd18:00
openstackgerritBob Callaway proposed a change to openstack-infra/config: Add puppet-manila to Stackforge  https://review.openstack.org/12380218:00
*** todd_dsm has quit IRC18:01
*** tonytan4ever has joined #openstack-infra18:01
clarkbare we ready for 122437?18:01
clarkbis that something we might want to announce?18:01
fungijeblair: regarding your comment on 122956, that's another one of those troubling stackforge/openstack co-mingling quandaries18:01
*** mfer has joined #openstack-infra18:02
*** james_li has quit IRC18:02
*** emagana has quit IRC18:03
jeblairclarkb: i think it should be announced.  lets see if sdague is ready to announce it before approving18:04
arosenclarkb:  remember when i was asking about that grub error i was encountering?18:05
*** marun has joined #openstack-infra18:05
arosenI think we tracked down why it is occurring in our cloud thanks to node pool :)18:05
arosenwhat ends up occurring is after the image is snapshotted nodepool deploys a bunch of nodes from that image right after.18:06
*** radez_g0n3 is now known as radez18:06
arosenWe use shared stored and gluster in our cloud.18:06
*** wenlock has joined #openstack-infra18:06
jeblairfungi: if the change is correct, then, agreed.  however, that doesn't make any sense to me, perhaps it's an error.18:06
*** johnthetubaguy is now known as zz_johnthetubagu18:06
arosenOur version of guster is 3.2 and in 3.3 they fixed a locking issue that if multiple writings are writing to the place on a mount point that data consistency will be ensured.18:07
fungijeblair: perhaps. i was assuming they avoided adding then to the gate pipeline so they wouldn't co-gate with official projects, but that's also something they could solve with a unique branch designator suffix on their jobs18:08
openstackgerritA change was merged to openstack-infra/config: Remove Nova v3 API tests  https://review.openstack.org/12302818:08
jeblairianw, yolanda: are we installing dib from master or from pypi releases?  if pypi, 123343 depends on 118246 being in a release; is it?18:08
*** gokrokve has joined #openstack-infra18:08
jeblairfungi: yeah, i'm leaning toward error because what would they gain running the neutronclient unit tests?18:08
*** james_li has joined #openstack-infra18:09
clarkbjeblair: jogo all 3 workers on 01 are zero length log files and one of three on 08 have a zero length log file18:09
clarkbin theory that means we have 56/60 workers doing stuff18:09
clarkba quick glance at cacti seems to confirm18:09
clarkbarosen: neat18:10
EmilienMjeblair: about #120285 : I have no idea if that should be part of the program or not.18:10
fungijeblair: ahh, right. i was talking about the ones in check but not gate for the first project (where they're running neutron dsvm jobs). but also now that i think about it, that won't actually be testing the gbp extension anyway18:12
jeblairEmilienM: i think it's up to lifeless to decide if it's in the tripleo program; if you or he don't want it to be, then you can just change the groups to be specific to the new project and not reference diskimage-builder-core18:12
*** mrmartin has quit IRC18:13
*** andreykurilin_ has joined #openstack-infra18:13
EmilienMjeblair: I don't have the pretension to be part of core team of this new project. I just want to be sure that people from TripleO can review this work.18:14
clarkbfungi: I wrote 122578 so you should probably review that one18:14
fungiclarkb: lgtm. honestly i thought we had already done that, but maybe not18:15
jeblairEmilienM: that sounds like a problem if you are proposing a project that only the tripleo core team can review, but you aren't in it.18:15
*** emagana has joined #openstack-infra18:15
jogoclarkb: so we have most of our workers and things are still behind, hmmmm18:15
AJaeger_clarkb: regarding 122437: Should those braces get removed or is that really a nit?18:16
jeblairEmilienM: if the tripleo program wants this project, then i would expect them to adopt it into their program.  if they don't want it, then i would expect another team to be reviewing it (which may have some of the same members)18:16
*** vhoward has left #openstack-infra18:16
*** jistr has quit IRC18:16
fungiAJaeger_: it's not as clean as it could be, but i believe it will work as written18:17
AJaeger_fungi: yes, it will work as written.18:17
EmilienMjeblair: we had a discussion in the ML and TripleO folks agreed18:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Sandbox - DevStack-like dev environment for StackTach.v3  https://review.openstack.org/12084418:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: StackTach.v3 Quincy REST API  https://review.openstack.org/12058218:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Winchester - A notification event processing library based on persistent streams.  https://review.openstack.org/12118218:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: StackDistiller - A notification extraction and transformation library.  https://review.openstack.org/12085818:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Timex - A time expressions library  https://review.openstack.org/12085218:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Klugman - cmdline tool and client library for StackTach.v3  https://review.openstack.org/12083818:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Quince - Quincy-compliant API implementation for StackTach.v3. Supports Winchester.  https://review.openstack.org/12083918:17
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Notigen - OpenStack-like notification generator/simulator.  https://review.openstack.org/12084918:17
EmilienMjeblair: http://lists.openstack.org/pipermail/openstack-dev/2014-September/045395.html18:18
fungipuh-puh-puh-pow18:18
AJaeger_sandywalsh: so you mastered the dependencies? Great! Let me review...18:18
sandywalshAJaeger_: thanks ... jeblair gave me the foo18:19
*** krtaylor has quit IRC18:20
jeblairEmilienM: i think you were asking the right question in the first message (about openstack/stackforge), but i don't think the thread cleared it up.  i think we need lifeless to indicate whether he thinks it should be in the tripleo program or not.18:20
*** arnaud has joined #openstack-infra18:20
HenryGIn openstack-infra/config there is slave_db.pp to install mysql and pg backends with 'openstack_citest' user. I would like this to be done for the machines running the neutron-dsvm-functional job. I would appreciate any hints on how to achieve this.18:21
openstackgerritClark Boylan proposed a change to openstack-infra/config: Add new experimental jobs to test dib based nodes  https://review.openstack.org/12324518:21
clarkbfungi: jeblair ^ that hit a merge conflict18:21
*** arnaud__ has joined #openstack-infra18:21
sandywalshjeblair: AJaeger_ I may have to rebase https://review.openstack.org/#/c/121109/ after the big dependency chain goes through, so please leave that one till last18:21
EmilienMjeblair: ok. So I'll get the insights from him18:22
openstackgerritBob Callaway proposed a change to openstack-infra/config: Add puppet-manila to Stackforge  https://review.openstack.org/12380218:22
finchclarkb: my life is going to be insane for the next 20 hours so if you want to poke more I can try to respond when I have internets, but I can definitely help more later. In any case, passenger should be restarting the ruby processes on a regular basis, and if it's not then that is probably an issue with the passenger config18:24
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: [WIP] Manage autoincrement in migration  https://review.openstack.org/12380918:25
*** krtaylor has joined #openstack-infra18:25
zaroclarkb: this doesn't fail on my local env.  what bug number should i use to reverify?  https://review.openstack.org/#/c/8960418:26
clarkbfungi: ok18:26
clarkbzaro: did you rebase it with master when you tried locally?18:26
*** zigo has quit IRC18:26
*** cnesa15 has joined #openstack-infra18:26
clarkbzaro: I do not think that is a bug and rechecking won't fix it18:26
zaronope18:27
sandywalshAJaeger_: can I do a patch up branch at the end to reorder into alphabetical? Will only take a second once we have all the branches merged.18:27
sandywalshAJaeger_: otherwise I'm in rebase hell18:27
*** bdpayne has joined #openstack-infra18:27
zaroclarkb: i'll try recheck to see if it happens again.18:28
clarkbzaro: it will happen again...18:28
zaroclarkb: can you give it a try?18:28
clarkbzaro: zuul always tests your changes against the target branch. I think that merging is causing the problem18:28
*** zigo has joined #openstack-infra18:28
*** rmcall has joined #openstack-infra18:28
*** SumitNaiksatam has joined #openstack-infra18:29
zaroclarkb: ok. i'll try rebase with master this time.18:29
*** sdake_ has joined #openstack-infra18:29
*** bswartz has quit IRC18:30
zaroclarkb: ok. failed after rebasing.  thanks.18:30
*** otherwiseguy has joined #openstack-infra18:31
clarkber finch ok18:31
clarkbsilly tab complete18:31
*** mriedem has quit IRC18:32
*** vhoward has joined #openstack-infra18:33
*** bswartz has joined #openstack-infra18:33
*** praneshp has quit IRC18:33
*** marcoemorais has quit IRC18:34
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add os-client-config to stackforge  https://review.openstack.org/12316718:35
*** marcoemorais has joined #openstack-infra18:35
clarkbjeblair: fungi 118944 seems like a rather large chagne to just push through18:35
clarkbwhat do you guys think of deferring that one?18:35
clarkbjesusaurus: ^18:35
*** marcoemorais has quit IRC18:36
*** praneshp has joined #openstack-infra18:36
*** Sukhdev has joined #openstack-infra18:36
AJaeger_sandywalsh: Let's ask jeblair, clarkb or fungi  what they think - https://review.openstack.org/#/q/owner:sandy.walsh%2540rackspace.com+status:open,n,z is the full patch set, I marked some as -1 because of ordering.18:36
*** marcoemorais has joined #openstack-infra18:37
sandywalshAJaeger_: thanks ... I'm working on the ordering rebase now, just in case :)18:37
*** marcoemorais has quit IRC18:37
fungijeblair: jesusaurus: clarkb: it looks like it would be a relatively easy one to repropose to the new project since the files it's touching don't see a lot of churn?18:37
clarkbAJaeger_: sandywalsh do we need alphabeitical to pass tests?18:37
krotscheckmordred, lifeless: So regarding managing the LP id’s in storyboard - a bit of trial-and-error shows me that you cannot change the autoincrement value to a number <= than the highest id that’s already in use. If I have a record ID of 10, and I try to change the auto_increment to 5, it ends up 11.18:37
clarkbif not it should be fine as is18:37
*** marcoemorais has joined #openstack-infra18:37
clarkbfungi: that is whawt I was thinking too18:37
AJaeger_clarkb: no, we don't need them to pass the testsuite18:38
*** marcoemorais has quit IRC18:38
*** marcoemorais has joined #openstack-infra18:38
*** andreaf has quit IRC18:38
AJaeger_clarkb: Ok, I'll remove my -118:38
krotscheckThe workaround I thought of earlier, which was to first create the record and then modify the id in an update, also doesn’t work because with the next insert the autoincrement resets itself.18:39
*** andreaf has joined #openstack-infra18:39
krotscheckSo I feel like the only real option is to set the autoincrement ID in storyboard to Large(tm)18:39
krotscheckAnd then insert the LP records underneath that.18:39
* anteaya is back from the dentist18:39
anteayait went better this time18:39
*** juzuluag has quit IRC18:39
fungikrotscheck: as opposed to keeping a mapping table and providing a convenience lookup?18:39
*** e0ne has joined #openstack-infra18:40
*** otherwiseguy has quit IRC18:40
AJaeger_clarkb, fungi let's merge 118944 in soonish - even if you defer. Would be nice to have bashate tests not failing the whole time, this is confusing...18:40
*** denis_makogon has joined #openstack-infra18:40
notmynamewho is tracy jones and ryan? and is gerrit really going to have checks for certain tags in the commit message? http://lists.openstack.org/pipermail/openstack-dev/2014-September/047028.html18:40
clarkbnotmyname: no to the second bit at least not globally. Individual projects can hack that into their tests if they want it18:41
clarkbnotmyname: infra has no plans to implement that18:41
notmynameClabbe: ok, thanks18:41
notmynameclarkb:18:41
morganfainbergclarkb, thanks for clarification.18:42
krotscheckfungi: maaaaybe. Storing the origin of an imported story makes some sense, I just don’t think I want a unique table called ‘launchpad-to-storyboard-mapping'18:42
notmynameclarkb: I saw no context for the message (and didn't recognize names), so I wasn't sure if this was some new thing imposed here at the end of the juno cycle18:42
*** pcrews has quit IRC18:42
funginotmyname: yeah, looks like maybe it's only sahara judging from the subject line tag?18:42
clarkbnotmyname: ya it is severly lacking in context not sure if that was maybe for an internal gerrit and was mis posted?18:42
fungiSergeyLukjanov: ^ ideas?18:43
morganfainbergnotmyname, my guess is it might be missing [sahara] or some other tag18:43
fungioh nevermind--i was looking at the previous message url18:43
AJaeger_Or [oslo.vmware] ?18:43
fungiyeah no clue18:43
* SergeyLukjanov reading18:43
fungienocontext18:43
fungiSergeyLukjanov: i was mistaken18:43
clarkbwow that merge check queue has been pretty impressive18:43
morganfainbergAJaeger_, yep18:44
*** hashar has joined #openstack-infra18:44
clarkbjeblair: ^ we rprobably should make the jobs in that pipeline a lower priority18:44
mordredkrotscheck: yes, that's right. we're going to need to adjust up18:44
AJaeger_morganfainberg: tracy jones works on nova only, just checked stackalytics18:44
mordredrather than low18:44
morganfainbergah18:44
anteayanotmyname: tjones is the nova bug czar18:44
anteayaso this might be some quality check coming from the sdague recon work on nova bug tracker18:45
krotscheckmordred: Any thoughts on storing the origin id instead of even caring about the id?18:45
anteayathat is my guess18:45
notmynameanteaya: thanks18:45
anteayanp18:45
morganfainberganteaya, names are start ringing more bells then18:45
anteayanotmyname: hey has that mentee been showing up in swift at all?18:45
anteayamorganfainberg: cool18:45
krotscheckmordred: It’s a thing that fungi suggested18:46
mordredkrotscheck: if we do, we may want a lookup by old id mechanism18:46
sdagueI wonder if tjones sent to the wrong list?18:46
notmynameanteaya: yes. she's been arounda while, and we chatted (briefly) this morning. we've also got an email thread going18:46
anteayawell I applaud any email that says "don't +2 your own patch"18:46
anteayaregardless of its target18:46
fungikrotscheck: oh, do we have extensible tagging? could we have an arbitrary bit of metadata which isn't baked into the db schema as an lp mapping specific column?18:46
sdagueanteaya: sure, though I haven't seen that be an issue18:46
anteayanotmyname: yay, so happy to hear she is showing up18:47
anteayasdague: glad to hear that too then18:47
krotscheckfungi: not yet18:47
anteayasdague: I haven't seen it myself but I don't get out much18:47
morganfainbergsdague, likely if somone is +2ing their own patch... they wont have access to do it for too long, just a hunch18:47
anteayanotmyname: let me know if I can ever help at all18:47
notmynameanteaya: ok, thanks. I'm pretty focused on getting stuff for juno wrapped up. but I'm sure I'll ask questions later :-)18:47
krotscheckfungi: However the concept of a story’s federated origin is one we’ve been discussing, this could just be the first step.18:48
anteayanotmyname: I'll do my best when you have time18:48
anteayanotmyname: go you \o/18:48
*** dannywilson has quit IRC18:48
*** e0ne is now known as e0ne_18:49
*** dannywilson has joined #openstack-infra18:49
*** flaviof is now known as flaviof_zzz18:49
sandywalshAJaeger_: if you run into problems, I have the rebase done as a precaution18:50
*** andreaf has quit IRC18:50
*** andreaf has joined #openstack-infra18:51
clarkbsolum devs18:51
clarkbyou are switching your job from f20 to trusty. in the past we switched from trusty to f20 because trusty rabbit did not work18:52
clarkbwell I guess it isn't my job to debug that for them /me approves18:52
clarkboh they resolved the trusty issues... why didn't they move to trusty then18:52
*** dannywilson has quit IRC18:53
morganfainbergnotmyname, and there is your answer18:54
AJaeger_sandywalsh: The cores have to give their +2 on these...18:54
notmynamemorganfainberg: heh :-)18:54
sandywalshAJaeger_: gotcha ... thanks again! (and thanks jeblair for the heads up on the deadline ... that could have been bad)18:55
openstackgerritBarnaby Court proposed a change to openstack-infra/jenkins-job-builder: Adds credentials-binding plugin support to wrappers  https://review.openstack.org/12018418:55
*** e0ne_ has quit IRC18:56
*** MaxV has quit IRC18:57
*** esker has joined #openstack-infra18:57
clarkbthe kolla change needs a rebase https://review.openstack.org/#/c/12297218:58
clarkbsdake: ^18:58
*** emagana has quit IRC18:58
*** emagana has joined #openstack-infra18:58
*** MaxV has joined #openstack-infra18:59
*** lcheng has joined #openstack-infra18:59
*** HeOS has joined #openstack-infra19:00
AJaeger_wow, you should more often split up repos - all my patches have been approved ;) Thanks, infra team!19:00
*** marcoemorais has quit IRC19:00
*** wenlock has quit IRC19:00
*** e0ne has joined #openstack-infra19:00
*** marcoemorais has joined #openstack-infra19:01
openstackgerritSumit Naiksatam proposed a change to openstack-infra/config: Enabling jobs for GBP project  https://review.openstack.org/12295619:02
*** morganfainberg is now known as morgan19:02
openstackgerritA change was merged to openstack-infra/config: Don't let a reqs sync fail kill the sync job  https://review.openstack.org/12257819:02
*** denis_makogon has quit IRC19:02
*** esker has quit IRC19:03
*** emagana has quit IRC19:03
*** morgan is now known as morganfainberg19:03
jeblairAJaeger_: we plan to make it a regular occurence!  just not for these same files.  ;)19:04
clarkbBobBall: jeblair I have read https://review.openstack.org/#/c/123785/1 and the bug that it is supposed to fix. I am not convinced we are fixing the bug though since we continue to use /tmp. You are relying on puppet running long enough to avoid the race which seems problematic for say if we get rid of puppet there19:04
clarkbBobBall: jeblair: we should be explicit about fixing the bug and check that /tmp is moutned properly before usint it19:05
sdakecalrkb incoming19:05
clarkbor something like that. otherwise we are bound to hit this sort of bug again19:05
sdakeclarkb incoming that is19:05
jesusaurusclarkb: fungi: yeah 118944 would be easy enough to repropose19:05
*** baoli has quit IRC19:05
AJaeger_sandywalsh: http://logs.openstack.org/09/121109/5/check/gate-config-projects-yaml/390f36a/console.html - alphabetication test failure19:06
sandywalshAJaeger_: k, on it ... thanks19:06
*** cnesa15 has quit IRC19:06
*** AJaeger_ has quit IRC19:07
openstackgerritSteven Dake proposed a change to openstack-infra/config: Add Kolla project - Dockerizing OpenStack  https://review.openstack.org/12297219:07
sdakeclarkb rebased19:07
jeblairclarkb: i agree the bug/problem could be better understood and likely a better fix devised.  though i don't think the existing patch is a problem.  (even if it weren't fixing a bug i'd probably accept it)19:07
clarkbjeblair: ok19:07
jeblairclarkb: that's not a strong opinion19:08
*** cnesa13 has joined #openstack-infra19:08
anteayaI don't get to vote in release management elections (has there been one?) or tripleo elections but I do find ttx's and SpamapS' emails always informative and entertaining19:08
* clarkb wonders if fungi has an opinion19:08
clarkbfungi: see 123785 if/when you have a chance19:08
*** hashar has quit IRC19:09
*** rmcall has quit IRC19:10
*** asettle has joined #openstack-infra19:10
*** asettle has quit IRC19:11
fungiclarkb: jeblair: yeah, i'm fine with the change as is. i agree it's not necessarily fixing anything specific, but we should try to closely couple creation and abandonment of temporarily files anywhere we can19:11
*** rmcall has joined #openstack-infra19:12
*** rmcall has quit IRC19:12
fungithe referenced bug needs more research before anything could be considered to actually fix it19:12
openstackgerritA change was merged to openstack-infra/config: Add direct-release for training-guides and ha-guide  https://review.openstack.org/12355819:12
openstackgerritSumit Naiksatam proposed a change to openstack-infra/config: UI, automation repos for group based policy project  https://review.openstack.org/12292219:12
openstackgerritA change was merged to openstack-infra/config: Remove Jenkins jobs of tricircle project  https://review.openstack.org/12362219:12
clarkbfungi: jeblair would it be appropriate to update the commit message to say related-bug instead of fixes-bug?19:12
clarkb(I don't want to cause unnecessary churn)19:12
fungiclarkb: sure, if you want to edit the commit message and approve it, go for it19:13
jeblairclarkb: wfm19:13
clarkbok will do19:13
*** coolsvap is now known as coolsvap|gone19:13
*** marcoemorais1 has joined #openstack-infra19:14
openstackgerritClark Boylan proposed a change to openstack-infra/config: Move static definition of forwarding.conf later in script  https://review.openstack.org/12378519:14
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Sandbox - DevStack-like dev environment for StackTach.v3  https://review.openstack.org/12084419:14
openstackgerritSandy Walsh proposed a change to openstack-infra/config: StackTach.v3 Quincy REST API  https://review.openstack.org/12058219:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Winchester - A notification event processing library based on persistent streams.  https://review.openstack.org/12118219:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: StackDistiller - A notification extraction and transformation library.  https://review.openstack.org/12085819:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Timex - A time expressions library  https://review.openstack.org/12085219:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: notification_utils: utility library for OpenStack notifications.  https://review.openstack.org/12110919:15
*** marcoemorais has quit IRC19:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Klugman - cmdline tool and client library for StackTach.v3  https://review.openstack.org/12083819:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Quince - Quincy-compliant API implementation for StackTach.v3. Supports Winchester.  https://review.openstack.org/12083919:15
openstackgerritSandy Walsh proposed a change to openstack-infra/config: Notigen - OpenStack-like notification generator/simulator.  https://review.openstack.org/12084919:15
sandywalshAJaeger: that should fix up all the ordering too19:15
lifelessjeblair: I'm fine with tripleo-puppet-elements being created; I thought it had already19:15
*** radez is now known as radez_g0n319:15
jeblairlifeless: the question is whether it should be in the tripleo program19:17
jeblairlifeless: the proposed creation patch has it in stackforge (not in tripleo) with only dib-core able to review (defacto in tripleo)19:17
openstackgerritA change was merged to openstack-infra/config: solum f20 gate is failing due to upstream dep.  https://review.openstack.org/12278219:17
*** ChuckC has quit IRC19:18
jeblairlifeless: those seem contradictory, and it's not clear to me which it should be19:18
*** Longgeek has quit IRC19:18
openstackgerritA change was merged to openstack-infra/config: Use check-requirements templates in layout.yaml  https://review.openstack.org/12294719:18
*** asettle has joined #openstack-infra19:19
lifelessjeblair: whats the contradiction?19:19
*** e0ne is now known as e0ne_19:20
*** dimtruck is now known as zz_dimtruck19:21
*** melwitt has joined #openstack-infra19:21
*** loki184 has quit IRC19:21
openstackgerritA change was merged to openstack-infra/config: swiftonfile: Add py26  https://review.openstack.org/12337819:22
*** shashankhegde has quit IRC19:22
jeblairlifeless: is it, or is it not, intended to be part of the tripleo program?19:22
fungiBobBall: i have a theory on your bug 1373505 there... check your fstab and the output of initctl show-config mounted-tmp19:23
uvirtbotLaunchpad bug 1373505 in openstack-ci "Ubuntu Trusty race: forwarding.conf not found" [Undecided,In progress] https://launchpad.net/bugs/137350519:23
lifelessjeblair: it is19:23
*** devkulkarni has joined #openstack-infra19:23
jeblairlifeless: okay, so it should go into the openstack/ namespace and there should be an accompanying patch to openstack/governance to add it to the tripleo program19:24
jeblairEmilienM: ^19:24
lifelessah19:24
lifelessso you'll want that done for tripleo-ansible too19:24
lifelessand for kolla19:24
arosenanteaya:  ping19:24
devkulkarnithanks jeblair clarkb for voting: https://review.openstack.org/#/c/122782/19:24
fungii think tripleo-ansible was already proposed to openstack rather than stackforge19:24
arosencan you update a public ssh key for me for our ci?19:24
jeblairlifeless: ya19:24
lifelesssdake: ^19:25
lifelessSpamapS: ^19:25
clarkbjeblair: lifeless: I think we care less about it being in stackforge initially if the dev groups are at least distinct in gerrit?19:25
clarkbthe having tripleo core be core on stackforge project that wants to be in openstack is what initial prompted the questions. Seems like it should just be in tripleo if that is the case19:25
jeblairclarkb: i'm pretty sure lifeless just said he wants it in tripleo19:26
openstackgerritBob Callaway proposed a change to openstack-infra/config: Add puppet-manila to Stackforge  https://review.openstack.org/12380219:26
clarkbjeblair: ya and thats fine too. Just wanted to clarify the general case situation19:26
*** mriedem has joined #openstack-infra19:27
*** DuncanT has quit IRC19:27
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Manage autoincrement in migration  https://review.openstack.org/12382119:28
*** DuncanT- has joined #openstack-infra19:28
openstackgerritA change was merged to openstack-infra/config: Add new experimental jobs to test dib based nodes  https://review.openstack.org/12324519:29
*** bswartz has quit IRC19:29
*** markmcclain has joined #openstack-infra19:29
krotscheck^ That’s the last of 4 patches that should address the infra migration issues raised in the meeting yesterday19:29
krotscheckThe script that should be executed on the storyboard server is here -> https://etherpad.openstack.org/p/infra-storyboard-migration19:30
krotscheck(Once those land)19:30
anteayaarosen: I have no permissions to update a public key19:31
*** andreaf has quit IRC19:31
anteayaarosen: and core is having a push to get as many config patches into config before the freeze in about 4.5 hours19:31
EmilienMjeblair: just read19:32
anteayaarosen: might I suggest you ping a core on friday and between you and I we can find a willing soul to do that update19:32
openstackgerritAdam Gandelman proposed a change to openstack-infra/config: Disable ironic job for stable/icehouse  https://review.openstack.org/12382219:32
*** andreaf has joined #openstack-infra19:32
openstackgerritA change was merged to openstack-infra/config: Add os-client-config to stackforge  https://review.openstack.org/12316719:32
anteayaarosen: is that acceptable?19:32
*** gyee has joined #openstack-infra19:34
*** sdake_ has quit IRC19:36
*** hashar has joined #openstack-infra19:38
openstackgerritEmilien Macchi proposed a change to openstack-infra/config: Introduce tripleo-puppet-elements into Stackforge  https://review.openstack.org/12028519:38
*** rmcall has joined #openstack-infra19:38
EmilienMjeblair: lifeless: updated ^^^^ - I'm looking at openstack/governance now.19:38
koolhead17Alex_Gaynor: salute.19:38
EmilienMjeblair: lifeless: and the patch in governance: https://review.openstack.org/12382619:41
*** radez_g0n3 is now known as radez19:41
jeblairEmilienM: thanks.  we might not get to merging the patch to create it today while we wait to make sure the tc doesn't object to it19:42
*** shashankhegde has joined #openstack-infra19:42
EmilienMjeblair: today was not in my mind, but I don't want to forget this one. It has some time already...19:42
*** emagana has joined #openstack-infra19:45
*** atiwari has quit IRC19:45
*** e0ne_ has quit IRC19:46
*** openstackgerrit has quit IRC19:46
*** openstackgerrit has joined #openstack-infra19:47
openstackgerritA change was merged to openstack-infra/config: Move static definition of forwarding.conf later in script  https://review.openstack.org/12378519:48
*** pcrews has joined #openstack-infra19:50
*** coolsvap|gone has quit IRC19:51
*** wenlock has joined #openstack-infra19:52
arosenanteaya: mind pointing me where the change needs to be made?19:53
arosenis this in the config repo?19:53
*** ociuhandu has joined #openstack-infra19:54
anteayaarosen: no this isn't a config repo change19:54
anteayait can only be done via ssh or the gui with someone with gerrit permissions19:54
*** lcheng has quit IRC19:54
anteayawhich I don't have19:54
*** ChuckC has joined #openstack-infra19:54
clarkbsdake: jeblair see comments on https://review.openstack.org/#/c/122972/19:54
clarkblifeless: ^19:54
fungiarosen: i saw your post to the ml. i'll take care of it real quick19:54
anteayafungi: thanks19:55
arosenfungi:  awesome thanks :)19:55
sdagueclarkb / fungi : in 9 minutes, when the top of the patches make it through, it's probably worth promoting - https://review.openstack.org/#/c/123774/19:55
sdaguethat's a presumed fix for the #1 gate affecting bug19:55
jeblairclarkb: thx19:55
*** _nadya_ has joined #openstack-infra19:55
clarkbsdague: noted19:56
*** mfer has quit IRC19:56
*** atiwari has joined #openstack-infra19:56
clarkbhttps://jenkins04.openstack.org/job/experimental-tempest-dsvm-full-dibtest/1/console is running \o/19:58
clarkblet us see how it does19:58
*** david-ly_ is now known as david-lyle19:58
*** sballe__ has joined #openstack-infra20:00
*** _nadya_ has quit IRC20:00
dkranzIs there a simple explanation why in https://review.openstack.org/#/c/123252/, jenkins says +1 but there are jobs that failed and are not marked as non-voting?20:00
notmynameseems to me that the wiki is logging you out very quickly these days (like maybe after an hour)?20:00
fungiarosen: i don't have a congress-ci account, just a vmware-congress-ci account. same one?20:00
jeblairdkranz: those jobs came from the experimental queue20:01
Alex_Gaynorkoolhead17: ?20:01
sdaguenotmyname: yeh, I was noticing that as well today. I wonder if it's the wiki or the openid provider20:01
fungiarosen: account_id=13050 full_name="VMWare Congress CI"?20:01
jeblairdkranz: why the experimental queue has both voting and nonvoting jobs is curious.  :)20:01
clarkbdkranz: looks like those jobs run in the exerimental queues. so they don't vote20:01
sdagueclarkb: possition 2 change is about to fail20:01
*** lcheng has joined #openstack-infra20:01
notmynamesdague: I made edits before the swift team meeting. tried to edit after the meeting was over and had to log in again20:02
*** bswartz has joined #openstack-infra20:02
clarkbsdague: I am ready when it processes the head20:02
dkranzclarkb, jeblair : ok, I had just been used to being suspicious of failing jobs that were not marked as non-voting20:02
sdagueman, it's taking a long time to process the head20:02
clarkbsdague: it has a bunch of results events it is chewing through20:03
sdaguegotcha20:03
sdagueok, the head merged20:03
clarkbthere it went. promoting now20:03
*** marcoemorais1 has quit IRC20:04
sdaguegreat, thanks20:04
clarkband done20:04
arosenfungi:  sorry yes that's it.20:04
*** marcoemorais has joined #openstack-infra20:04
*** atiwari has quit IRC20:05
*** marcoemorais has quit IRC20:05
jaypipesjeblair: really diggin gertty :)20:05
*** marcoemorais has joined #openstack-infra20:05
*** hashar has quit IRC20:05
fungiarosen: it gives me "fatal: Identity in use by another account" so i think that e-mail address must already be associated with another existing gerrit account?20:05
stevemarwhat's it mean if a job is appearing as `NOT_REGISTERED`? - looking at 122605 on http://status.openstack.org/zuul/20:06
arosenfungi:  i guess we can keep the same email address that's already on there then.20:06
*** zz_dimtruck is now known as dimtruck20:06
fungiarosen: okay, added the requested key20:07
sdagueso the zuul waiting job queue hasn't got to 0 in a long time. is that new normal with the merge checks?20:07
*** packet has joined #openstack-infra20:07
anteayastevemar: it means that zuul/jenkins couldn't find it20:07
arosenfungi:  thanks again20:08
*** devkulkarni has quit IRC20:09
fungistevemar: yeah, either zuul's layout updated before the jjb created those jobs, or they disagree on what node type to use20:09
*** boden has quit IRC20:09
*** boden has joined #openstack-infra20:10
fungistevemar: looks like it's registering for both devstack-precise and devstack-trusty node types now, so probably just a race between zuul and jjb20:10
jeblairjaypipes: woohoo!20:11
fungistevemar: yeah... 2014-09-24 19:22:55,923 ERROR zuul.Gearman: Job <gear.Job 0x7f556975da10 handle: None name: build:check-osc-functional:bare-trusty unique: 31d513475ae84228b7216af001d6c5a2> is not registered with Gearman20:11
fungistevemar: if you recheck once that change reports, it should be fine20:11
jeblairsdague: yeah, merge checks are counted as jobs in that graph20:12
sdagueright, so the merge checks on nova seem to be taking up to 45 minutes20:12
sdagueat least I saw that this morning20:12
*** mfer has joined #openstack-infra20:12
stevemarfungi, alrighty, i'll try another recheck when this one finishes up - hope that's the case20:12
sdagueso... maybe it's better to just stop doing those20:13
*** baoli has joined #openstack-infra20:13
sdagueas if that's going to add big chunks of time to restarting gate queue after a merge, that seems like it's not worth the early info20:13
*** kashyap has quit IRC20:13
stevemarmerge check is all nova, thats impressive20:14
sdaguestevemar: well... we merged a nova change :)20:14
sdagueso that's what you'd expect20:14
jeblairsdague: i don't think they take that long20:14
sdaguejeblair: well they've been queued 10 minutes at this point20:14
jeblairsdague: and the better solution is to have the merge check job take on the gear precedence of the pipeline they are in.  then the impact should be less.20:14
jeblairsdague: also, we can throw more cloud at it.20:15
sdaguejeblair: more cloud would be great20:15
*** boden has quit IRC20:15
stevemarwouldn't we have to take away cloud from somewhere to add more cloud at merge-check?20:15
jeblairsdague: time in queue is also not the same as the time it takes to process those changes since (due to lack of precedence pass-through) they are sharing the queue with all other merge work20:15
jeblairsdague: (it could be 5 mins before zuul gets around to starting the first merge-check, but that still counts as time in queue)20:16
sdaguejeblair: sure, it looks like it's chewing through 10 - 15 a minute20:16
sdaguebut.... a nova merge creates ~300 of them20:17
*** marcoemorais has quit IRC20:18
*** mwagner_lap has quit IRC20:18
*** marcoemorais has joined #openstack-infra20:18
*** marcoemorais has quit IRC20:18
*** marcoemorais has joined #openstack-infra20:18
sdagueso is the integrated queue on hold until that's processed?20:19
clarkbsdague: its more subtle than that20:19
sdagueok20:19
*** mfer has quit IRC20:19
clarkbsdague: it is a fifo, so merge requests for any pipeline are served in order20:20
*** asettle has quit IRC20:20
clarkbsdague: the merge check queue happens to put a ton of jobs on that fifo all at once20:20
*** james_li has quit IRC20:20
jeblairanyway, passing the precedence through should not be a difficult change if anyone wants to hack on that20:21
*** dannywilson has joined #openstack-infra20:21
jeblairnor is spinning up zm0520:21
sdaguejeblair: so what would the impact of that be? (the precedence change)20:22
* SpamapS is just catching up20:22
clarkbjeblair: can we just start with setting the precendence to normal or low?20:22
clarkbjeblair: that solves the issue for the gate20:22
SpamapSfungi: AFAIK, we never proposed tripleo-ansible to openstack/ .. it's been in stackforge for a few days now20:23
jeblairclarkb: no, the _feature_ to set precedce for merge workers does not exist20:23
*** esker has joined #openstack-infra20:23
jeblairsdague: the merge jobs for the merge-check queue could operate at low precedence meaning merge jobs for higher precedence pipelines such as check or gate would be handled first regardless of queue order20:24
*** atiwari has joined #openstack-infra20:24
jeblairsdague: this is the current state for _actual_ jobs, just not _merge_ jobs20:24
*** james_li has joined #openstack-infra20:24
*** esker has quit IRC20:24
sdagueoh, so the issue is that merge workers are always high precedence?20:25
sdagueso merges trump other work20:25
jeblairsdague: merge jobs are handled by zuul-merger workers, not by jenkins.  they only handle merge jobs (and jenkins never handles merge jobs)20:26
jeblairsdague: so there's no conflict between jenkins worker nodes and the mergers20:26
*** mfer has joined #openstack-infra20:26
jeblairsdague: but all of the merge work, including prepping a change for test by a worker, is handled by the mergers and all of that work has the same precedence currently so it's strict fifo20:27
*** packet has quit IRC20:27
sdagueok, so forgive my next stupid question20:27
*** packet has joined #openstack-infra20:28
sdagueis the merge queue having content delaying content being scheduled in other queues?20:28
sdagueor is that just coincidence20:28
clarkbjeblair: gotcha20:28
jeblairsdague: so if we say "the merge job we're putting on the queue has the precedence of the pipeline it is for", then suddenly check and gate merge jobs get serviced first, and merge-check jobs get filled when there isn't anything higher-priority needing to be done20:28
*** packet has quit IRC20:29
*** adalbas has quit IRC20:29
jeblairsdague: sometimes, yes.  the gate queue is currently waiting on mergers to create the changes that it will test, and they are busy doing merge-check work20:29
*** packet has joined #openstack-infra20:30
ekarlsophschwartz: u tharr ?20:30
*** dkranz has quit IRC20:30
sdagueok, so zm05 would just increase the throughput by 25%20:31
*** berendt has quit IRC20:31
jeblairyep20:31
sdaguewhich would still mean that after a nova merge we'd take a 20+ minute delay20:31
*** tonytan4ever has quit IRC20:31
sdagueok, gotcha. Well I have to run for the day here shortly, so the fix isn't something I can really look at.20:32
*** imcsk8 has quit IRC20:32
sdaguedid want to bring up the behavior that I'm seeing, which is definitely impacting our throughput on bug fixes20:33
*** MaxV has quit IRC20:34
sdagueto the point where I'd be in favor of just dropping the merge-check part to get the throughput back and be able to think about the real fix when not in crunch time20:34
fungii'm not convinced the impact is a strict 1:1 though. when zuul's mergers are occupied processing merge-check jobs, nodepool is catching up on building nodes and the providers are catching up on pending deletes, so the ramp back up is a good bit faster compared to an outright delay20:34
*** YorikSar has quit IRC20:34
fungiit does make the jenkins job start times more bursty, but isn't necessarily impacting throughput directly20:35
sdaguefungi: well it's injecting delays. And while it means you get all the nodes at once, there is enough variability in node times, that getting the nodes faster for the first patch in the queue I think really helps20:36
openstackgerritJames E. Blair proposed a change to openstack-infra/zuul: Add precedence to merge jobs  https://review.openstack.org/12384620:36
*** james_li has left #openstack-infra20:36
jeblairsdague, clarkb, fungi: ^20:37
sdagueor jeblair will just fix the world :)20:37
*** Sukhdev has quit IRC20:37
*** dprince has quit IRC20:37
*** dustins has quit IRC20:38
jeblairi just realized we passed change 123456; was uploaded yesterday20:39
jeblairto openstack/training-guides20:39
sdagueyep, I noticed that yesterday20:39
sdagueas I had 123424 IIRC20:39
*** asettle has joined #openstack-infra20:45
*** radez is now known as radez_g0n320:46
*** tonytan4ever has joined #openstack-infra20:49
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch gerrit to project-config  https://review.openstack.org/12289420:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Doc updates for project-config  https://review.openstack.org/12290820:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch jenkins to project-config  https://review.openstack.org/12289320:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch specs to project-config  https://review.openstack.org/12290420:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch gerritbot to project-config  https://review.openstack.org/12290520:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch nodepool to project-config  https://review.openstack.org/12290220:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch accessbot to project-config  https://review.openstack.org/12290320:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Switch zuul and zuul-dev to project-config  https://review.openstack.org/12253320:50
openstackgerritJames E. Blair proposed a change to openstack-infra/config: Move zuul layout into /etc/zuul/layout  https://review.openstack.org/12253220:50
jeblairrebased due to conflict ^20:50
boris-42jeblair hi there20:51
*** imcsk8 has joined #openstack-infra20:51
boris-42jeblair https://review.openstack.org/#/c/123421/ we got from designate team +120:51
boris-42jeblair is it enough?20:52
*** dannywilson has quit IRC20:53
jeblairboris-42: i'd like Kiall to weigh in if possible20:53
boris-42ekarlso ^20:53
*** dannywilson has joined #openstack-infra20:53
boris-42jeblair hm but it's just experimental job on rally patches20:54
boris-42jeblair or just to avoid mistakes?20:54
jeblairboris-42: oh, i thought it was being added to designate.  sorry, it's fine then.20:54
boris-42jeblair no we are just adding experimental to rally, to test that designate benchmarks works properly20:54
boris-42jeblair and experimental to safe some resources20:55
boris-42jeblair of inra20:55
boris-42infra*20:55
*** koolhead17 has quit IRC20:55
jeblairboris-42: aprvd20:55
boris-42jeblair thank you sir20:55
jeblairnp, thanks for bringing that to my attention20:55
clarkbok lunch has been consumed. I am reviewing that zuul change now20:56
jeblairzaro: the jjb config test is going to be disabled during the project-config freeze/move.  we should be extra careful merging jjb changes (or just not merge them) until it's back up again.20:57
arosenanyone know off hand if there is a way to tell tempest which tests you want to run rather than skip?20:57
* arosen through devstack-gate20:57
SergeyLukjanovanteaya, hey, I'm now looking on 3rd party accs requests and have a question re http://lists.openstack.org/pipermail/third-party-requests/2014-September/000008.html20:57
*** dannywilson has quit IRC20:58
clarkbarosen: yes, tox -efull regextomatchhere20:58
SergeyLukjanovanteaya, sounds like we don't like microsoft-cinder-ci20:58
*** dane_leblanc has quit IRC20:58
jeblairzaro: actually, since it's not voting, i'll probably leave it running but it may fail at some point20:58
anteayaSergeyLukjanov: correct20:59
clarkbjeblair: zuul change is nice and simple. +220:59
anteayaSergeyLukjanov: Microsoft iscsi CI works for me21:00
anteayaSergeyLukjanov: how do you feel about that?21:00
*** armax has quit IRC21:00
jeblairanteaya: can you un-wip https://review.openstack.org/#/c/122199/5 ?21:01
jeblairanteaya: we need to merge that after we finish flushing the queue and before you start your extraction work21:01
jeblair(it should be the last change we merge before the freeze)21:01
SergeyLukjanovanteaya, I'm ok with it21:01
jeblairanteaya: and actually, it needs to be rebased so it does not depend on the repo creation21:01
jeblairanteaya: (the repo creation can depend on it, actually)21:02
*** bcourt has quit IRC21:02
*** otherwiseguy has joined #openstack-infra21:02
anteayajeblair: done21:03
anteayaI will now rebase21:03
jeblairanteaya: cool, thanks21:03
*** emagana has quit IRC21:03
anteayathe done was for the un-wip'ing21:03
jeblairunderstood21:04
jeblairanteaya: the thanks was for past and future work ;)21:04
*** emagana has joined #openstack-infra21:04
*** sballe__ has quit IRC21:05
*** emagana has quit IRC21:05
*** tonytan4ever has quit IRC21:05
*** emagana has joined #openstack-infra21:05
anteayajeblair: :D21:05
*** homeless has quit IRC21:05
*** maxbit has quit IRC21:06
*** mfer has quit IRC21:06
*** marcoemorais has quit IRC21:06
*** homeless has joined #openstack-infra21:06
*** marcoemorais has joined #openstack-infra21:06
fungisandywalsh: seconding AJaeger's question on 120858, just want to make sure you intend to import that 0.10 tag for stackdistiller21:07
*** marcoemorais has quit IRC21:07
*** marcoemorais has joined #openstack-infra21:07
fungisandywalsh: nevermind, i see you answered the same question earlier when anteaya asked21:08
* fungi should look more closely at older review comments21:08
*** marcoemorais has quit IRC21:08
ekarlsohas K opened ? :P21:08
sandywalshfungi: I can nuke it if it's a pain ... it's likely stale21:08
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Removes redundant jobs from config gate and check  https://review.openstack.org/12219921:09
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Creates a new repo, project-config  https://review.openstack.org/12165021:09
fungisandywalsh: it's not a pain as long as you don't ask to have it deleted later ;)21:09
anteayadon't review them yet21:09
*** aysyanne has quit IRC21:09
SergeyLukjanovanteaya, I've asked for email in http://lists.openstack.org/pipermail/third-party-requests/2014-September/000011.html21:09
anteayaI forgot to add the source url21:09
anteayasorry21:09
sandywalshfungi: no, that's fine. Best to have it. I think there may be references to it21:09
anteayaSergeyLukjanov: well we don't need email anymore21:09
anteayaSergeyLukjanov: since we have the wikipages21:09
SergeyLukjanovanteaya, correct21:09
*** marcoemorais has joined #openstack-infra21:10
SergeyLukjanovwill create an account for them, anteaya21:10
*** boden has joined #openstack-infra21:10
SergeyLukjanovanteaya, re anteaya, and re http://lists.openstack.org/pipermail/third-party-requests/2014-September/000013.html looks like we need some account name for them21:10
*** marcoemorais has quit IRC21:10
*** marcoemorais has joined #openstack-infra21:10
SergeyLukjanovanteaya, like brocade-fiber-ci21:10
*** marcoemorais has quit IRC21:11
clarkbhttps://jenkins04.openstack.org/job/experimental-tempest-dsvm-full-dibtest/1/ was a success21:11
clarkbjeblair: fungi ^ the test time is a little long but I Think that is true of hpcloud so not super worried about it right now21:11
*** marcoemorais has joined #openstack-infra21:11
*** devkulkarni has joined #openstack-infra21:11
fungiSergeyLukjanov: anteaya: it's fibre (if in the sense of "fibre channel scsi")21:11
SergeyLukjanovfungi, yup, typi21:12
SergeyLukjanovtypo*21:12
openstackgerritA change was merged to openstack-infra/config: Add rally-designate experimental job  https://review.openstack.org/12342121:13
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Creates a new repo, project-config  https://review.openstack.org/12165021:13
clarkbfungi: jeblair also https://jenkins02.openstack.org/job/experimental-tempest-dsvm-neutron-full-dibtest/1/21:13
jeblairanteaya: would it be possible to add the following files from the tools/ directory to the project-config migration: check_projects_yaml_alphabetized.sh check_upstream_url_scheme.py normalize_acl.py run-compare-xml.sh run-layout.sh21:13
jeblairanteaya: ^ oops, sent that in another window21:13
*** wenlock has quit IRC21:13
anteayaSergeyLukjanov: fibre works for me21:14
bswartzis anyone here familiar with ttx's wait_for_tarball.py script?21:14
*** morgan_remote_ has joined #openstack-infra21:14
anteayajeblair: yes we can add to project-config21:14
*** boden has quit IRC21:14
SergeyLukjanovanteaya, okay, so, I'm creating 3 accounts from the third-party-requests ml21:14
anteayacan we add them here: https://etherpad.openstack.org/p/Le2P9YegAx so I can visualize?21:14
anteayaSergeyLukjanov: thanks21:15
SergeyLukjanovanteaya, sorry for delays - /me double booked on internal stuff and release21:15
jeblairanteaya: done, and thanks21:16
anteayaSergeyLukjanov: I understand21:16
anteayaSergeyLukjanov: thanks for clearing the list21:16
fungisandywalsh: i'm +2 on that entire stack, but note that you didn't actually need to add sections for it to jenkins_job_builder/config/projects.yaml since you're not adding any jenkins job templates which need parameters specified (or any at all in fact)21:16
clarkbmordred: dhellmann: https://bugs.launchpad.net/bugs/1373623 just making sure you see that21:16
davidlenwellheya.. I'm sure this is way off topic.. but I was wondering if there what if any hold is there for not actually merging this doc change? https://review.openstack.org/#/c/107024/7  is it just docs so its not high on the list? or is the 1 -1 holding up merge?21:16
uvirtbotLaunchpad bug 1373623 in pbr "prb fails to parse certain requirements" [Undecided,New]21:16
anteayajeblair: thank you, I will create a new github:project-config and reorganized-project-config-03 with these changes for review21:17
sandywalshfungi: thanks. We will be enabling the tests later, but good to know.21:18
jeblairdavidlenwell: just workload for me.  i'm currently working to make space in my queue for that, so i hope to get to infra-manual soon21:19
davidlenwelljeblair: thanks.. just making sure it wasn't held up on my end.21:19
davidlenwelltrying to clear my review queue as well ..21:19
stevemarfungi, bah humbug, it's NOT_REGISTERED again,21:22
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Subscription  https://review.openstack.org/10400421:22
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard-webclient: Subscription events now appear in UI  https://review.openstack.org/11440421:22
fungistevemar: i'll carefully compare logs and gearman status output but it looked correct21:22
jeblairanteaya: i don't think the dependency on https://review.openstack.org/#/c/122199/ is right.  i'd start with a new git branch based on master, then cherry-pick c98c22d4ce11d67fee3e0edeee3424d987b49be0 onto it, then git-review that.21:23
*** emagana has quit IRC21:23
ianwjeblair: sorry, should have called out the version in the CL ... it will need 0.1.32; lifeless told me that it releases once a week and the last release was 7 days ago21:24
*** emagana has joined #openstack-infra21:24
*** asettle has quit IRC21:24
fungistevemar: aha! zuul thinks it should run on bare-trusty but you have configured jenkins to associate it with devstack-trusty instead. i'm betting you need an override in the zuul parameter script to deal with our current overrides, since your job name doesn't have "dsvm" in it21:24
*** asettle has joined #openstack-infra21:24
jeblairianw: okay, so we should hold off on that for now.  thanks.21:24
clarkbfungi: the job name should have dsvm in it if it runs on devstack noes21:24
*** mmedvede has quit IRC21:25
fungiclarkb: good point. perhaps a change to rename the job is better21:25
funginoting that the devstack-based functional test jobs for swift, neutron et all do have dsvm in their job names21:25
fungistevemar: ^21:25
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Removes redundant jobs from config gate and check  https://review.openstack.org/12219921:25
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Creates a new repo, project-config  https://review.openstack.org/12165021:25
anteayajeblair: sorry I can do that next21:26
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Project may now be assigned to multiple project groups.  https://review.openstack.org/11181521:26
clarkbfungi: horizon too.21:26
*** JayJ_ has quit IRC21:26
stevemarfungi, ahh, i didn't think having dsvm in the name was that necessary21:26
*** emagana has quit IRC21:26
fungistevemar: we have a pattern match for that string so that zuul knows what sort of workers to expect jenkins to register for them21:26
*** emagana has joined #openstack-infra21:27
stevemarfungi, gotcha, so i need to add dsvm eh21:27
stevemaralrighty21:27
anteayahome heating issue back in 10 minutes21:27
* krotscheck just updated https://review.openstack.org/111815 to add the project groups that have been added since it went up for review. He hopes he won’t have to do that too often.21:27
krotscheckAlso, I’ve got 3x +2 on https://review.openstack.org/#/c/111814/.21:27
*** emagana has quit IRC21:28
*** emagana has joined #openstack-infra21:29
*** armax has joined #openstack-infra21:29
openstackgerritMichael Krotscheck proposed a change to openstack-infra/config: Add puppet-storyboard to storyboard  https://review.openstack.org/11843521:29
openstackgerritSteve Martinelli proposed a change to openstack-infra/config: Rename osc-functional to osc-dsvm-functional  https://review.openstack.org/12385821:31
stevemarfungi, ^21:32
fungistevemar: lgtm21:33
stevemarhoping to get this one in quick :) i'm testing out a bunch of changes locally :D21:33
openstackgerritA change was merged to openstack-infra/jeepyb: Multiple project groups.  https://review.openstack.org/11181421:33
anteayaback21:33
*** emagana has quit IRC21:33
*** devkulkarni has left #openstack-infra21:34
*** emagana has joined #openstack-infra21:34
fungikrotscheck: i approved the jeepyb prereq for the groups support in projects.yaml, though that latter change is going to merge-conflict with the flurry of new project and similar changes we're rushing to merge before the freeze in a couple hours. should we hold off and repropose it to the project-config repo tomorrow?21:34
*** alexpilotti has joined #openstack-infra21:34
openstackgerritAnita Kuno proposed a change to openstack-infra/config: Removes redundant jobs from config gate and check  https://review.openstack.org/12219921:35
openstackgerritA change was merged to openstack-infra/storyboard: Event actor now added to message payload  https://review.openstack.org/12020121:35
anteayajeblair: better ^?21:35
fungikrotscheck: alternatively we can try to get it in as the last change before the freeze21:35
*** alexpilotti has quit IRC21:35
jeblairfungi: second to last :)21:35
krotscheckfungi: I’m ambivalent.21:35
fungijeblair: penultimate, right21:36
*** alexpilotti has joined #openstack-infra21:36
fungikrotscheck: but there are likely to be further updates from these other changes we're merging ahead of it21:36
fungiso it'll almost certainly need another rebase and probably additional entries21:36
krotscheckI don’t mind doing another patch, OR moving everything to a different CR altogether. I just don’t want to continually dance on quicksand.21:37
openstackgerritMichael Krotscheck proposed a change to openstack-infra/storyboard: Python based daemon management  https://review.openstack.org/12289021:38
openstackgerritA change was merged to openstack-infra/config: Notigen - OpenStack-like notification generator/simulator.  https://review.openstack.org/12084921:39
openstackgerritA change was merged to openstack-infra/config: Klugman - cmdline tool and client library for StackTach.v3  https://review.openstack.org/12083821:40
openstackgerritSumit Naiksatam proposed a change to openstack-infra/config: UI, automation repos for group based policy project  https://review.openstack.org/12292221:40
mwhudsonum, this feels like a silly question, but where do the artifacts uploaded at the end of something like https://jenkins06.openstack.org/job/gate-tempest-dsvm-full/9482/consoleText end up?21:40
jeblairboris-42: 123396 needs a rebase21:40
boris-42jeblair I am facing some small issue with new job21:40
clarkbmwhudson: on http://logs.openstack.org there should be a link to them in the job21:40
boris-42jeblair http://logs.openstack.org/42/123442/2/experimental/gate-rally-dsvm-designate-rally/ed32116/console.html21:40
jeblairmwhudson: huh, i thought we had a link in there21:40
boris-42jeblair 2014-09-24 21:32:23.271 | /tmp/hudson7322189625713021231.sh: line 20: syntax error near unexpected token `{{'21:41
boris-42how to access this file?)21:41
clarkbmwhudson: also the report from jenkins/zuul in gerrit has a link to them21:41
jeblairDetailed logs: http://logs.openstack.org/74/123774/2/gate/gate-tempest-dsvm-full/98861ab/21:41
jeblairmwhudson: ^ there it is21:41
mwhudsonoh yes21:41
mwhudsonsee, i said it was a silly question :)21:41
mwhudsonthanks21:41
jeblairboris-42: it's one of the shell builders in the jjb configuration for that job21:41
lifelessmwhudson: oh hai21:41
krotscheckbtw, does anyone have the time to poke at the storyboard logs to see if rabbit is now not dying on us?21:41
mwhudsonlifeless: hello!21:41
jeblairkrotscheck: can do21:41
openstackgerritA change was merged to openstack-infra/config: Quince - Quincy-compliant API implementation for StackTach.v3. Supports Winchester.  https://review.openstack.org/12083921:41
krotscheckjeblair: awesome, tahnks21:42
openstackgerritA change was merged to openstack-infra/config: Sandbox - DevStack-like dev environment for StackTach.v3  https://review.openstack.org/12084421:42
jeblairkrotscheck: [Wed Sep 03 07:13:22 2014] [error] ConnectionClosed21:43
*** bswartz has quit IRC21:43
jeblairkrotscheck: that's the last connection closed exception in the log21:43
krotscheckjeblair: That’s promising.21:43
jeblairkrotscheck: https://etherpad.openstack.org/p/oL5fQPtEVh21:43
*** marcoemorais has quit IRC21:43
jeblairkrotscheck: there's the most recent related log entries21:43
jeblairwhich also look promising21:44
openstackgerritA change was merged to openstack-infra/config: StackDistiller - A notification extraction and transformation library.  https://review.openstack.org/12085821:44
jeblair(eg, the underlying behavior is still happening but it looks like we are recovering)21:44
krotscheckIndeed! The [error] block there looks a little weird.21:44
openstackgerritA change was merged to openstack-infra/config: Timex - A time expressions library  https://review.openstack.org/12085221:44
*** asettle has quit IRC21:44
krotscheckBut yeah, looks like the recovery is there.21:44
jeblairkrotscheck: i think that's from apache21:44
krotscheckAh, point.21:44
jeblairkrotscheck: (vs the INFO/WARNING which are from python logging)21:44
krotscheckCooooooooool21:45
krotscheckjeblair: Thanks for checking :)21:45
jeblairkrotscheck: np; i just pasted in a line i missed21:45
arosenI have a git repo on stackforge that needs to be in another projects requirement.txt file in order to work. If I haven't push anything to pip is there anyway to get around this?21:46
jeblairkrotscheck: and while i'm here; i've pasted some other random errors in the logs too (no other context for those)21:46
boris-42jeblair so not much lines here … https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/rally.yaml#L51-L6021:46
openstackgerritA change was merged to openstack-infra/config: Winchester - A notification event processing library based on persistent streams.  https://review.openstack.org/12118221:46
openstackgerritA change was merged to openstack-infra/config: StackTach.v3 Quincy REST API  https://review.openstack.org/12058221:46
boris-42jeblair oh this https://github.com/openstack-infra/config/blob/master/modules/openstack_project/files/jenkins_job_builder/config/rally.yaml#L91-L10321:47
clarkbarosen: you can depend on tarballs or git urls. we usually recommend against this21:47
openstackgerritA change was merged to openstack-infra/config: notification_utils: utility library for OpenStack notifications.  https://review.openstack.org/12110921:47
krotscheckjeblair: Thanks, I’ll dig into those.21:47
*** harlowja has quit IRC21:47
fungiclarkb: and also it isn't supported by pbr (re: the bug just filed a moment ago)21:47
*** harlowja has joined #openstack-infra21:47
clarkbfungi: ya that21:47
openstackgerritA change was merged to openstack-infra/config: Allow nova-core/milestone to rebase nova changes  https://review.openstack.org/12222021:48
arosenclarkb:  how do changes to clients get tested within tempest? I'm guessing they aren't?21:48
fungiarosen: devstack-gate clones, checks out and installs from local source21:48
arosenfungi: though  when tempest is run doesn't it build a virtual env to run the tests?21:49
clarkbtempest is run with site packages enabled so it uses all of the things installed globally21:49
fungiarosen: there is a venv tempest installs itself into, but the clients/libs/servers are pip-installed from source into the system global context not a venv21:49
arosenAh so that prevents one from adding dependencies to tempest and having it magically work because another project has them as well?21:50
mwhudsonif i want to run devstack+tempest 'the way zuul does it', is it as simple as "git clone git://git.openstack.org/openstack-infra/devstack-gate && ./devstack-gate/devstack-vm-gate-wrap.sh" ?21:51
fungimwhudson: it's more like 'git clone git://git.openstack.org/openstack-infra/devstack-gate && view README.rst'21:52
*** marun has quit IRC21:52
jeblairfungi, clarkb: 123802 needs an extra review21:52
clarkblooking21:53
fungi(well, view devstack-gate/README.rst)21:53
mwhudsonfungi: ah, thanks21:53
*** dannywilson has joined #openstack-infra21:53
clarkbfungi: though now we can point people to dib at least for trusty devstack base node stuff21:53
fungimwhudson: yeah, it's not extremely straightforward but we did try to document how to do it21:53
clarkbfungi: the tests are passing so that should get you a like for like install. However it may not set up users in a way that is consumable by others21:53
jeblairclarkb, fungi: should we '#status notice' a 2-hour notice of the freeze?21:53
fungiclarkb: sure, as soon as we put the images somewhere they can download from, right?21:53
clarkbmordred: ^ have you considered how to fix that?21:53
lifelessmwhudson: whatcya hacking on?21:54
clarkbjeblair: sure. I think reminders are good21:54
mordredclarkb: only marginally21:54
mwhudsonlifeless: openstack on arm6421:54
mordredclarkb: but we should certainly do that21:54
lifelessmwhudson: shiny!21:54
mwhudsonlifeless: i can run tempest and get a bunch of failures21:54
mordredclarkb: we could also upload them to swift21:54
mwhudsonbut i don't understand how some of these work on intel21:54
mordredclarkb: or - rather, we could upload to swift then when we "upload" to glance we give the swift URL21:54
clarkbmordred: ya21:54
clarkbmordred: but one step at a time21:55
fungijeblair: yeah, though hopefully us freezing config changes for a day isn't something terribly impacting to the rest of our dev community21:55
mwhudsonso i want to understand how they do, and then try to figure out which arm64 failures are 'real'21:55
mordredtotes21:55
lifelessmwhudson: ah21:55
clarkbI want to get rax uploads working and switch to dib, then we can publish to the masses21:55
jeblairfungi: mostly i'd like to give people a chance to come out of the woodwork if we missed something21:55
fungisure21:55
mwhudsonlifeless: also interested in running arm64 tests from zuul eventually21:55
*** dims_ has joined #openstack-infra21:55
*** mriedem has quit IRC21:55
mordredclarkb: what's up with rax uploads?21:56
fungimwhudson: you have arm64 boards, or do you mean "eventually when you can get your hands on some"?21:56
jeblairThe openstack-infra/config repo will be frozen for project-configuration changes starting at 00:01 UTC.  If you have a pending configuration change that has not merged or is not in the queue, please see us in #openstack-infra.21:56
clarkbmordred: we need to disable nova agent and set hvm in the metadata21:56
jeblairclarkb, fungi: ^ ?21:56
clarkbmordred: ianw wrote a patch to do it21:56
mwhudsonfungi: we have a small pile of them, yes21:56
clarkbjeblair: lgtm21:56
fungijeblair: lgtm21:56
anteayajeblair: lgtm21:57
mordredclarkb: oh - right. good21:57
jeblair#status notice The openstack-infra/config repo will be frozen for project-configuration changes starting at 00:01 UTC.  If you have a pending configuration change that has not merged or is not in the queue, please see us in #openstack-infra.21:57
openstackstatusjeblair: sending notice21:57
mwhudsonalmost certainly not enough to do anything interesting in this area yet, which is something else i want to think about21:57
clarkbjeblair: fungi 123802 adds github-org back intp projects.yaml21:57
clarkbjeblair: fungi: is that something we care about?21:57
*** rmcall has quit IRC21:57
*** todd_dsm has joined #openstack-infra21:57
*** emagana has quit IRC21:57
jeblairclarkb: yes, we've -1d for that before21:57
openstackgerritDavid Shrewsbury proposed a change to stackforge/dox: Use base class for config file parsers  https://review.openstack.org/12386421:57
fungiclarkb: it should probably be cleaned up, yes21:57
clarkbok I will -121:57
*** dims__ has joined #openstack-infra21:58
-openstackstatus- NOTICE: The openstack-infra/config repo will be frozen for project-configuration changes starting at 00:01 UTC. If you have a pending configuration change that has not merged or is not in the queue, please see us in #openstack-infra.21:58
ianwclarkb / jeblair : yeah, https://review.openstack.org/#/c/123639/ (Add meta option for passing meta-data)21:58
*** emagana has joined #openstack-infra21:58
*** emagana has quit IRC21:58
fungimwhudson: neat. the most powerful arm boards i have for now are 32-bit (quad-core though at least)21:58
mwhudsonfungi: yeah, for a long time all arm32 board had at least one terrible flaw21:58
*** emagana has joined #openstack-infra21:58
openstackstatusjeblair: finished sending notice21:59
mwhudsonlike, terrible IO, or requiring random binary firmware to boot or ...21:59
*** rmcall has joined #openstack-infra21:59
*** dims has quit IRC21:59
mwhudsonthe arm64 is all under NDA still, but i think i can say that it's a heck of a lot nicer to work with21:59
openstackgerritDavid Shrewsbury proposed a change to stackforge/dox: Use base class for config file parsers  https://review.openstack.org/12386421:59
clarkbI should resurrect my beagle*21:59
fungimwhudson: heh, all the ones i have still have pretty atrocious i/o and require random binary firmware to boot21:59
mwhudson8 cores, multiple gigs of ram, sata, ssd, 10G ethernet...22:00
mwhudsonfungi: i said "at least one" carefully :)22:00
*** dims_ has quit IRC22:00
clarkbok 123802 -1'd22:00
arosenclarkb:  i tried: DEVSTACK_GATE_TEMPEST_REGEX='tempest.scenario.test_congress_*' but it still seems to run all the tests?22:00
*** david-lyle has quit IRC22:00
reedfungi, jeblair: what chances has this patch to go in before the freeze? https://review.openstack.org/#/c/119364/22:00
fungimwhudson: i'd gladly give up the additional cores, extra ram, faster network and disk for sane booting and non-proprietary firmware updating tools22:01
clarkbarosen: if you aren't running the -efull env it will do that22:01
*** david-lyle has joined #openstack-infra22:01
clarkbarosen: you may need to set another flag to do the full run22:01
jeblairreed: it's not affected by the freeze22:01
fungimwhudson: but then again, what you have sounds server-oriented while i'm trying to do ultra-compact stuff instead22:01
reedof course, sorry22:01
mwhudsonyes, this is very much server oriented22:02
mwhudsonfungi: beagle bone black has non insane booting i think?22:02
mwhudsonit's omap and so terrible IO of course :)22:02
arosenclarkb:  it looks like we need to add another var to devstack-gate to do this22:02
*** mjturek has quit IRC22:03
*** harlowja has quit IRC22:03
ianwjeblair: two questions; firstly wasn't 100% sure what to do with config file failures, like the meta-data being too long, secondly i can probably put something in to check the values make it to the fake provider, but that doesn't seem that useful22:03
arosenclarkb:  it only has a way to run with: -eall22:03
clarkbarosen: -eall may be the thing22:03
clarkbmtreinish: ^22:03
*** harlowja has joined #openstack-infra22:03
fungimwhudson: oh, and support for the entire soc in mailline linux kernel releases without needing hacked-up drivers out of android22:03
*** rmcall has quit IRC22:03
jeblairianw: i'm not sure i have the context for this conversation22:03
mwhudsonfungi: haha22:04
fungis/mailline/mainline/22:04
*** rmcall_ has joined #openstack-infra22:04
mwhudsonyes22:04
ianwjeblair: sorry, https://review.openstack.org/#/c/123639 :)22:04
jogofungi: its time to check if you have patches in repos that don't have PTLs running:  https://wiki.openstack.org/wiki/PTL_Elections_September/October_201422:04
jogofungi: so we can run against eachother22:04
clarkbare we going to make fungi ptl of all te misfit projects?22:05
anteayahe needs the work22:05
jeblairreed, fungi: in addition to devstack.org, i think we need to get openstackid.org added to the 'openstack' rackspace account to manage dns22:05
clarkbwhy did openstackid get its own domain?22:05
fungijeblair: oh, right. i believe jbryce registered it but it's not manageable in rackspace yet22:05
jeblairclarkb: openid best practice22:05
ociuhanduSergeyLukjanov, anteaya: I was away and read now the talk you had on the naming for the microsoft  cinder CI. I think iscsi should not be in the name as we’re currently adding samba support and that will also be tested in this CI.22:05
reedjeblair, how can I help?22:05
clarkbjeblair: oh interesting22:06
fungiclarkb: more specifically, to avoid cross-domain scripting within trusted subdomains22:06
clarkbgotcha22:06
fungiclarkb: so authentication fior openstack.org sites is not being handled by an openstack.org site22:06
clarkbya22:06
fungiextra safety precaution22:06
clarkbwhich as a user seems weird22:06
clarkbbut whatever22:06
fungiit's to deal with browser crazy22:07
jeblairclarkb: also i think morganfainberg told us to :)22:07
jeblairreed: tbh, i don't know the process22:07
fungii like to blame everything on morganfainberg22:07
anteayaociuhandu: it took me a week and a half to get you the account created22:07
*** dannywilson has quit IRC22:07
mwhudsonfungi: another advantage of server parts is not having to worry about mali/powervr/...22:07
fungireed: it's on me to find out how to get the authoritative nameservers for those domains switched to rackspace's so they can be added in our tenant for dns management22:07
*** dannywilson has joined #openstack-infra22:07
anteayaociuhandu: I don't have any energy to discuss renames now since we are spliting the config repo tonight/tomorrow22:08
fungireed: but if you want to find out for me, that'll help speed it along22:08
anteayaociuhandu: if you want to discuss it try me next week22:08
ociuhanduanteaya: ok, thanks. Will do :)22:08
reedfungi, easy peasy :) I have the contact with the agency that manages those domains22:08
anteayaociuhandu: thanks for your understanding22:08
*** jgrimm is now known as zz_jgrimm22:08
clarkbianw: are the limits of 5 entries and 255 chars each imposed by nova?22:08
ociuhanduanteaya: sorry for missing the discussion earlier :(22:09
*** harlowja has quit IRC22:09
*** harlowja_ has joined #openstack-infra22:09
clarkbianw: and can we apply them globally to an image or must we provide then when we boot servers?22:09
jeblairreed, fungi: it's probably a little bit of asking the agency to point NS records at the rackspace NS records, and it's also probably a bit of asking rackspace to add it to the openstack accound as a domain that it should be able to manage.22:09
fungijeblair: yeah, depending on the provider, some don't let you add a domain for management until they see the authroitative glue listing them in the corresponding tld22:10
anteayaociuhandu: that is okay, since I wouldn't have had the energy to dicuss it then either, we would have just not created an account for you22:10
ianwclarkb: the limit i got from http://docs.openstack.org/developer/python-novaclient/ref/v1_1/servers.html22:10
jeblairi mean, there's a 'create domain' button there.  i could try pressing it.  :)22:10
anteayaociuhandu: so I think you are further ahead with the account now then you were22:10
*** datsun180b has quit IRC22:10
jeblairmy guess is that just registers it though.  probably the process involves opening a ticket.22:10
ociuhanduanteaya: that’s for sure, thanks :)22:10
anteayaociuhandu: okay22:10
openstackgerritBob Callaway proposed a change to openstack-infra/config: Add puppet-manila to Stackforge  https://review.openstack.org/12380222:10
*** boden has joined #openstack-infra22:11
*** imcsk8 has quit IRC22:11
*** imcsk8 has joined #openstack-infra22:11
stevemarwhats the impact of the config repo being 'frozen' ?22:11
mattoliverauMorning22:11
fungijeblair: as reed pointed out, we apparently already have openstackid.org pointed to rackspace's nameservers at the registrar22:11
anteayamorning mattoliverau22:11
clarkbianw: grepping for meta on that page it looks like you can attach meta data to images. at least when you make a snapshot. In our case we are uploading a new image directly. I think we should see if we can attach meta data when we upload the image22:11
anteayastevemar: we won't be accepting changes to config during the freeze22:12
clarkbianw: as I think that is where rax said they wanted it?22:12
stevemaranteaya, eep!22:12
*** dannywilson has quit IRC22:12
anteayastevemar: any changes that are to repos taht are going to end up in the project-config repo will have to be abandoned22:12
*** juzuluag has joined #openstack-infra22:12
anteayastevemar: and resubmitted from a project-config parent commit22:12
stevemaranteaya, whoa! okay... can this patch get love? https://review.openstack.org/#/c/123858/22:13
dhellmannclarkb: thanks. I wonder if it's even legal to have a '-' in the middle of an egg name?22:13
anteayastevemar: have you any patches that are not currently approved and in the gate queue?22:13
clarkbdhellmann: that I do not know22:13
clarkbdstufft: do you know ^22:13
morganfainbergjeblair, fungi, huh?22:13
dstufftin an egg name? Yea I think so22:13
fungimorganfainberg: blaming you for the suggestion in the openstack openid provider design summit session that it should use a separate domain name22:14
anteayastevemar: I love it22:14
morganfainbergfungi, oh yeah blame away.22:14
stevemaranteaya, yay22:14
*** marcoemorais has joined #openstack-infra22:14
fungimorganfainberg: s/blaming/crediting/ of course! ;)22:14
morganfainbergfungi, :)22:14
anteayaclarkb SergeyLukjanov jeblair 123858 adds dsvm- to job names so they will run, +2 from fungi, +1 from me22:14
clarkbanteaya: stevemar I am reviewing now22:15
anteayaclarkb: thanks22:15
anteayastevemar: anything else?22:15
morganfainbergfungi, i took it as a compliment! I regularly use blame as a positive thing ;) [sometimes i don't phrase it like that though to save people getting upset]22:15
ianwclarkb: ok, i can move it22:15
*** boden has quit IRC22:15
*** marun has joined #openstack-infra22:15
anteayamorganfainberg: at least we are thinking of you22:15
*** todd_dsm has quit IRC22:16
*** flaper87 is now known as flaper87|afk22:16
fungimorganfainberg: git has completely changed my use of that word22:16
morganfainberganteaya, hehe22:16
anteaya:D22:16
morganfainbergfungi, svn blame, now that was always a negative thing :P22:16
fungicvs needed blame22:17
morganfainbergfungi, ++22:17
clarkbmorganfainberg: 'annotate' is the politically correct alias22:17
stevemaranteaya, nope, that's it22:18
fungiit had annotate, which was way too politically correct22:18
* mwhudson looks at errors from puppet22:18
morganfainbergehe22:18
anteayajeblair: you wanted to merge this, is is rebased, has no dependencies and jenkins is +1: https://review.openstack.org/#/c/122199/22:18
clarkbfungi: borland starteam needed anything like that22:18
fungiclarkb: annotate actually came from cvs22:18
*** homeless has quit IRC22:18
anteayastevemar: kk, thanks for paying attention22:18
alexpilottifungi: hi, I need a last +2a on a config commit waiting since Sept 8th: https://review.openstack.org/#/c/119909/22:18
clarkbyes  Ihave used borland starteam22:18
clarkbdo not recommend22:18
mwhudsonError: Could not install module 'puppetlabs-inifile' (v1.0.0)22:18
mwhudson  Module 'puppetlabs-inifile' (v1.1.3) is already installed22:18
alexpilottifungi: any chance you could please help with this one?22:18
mwhudsonfungi: is there a version of the instructions for trusty rather than precise?22:18
mwhudson(if that's actually my problem)22:18
stevemaranteaya, np! i had no idea this happens... is there a freeze like this every release?22:19
anteayano22:19
anteayafirst time22:19
clarkbmwhudson: I think jesusaurus figured out why that happens (bash array iteration doesn't garuntee order)22:19
fungialexpilotti: yep, i'm working on that one now22:19
clarkbmwhudson: that said it should be harmless. I think we are hitting it and should probably push a change to bump the version and maybe be more explicit about the order22:19
anteayaour config repo is too big and only limited folks can navigate it22:19
morganfainbergclarkb, it.. doesn't? and they still call it an array?!22:19
anteayawhich reduces our reviewers22:19
clarkbmwhudson: and finally the instructions for precise should be the same as trusty22:19
fungialexpilotti: trying to get it in before the freeze in a couple hours22:19
alexpilottifungi: wow, perfect timing tx!22:19
anteayaso split to make it easier to review22:19
clarkbmorganfainberg: well its an associative array22:19
mwhudsonclarkb: oh22:19
morganfainbergclarkb, ah ok22:19
morganfainbergclarkb, sure.22:19
morganfainbergclarkb, makese sense once i start thinking about the use of it.22:20
alexpilottifungi: that’d be really great!22:20
mwhudsonclarkb: because later stuff fails too22:20
mwhudsoncat: .ssh/id_rsa.pub: No such file or directory22:20
clarkbmwhudson: I think it expects you to have a key? that shouldn't be related to the puppet modules stuff22:20
*** andreykurilin_ has quit IRC22:21
*** doug-fish has left #openstack-infra22:21
mwhudsonoh22:21
morganfainberganteaya, I keep trying to review things in config, and it's hard for me to get my head wrapped around all the subtleties (and keep keystone knowledge even remotely close to the surface) even though i've worked on contributing to infra/config22:21
morganfainberganteaya, i totally support the split :)22:21
* mwhudson tries to read instructions more carefully again22:21
*** gondoi is now known as zz_gondoi22:21
anteayamorganfainberg: good :D22:21
*** imcsk8 has quit IRC22:21
*** imcsk8 has joined #openstack-infra22:21
anteayaand yes, I think many folks, including me, are singing the same refrain22:21
*** shashankhegde has quit IRC22:22
*** signed8bit has quit IRC22:22
jesusaurusclarkb: yeah, bash arrays iterate in an arbitrary order based on their internal hash value (not they key value)22:23
jesusaurusclarkb: yolanda has a change up to create an array of keys to iterate through to enforce order22:23
clarkbjesusaurus: yeah so if we isntalled that module earlier with the version we wanted more reliably the error would go away22:23
clarkbjesusaurus: is that pushed up?22:23
jesusaurusclarkb: https://review.openstack.org/#/c/12213222:24
alexpilottifungi: tx!22:24
jesusaurusclarkb: indexed arrays are iterated in index order, associative arrays are iterated in a seemingly arbitrary (but deterministic) order22:25
openstackgerritA change was merged to openstack-infra/config: Rename osc-functional to osc-dsvm-functional  https://review.openstack.org/12385822:25
adam_ghere's a config change that'll get a failing job out of the stable/icehouse gate: https://review.openstack.org/#/c/123822/22:25
*** julim has quit IRC22:25
mwhudsonCould not find declared class openstack_project::single_use_slave22:26
mwhudsonError: Puppet::Parser::AST::Resource failed with error ArgumentError: Could not find declared class openstack_project::single_use_slave at line 1 on node devstack.openstacklocal22:26
mwhudsonseems bad?22:26
clarkbjeblair: there is an interesting thread on dealing with db migrations in feature brances (for the neutron lbaasv2 branch). I don't know enough about alembic to chime in but probably something we want to track22:26
jheskethMorning22:26
clarkbmwhudson: ya that should be fatal. And is probably related to your puppet module path22:26
anteayamorning jhesketh22:26
fungiclarkb: i think maybe it's a subset of the discussion from a cycle or two ago about how to deal with migrations on stable branches and rc22:26
*** maxbit has joined #openstack-infra22:27
clarkbfungi: hrm could be22:27
*** homeless has joined #openstack-infra22:27
clarkbbasically the migrations branch22:27
clarkband what do22:27
fungiclarkb: the whole "can't go back and reorder later" problem22:27
openstackgerritA change was merged to openstack-infra/config: Run the precise nonha job on tripleo projects  https://review.openstack.org/12209922:27
clarkbmwhudson: in the puppet apply command from the readme22:27
fungiclarkb: i think the sanest suggestion at the time was to have gaps or stubs inserted which allowed inserting migrations at strategic points in the chain22:27
mwhudsonclarkb: i'm just running the command from the readme :/22:28
mwhudsonoh22:28
clarkbmwhudson: make sure that the --modulepath=/some/path is pointing to where you have config cloned22:28
mwhudsonyay sudo22:28
clarkboh ya if there are perms issues that will help too :)22:28
mwhudsonwell it's more that i'm running sudo -s so $HOME is still ~ubuntu22:28
clarkbmwhudson: also if you find anything about the doc to be particularly troubleing we do accept patches and are happy to update if we get feedback too22:28
openstackgerritIan Wienand proposed a change to openstack-infra/nodepool: Add meta option for passing meta-data  https://review.openstack.org/12363922:29
ianwclarkb: ^ moved to create_image() call22:29
clarkbmwhudson: this process should be getting easier in the nearish future as we move more to prebaked images that can be public or built on your end22:29
clarkbmwhudson: however we build them for x86 and not sure wat would be involved for arm6422:29
mwhudsonwell automation++22:29
*** imcsk8 has quit IRC22:29
*** imcsk8 has joined #openstack-infra22:29
openstackgerritIan Wienand proposed a change to openstack-infra/nodepool: Add meta option for passing meta-data  https://review.openstack.org/12363922:30
mwhudsonso long as the way the images were baked was public22:30
openstackgerritA change was merged to openstack-infra/config: Adds cloudbase-init to StackForge  https://review.openstack.org/11990922:30
fungimwhudson: yep, we document and release all the code/configuration which does it, except for redacting credentials22:30
clarkbmwhudson: yup we are trying to use disk image builder to do it. I would link you to things but we are in the middle of completely reorganizing the code right now22:30
clarkbwe are super public about everything we do22:30
anteayamwhudson: the reorganization should be complete tomorrow sometime22:31
mwhudsonoh22:31
mwhudsonthat's a fairly short timeframe :)22:31
anteayayeah22:31
mwhudsonmaybe i should just go and sit on the beach for the rest of the day or soemthing22:31
openstackgerritA change was merged to openstack-infra/config: Packstack changes  https://review.openstack.org/11833322:31
anteayawe have been prepping for a week and a half22:31
mwhudsongo me for timing22:31
anteaya:D22:31
clarkbianw: you updated the snapshotimageudpater which I think is ok. we want to update the diskimageupdater instead22:32
*** salv-orlando_ has joined #openstack-infra22:32
mwhudson(i've been meaning to come and have this conversation in here for at least a couple of months)22:32
anteayatomorrow about this time will be a great day for it22:32
mwhudsonrunning things under sudo su and in ~root is working a lot better22:32
*** dannywilson has joined #openstack-infra22:33
mwhudsonoh yeah, so the instance i'm running on this is behind a proxy22:34
mwhudsonis that just going to lead to pain?22:34
clarkbmwhudson: if you have http_proxy set most things will work. I believe that pip may not always work though22:34
*** nijaba_ has joined #openstack-infra22:35
*** asettle has joined #openstack-infra22:35
mwhudsonok22:35
*** nijaba has quit IRC22:35
*** salv-orlando has quit IRC22:35
*** salv-orlando_ is now known as salv-orlando22:35
clarkbmwhudson: because puppet pip provider doesn't inherit http_proxy and because pip (well really python requests) has had tuoble going through proxies22:35
fungiwow, some serious server-to-server lag on freenode at the moment22:35
mwhudsonsomething seems to be hanging22:35
clarkbhrm would our dib stuff need to run in qemu to arm64?22:36
clarkbI am sure fungi and lifeless would know22:36
*** bknudson has quit IRC22:36
sandywalshAJaeger fungi jeblair: thanks for the help on those new projects! Much appreciated!22:37
mwhudsonqemu-aarch64 is a thing now22:37
mroddenaarch64?22:37
fungiclarkb: depends on how you bootstrap the filesystem. you can cross-strap and cross-build enough for a bootable system depending on your needs22:37
jeblairsandywalsh: you're welcome22:38
stevemarfungi, how'd did you check the logs for the why a job was set as NOT_REGISTERED22:38
openstackgerritSteve Baker proposed a change to openstack-infra/elastic-recheck: Add query for rare heat delete race bug 1355942  https://review.openstack.org/12387622:38
jeblairincidentally, we have added 19 projects this week; current count is 426.22:38
uvirtbotLaunchpad bug 1355942 in tempest "tempest.api.orchestration.stacks.test_stacks fails An object with that identifier already exists" [Undecided,Invalid] https://launchpad.net/bugs/135594222:38
clarkbfungi: well in our case I think we expect some things to compile and not sure how you get pip installs to do that right22:38
ianwclarkb: yeah, ok ... have to think about that a bit, will fix soon22:38
clarkbianw: no rush and thank you22:38
jeblairbiab22:38
fungiclarkb: i've definitely built arm binaries (even linux kernels) and chroots full of arm-targeted debian packages, you just can't expect to run the contents without an emulator22:39
clarkbfungi: yes you can cross compile, but how does pip do that when the gcc in the chroot is arm6422:39
fungibuilt and bootstrapped arm binaries/filesystems from an amd64-based system22:39
clarkbfungi: you would have to cross compile with a gcc from elsewhere22:39
fungiclarkb: right, i didn't use pip for any of it22:39
clarkbor put the cross compile toolchain in the image too even though you only need it at image build time22:40
*** _buzz has quit IRC22:40
fungiclarkb: just downloading arm architecture packages and unpacking them into the chroot22:40
clarkbI am sure these things can be sorted22:40
fungiyeah, debian/ubuntu cross-architecture-strapping is already nontrivial, they've done a ton of work to get it to where it is. i wouldn't expect other package management systems to have even considered the possibility probably22:41
clarkbya22:41
mwhudsonclarkb: would running natively be especially hard?22:42
mwhudsonassuming hw access22:42
*** _buzz has joined #openstack-infra22:42
fungistevemar: those service logs are accessible to shell users on the zuul server. there aren't many of us22:42
jogowe are about to hit a gate reset  https://jenkins04.openstack.org/job/gate-tempest-dsvm-full/14405/consoleFull22:42
stevemarfungi, ah okay, didn't want to bug you if i could do it myself22:43
clarkbmwhudson: yes, because we don't run any hardware22:43
clarkbmwhudson: if however you wanted ot run the things on your end that should work fine22:43
jogoand we have a nova patch that helps with one of our bigger gate bugs  that bounced out of the gate https://review.openstack.org/#/c/123774/222:43
jogocan we get it promoted?22:43
clarkbwe just wouldn't be able to test it very effectively22:43
stevemarfungi, could you take another looksy at why the osc-functional job is NOT_REGISTERED? i thought the last patch would have fixed it22:43
mwhudsonclarkb: if you had access to a hypothetical arm64 cloud?22:43
fungimwhudson: you might want to consider teaming up with tripleo though, and working on an alternative-processor-architecture region to the tripleo community cloud22:44
jogo287 hits in 10 days22:44
fungistevemar: did it already merge?22:44
jogofungi: ^22:44
clarkbmwhudson: if we had access to that it should be relatively easy. we intend on having image builds be farmed out to workers some of which could run on arm64 cloud22:44
fungijogo: i can promote that, just a sec22:44
* anteaya prepares to delete her github/project-config repo and replace it with the latest sample22:44
stevemarfungi, yep, i waited for the POST job to finish before recheck-ing something22:44
jogofungi: thanks!22:44
clarkbmwhudson: if you run https://git.openstack.org/cgit/openstack-infra/config/tree/tools/build-image.sh as a user with sudo access you should be able to generate the images we are trying to move to22:45
clarkbmwhudson: I think you need to override CONFIG_REF and set it to eg master right now though. the rev parse thing there doesn't work as I expected it to22:45
fungistevemar: once that merges, the puppet update pulse hits roughly every 15 minutes and at some point in the ensuing 10 minutes pushes that to zuul. you should see a timestamp for zuul's most recent config update at the bottom of its status page22:45
mwhudsonclarkb: let me finish my fight with devstack-gate ... :)22:45
stevemarfungi, so you're saying I need to work on one of my virtues :)22:46
fungistevemar: yes, laziness is the highest of the hacker virtues22:46
*** openstackgerrit has quit IRC22:47
stevemarfungi, lol - i'll recheck again when i see the time stamp change, thanks for having all the answers :)22:47
*** openstackgerrit_ has joined #openstack-infra22:47
*** openstackgerrit_ is now known as openstackgerrit22:48
*** xyang0 has quit IRC22:49
anteayajeblair: please confirm this is the expected output of the filter branch before the reoganization of project-config: https://github.com/anteaya/project-config22:49
*** ZZelle_ has joined #openstack-infra22:51
*** dkehn_ has joined #openstack-infra22:51
lifelessclarkb: yes, to build arm64 from intel22:54
lifelessclarkb: arm64 from arm64 is a nobrainer22:54
lifelessclarkb: some of our moonshot team have done the dib support forthis already22:54
fungijogo: i promoted 123774,2 just now, right after that failure reported22:54
openstackgerritKhai Do proposed a change to stackforge/python-jenkins: Support node configuration  https://review.openstack.org/8960422:55
*** baoli has quit IRC22:56
jogofungi: thanks! hopefully it will land this time22:58
*** atiwari has quit IRC22:59
*** david-lyle has quit IRC22:59
*** lcheng has quit IRC22:59
*** dmsimard is now known as dmsimard_away22:59
*** dkehn_ is now known as dkehnm23:00
*** ZZelle_ has quit IRC23:02
*** juzuluag has quit IRC23:04
*** dangers is now known as dangers_away23:07
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: multinode_setup allow fixed ip  https://review.openstack.org/11484123:08
*** markmcclain has quit IRC23:08
*** packet has quit IRC23:10
*** jpich has joined #openstack-infra23:10
*** boden has joined #openstack-infra23:11
funginibalizer: if you want 119543 approved before the project split freeze in 45 minutes, it's missing jjb configuration23:15
*** boden has quit IRC23:16
*** pcm_ has quit IRC23:17
*** shashankhegde has joined #openstack-infra23:18
nibalizerfungi: no worries! freeze away23:18
nibalizerim at puppetconf anyways23:18
openstackgerritA change was merged to openstack-infra/config: Cleanup javascript build  https://review.openstack.org/11140723:19
funginibalizer: have fun with your puppetbeer23:19
anteayajeblair: also this https://github.com/anteaya/reorganized-project-config-0323:19
*** morgan_remote_ has quit IRC23:20
*** Sukhdev has joined #openstack-infra23:21
*** zns has quit IRC23:22
*** dims__ has quit IRC23:22
*** otherwiseguy has quit IRC23:23
jeblairanteaya: 03 doesn't seem to have had the mv commands applied23:27
openstackgerritA change was merged to openstack-infra/config: Quote hostname in ansible clean workspaces command  https://review.openstack.org/12285823:27
*** marcoemorais has quit IRC23:27
*** marcoemorais has joined #openstack-infra23:28
jeblairanteaya: project-config lgtm23:28
*** marcoemorais has quit IRC23:28
*** marcoemorais has joined #openstack-infra23:29
*** marcoemorais has quit IRC23:29
*** marcoemorais has joined #openstack-infra23:29
*** marun has quit IRC23:31
anteayayou are correct -03 doesn't23:31
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add flame project on StackForge  https://review.openstack.org/12311323:31
anteayaI hadn't committed the changes, d'oh23:32
anteayathey are stuck on local23:32
*** rmcall_ has quit IRC23:33
anteayajeblair: how about now? https://github.com/anteaya/reorganized-project-config-03/23:33
*** rmcall has joined #openstack-infra23:35
jeblairanteaya: looks good!23:36
*** bknudson has joined #openstack-infra23:36
*** bknudson has quit IRC23:36
*** dims has joined #openstack-infra23:37
stevemarfungi, anteaya thanks for all your help today :) my new job is actually running stuff! it's failing, but thats why we make things non-voting right!23:37
*** coolsvap has joined #openstack-infra23:37
*** bknudson has joined #openstack-infra23:39
anteayajeblair: thanks23:40
anteayastevemar: you are welcome, glad to have you around23:40
anteayaa big part of running stuff is watching it fail23:41
anteayayou are earning your stripes23:41
mwhudsonah uh23:41
jeblairnibalizer, clarkb: puppet question: https://review.openstack.org/#/c/122902/5/modules/openstack_project/manifests/nodepool_prod.pp23:42
mwhudsonthe instructions from https://github.com/openstack-infra/devstack-gate/blob/master/README.rst seem to be breaking DNS for me23:42
jeblairnibalizer, clarkb: the nodepool class invocation uses a value that comes from the project_config class instantiation; how do i make sure they are ordered correctly?23:42
clarkbmwhudson: oh yes, we install unbound and point at google dns23:43
clarkbmwhudson: if you have a firewall that will need to be turned off23:43
*** dannywilson has quit IRC23:43
fungimwhudson: our puppet manifest installs a caching dns resolver and sets it to firward lookups to google's 8.8.8.8 recursive resolver, so if you're depending on locak dns entries for things that may break and need overriding23:43
jeblairnibalizer, clarkb: actually, strike that; the problem is that the vcsrepo that project_config creates isn't in place; so i really need the nodepool class to require => the vcsrepo in project_config23:43
fungier, what clarkb said23:43
mwhudsonah yes, that's not going to work in this env23:43
*** dannywilson has joined #openstack-infra23:43
* mwhudson is tempted to remember that softlayer will rent you physical systems by the hour23:44
openstackgerritA change was merged to openstack-infra/config: add openstack-rating channel to eavesdrop  https://review.openstack.org/11705723:44
jrollmwhudson: so will rackspace :P23:44
fungimwhudson: we do that because our providers who donate virtualization services to us each have dns broken in different ways23:44
jroll(by the minute, even)23:44
mwhudsonfungi: :-)23:44
*** gyee has quit IRC23:45
mwhudsonjroll: really?  i thought so to but couldn't find it last time i went looking23:46
jrollmwhudson: rackspace.com/onmetal23:46
jrollmwhudson: tl;dr, nova boot --flavor onmetal-compute123:47
*** timrc is now known as timrc-afk23:47
jroll(or io1 or memory1, but compute is the cheapest)23:47
*** hdd has quit IRC23:47
fungijeblair: i take it the expectation is to split 120285 up into multiple changes once the corresponding governance change 123826 gets approved?23:47
*** jaypipes has quit IRC23:47
mwhudsonjroll: oh right, the pricing being listed per month confused me i think23:47
jrollmwhudson: yeah, marketing fail23:47
jrollit's by the minute, maybe second, dunno23:47
fungijeblair: since it's unlikely anyone's approving governance changes in the next 10 minutes23:48
*** dannywilson has quit IRC23:48
openstackgerritA change was merged to openstack-infra/config: Disable ironic job for stable/icehouse  https://review.openstack.org/12382223:48
mwhudson$550/month is ???/hour23:48
jeblairfungi: what would the multiple changes be?23:48
mwhudsonabout 75c23:48
fungijeblair: oh, good point. it just needs to be reproposed against the new repo23:48
openstackgerritA change was merged to openstack-infra/config: Enabling jobs for GBP project  https://review.openstack.org/12295623:50
stevemaranteaya, i'm also great at watching downloads and installs progress - something neat about progress bars :P23:50
jrollmwhudson: they are a bit overkill for devstack, though, I typically run it on an 8gb cloud server to match what infra does23:50
*** dannywilson has joined #openstack-infra23:50
mwhudsonclarkb: is it easy to disable this installing of unbound?23:50
mwhudsonvi config/modules/openstack_project/manifests/template.pp ?23:50
clarkbmwhudson: yeah I think asselin added it to our puppet. one sec23:51
anteayastevemar: yay, you are marked as easily enertained in my book23:51
jeblairnibalizer, clarkb: so i guess we can still use the require => metaparameter on a class?  so just add:     require => Vcsrepo['/etc/project-config'],23:51
anteayastevemar: you will fit in well around here23:51
clarkbjeblair: ya that should still be correct23:51
openstackgerritJeremy Stanley proposed a change to openstack-infra/config: Add experimental zaqar-rally job  https://review.openstack.org/12339623:51
jeblaircool, i'll turn that into a parameter so it's not hardcoded, then update && test23:51
clarkbmwhudson: https://git.openstack.org/cgit/openstack-infra/config/tree/modules/openstack_project/manifests/single_use_slave.pp#n19 needs to be set to false23:52
mwhudsonclarkb: ok23:52
jheskethHey guys, are we going into repo freeze in 5min?23:52
anteayajeblair fungi clarkb this is how I have config cleaned up after the filter branch: https://github.com/anteaya/cleaned-config23:52
nibalizerjeblair: sweet23:52
anteayajhesketh nibalizer ^^23:52
clarkbmwhudson: in your command as given in the readme add enable_unboud => false, after install_userse => false23:52
anteayatell me what I have missed23:52
fungijhesketh: yep23:53
mwhudsonclarkb: i guess doing that again won't mysteriously fix this instance :)23:53
clarkbmwhudson: probably not23:53
mwhudson(it didn't)23:54
*** dmsimard_away is now known as dmsimard23:54
anteayajhesketh: you could look at project-config, reorganized-project-config-03, and cleaned-config: https://github.com/anteaya?tab=repositories to ensure they look like they should23:54
anteayajhesketh: this etherpad is helpful: https://etherpad.openstack.org/p/Le2P9YegAx23:54
anteayaand the spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/config-repo-split.html23:55
* jhesketh has a meeting in a minute, but will do later23:55
anteayakk23:55
*** openstackstatus has quit IRC23:55
*** openstack has joined #openstack-infra23:56
-sendak.freenode.net- [freenode-info] channel flooding and no channel staff around to help? Please check with freenode support: http://freenode.net/faq.shtml#gettinghelp23:56
*** bknudson has quit IRC23:56
anteayayay23:56
*** asettle has quit IRC23:57
jeblair$config_dir = Vcsrepo['/etc/project-config']23:57
jeblair...23:57
jeblairrequire => $config_dir23:57
jeblairnibalizer, clarkb: ^ that is not a pattern that we have done before, but empirically it seems to work.  :)23:58
*** bknudson has joined #openstack-infra23:58
jeblairis that a bad idea?23:58
clarkbjeblair: I don't think so. did similar with iptables iirc23:58
jeblairoh ok cool23:58
clarkbjeblair: beacuse hte iptable service is different on fedora and ubuntu23:58
jeblairthen i think that will end up looking very nice23:59
openstackgerritA change was merged to openstack-infra/config: Add puppet-storyboard to storyboard  https://review.openstack.org/11843523:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!