Monday, 2017-04-03

openstackgerritMerged openstack-infra/system-config master: Rework intro
*** yamahata has joined #openstack-infra00:02
ianwpabelanger: while it's quiet do you want me to make the volumes for 437100?00:03
*** jkilpatr has quit IRC00:04
pabelangerianw: sure!00:04
*** masber has joined #openstack-infra00:05
ianwpabelanger: i assume "opensuse" (as opposed, to say, suse?)00:06
pabelangerianw: ya, that's what I have been using00:07
*** aeng has quit IRC00:34
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Add missing DEVSTACK_LOCAL_CONFIG+= for opensuse nodepool job
*** erlon has quit IRC00:45
*** aeng has joined #openstack-infra00:51
*** baoli has quit IRC00:52
*** s-shiono has joined #openstack-infra00:54
*** baoli has joined #openstack-infra00:58
ianwhmm POST_FAILURE on
*** yamahata has quit IRC01:03
*** s-shiono_ has joined #openstack-infra01:04
*** kien-ha has quit IRC01:05
*** jamielennox|away is now known as jamielennox01:06
*** s-shiono has quit IRC01:07
openstackgerritMerged openstack-infra/system-config master: Mirror openSUSE Leap 42.2 repositories
*** gcb has joined #openstack-infra01:14
*** aeng has quit IRC01:16
openstackgerritIan Wienand proposed openstack-infra/system-config master: Mirror rdo centos7 repo to AFS
*** baoli has quit IRC01:19
*** Julien-zte has joined #openstack-infra01:27
*** baoli has joined #openstack-infra01:29
*** aeng has joined #openstack-infra01:30
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs
*** tuanluong has joined #openstack-infra01:35
*** thorst has joined #openstack-infra01:37
*** thorst has quit IRC01:42
*** baoli has quit IRC01:43
*** namnh has joined #openstack-infra01:48
*** rwsu has quit IRC01:54
*** fandi has joined #openstack-infra01:57
*** rcernin has joined #openstack-infra01:58
*** Guest72255 has joined #openstack-infra01:59
*** rwsu has joined #openstack-infra02:00
*** Hal has quit IRC02:11
*** Guest72255 has quit IRC02:11
*** Hal has joined #openstack-infra02:13
*** lucas-afk has quit IRC02:23
*** lucasagomes has joined #openstack-infra02:25
*** Julien-zte has quit IRC02:26
*** baoli has joined #openstack-infra02:28
*** yamamoto has joined #openstack-infra02:29
*** Hal has quit IRC02:29
*** baoli_ has joined #openstack-infra02:30
*** baoli has quit IRC02:33
*** thorst has joined #openstack-infra02:38
*** amotoki has joined #openstack-infra02:39
*** gouthamr has quit IRC02:41
*** rcernin has quit IRC02:45
*** cody-somerville has joined #openstack-infra02:46
*** Hal has joined #openstack-infra02:48
*** csomerville has quit IRC02:49
*** baoli_ has quit IRC02:50
*** baoli has joined #openstack-infra02:51
*** dave-mccowan has joined #openstack-infra02:55
*** thorst has quit IRC02:57
*** dave-mcc_ has quit IRC02:57
*** sekelso has joined #openstack-infra02:58
*** john-davidge has joined #openstack-infra02:58
*** amotoki has quit IRC03:00
*** ramishra has joined #openstack-infra03:00
*** john-davidge has quit IRC03:03
*** dave-mccowan has quit IRC03:05
*** namnh has quit IRC03:06
*** tjones has joined #openstack-infra03:08
*** tjones has quit IRC03:08
*** baoli has quit IRC03:16
*** sree has joined #openstack-infra03:20
*** amotoki has joined #openstack-infra03:20
*** mriedem has quit IRC03:21
*** vikrant has joined #openstack-infra03:23
*** sree has quit IRC03:28
*** sree has joined #openstack-infra03:28
*** amotoki has quit IRC03:29
*** aeng has quit IRC03:31
*** sree has quit IRC03:33
*** sekelso has quit IRC03:35
*** yamamoto has quit IRC03:37
*** sbezverk has quit IRC03:38
*** amotoki has joined #openstack-infra03:38
*** aeng has joined #openstack-infra03:45
openstackgerritIan Wienand proposed openstack-infra/puppet-planet master: Rotate planet log file
openstackgerritIan Wienand proposed openstack-infra/puppet-planet master: Up cron to 15 minute runs
*** thorst has joined #openstack-infra03:54
*** dharinic has joined #openstack-infra03:54
*** aeng has quit IRC03:54
openstackgerritHironori Shiina proposed openstack/diskimage-builder master: Fix code-block in README of rhel7 element
*** thorst has quit IRC03:58
*** Dinesh_Bhor has joined #openstack-infra04:01
*** amotoki has quit IRC04:01
*** kaisers has joined #openstack-infra04:04
*** aeng has joined #openstack-infra04:07
*** amotoki has joined #openstack-infra04:08
*** yamamoto has joined #openstack-infra04:16
*** persia has quit IRC04:17
*** psachin has joined #openstack-infra04:26
*** fandi__ has joined #openstack-infra04:41
*** dhajare has joined #openstack-infra04:43
*** fandi has quit IRC04:45
openstackgerritIan Wienand proposed openstack-infra/system-config master: openSUSE mirror : fix pathes
openstackgerritIan Wienand proposed openstack-infra/system-config master: openSUSE mirror : fix paths
*** kaisers has quit IRC04:49
*** cody-somerville has quit IRC04:51
*** thorst has joined #openstack-infra04:54
*** rcernin has joined #openstack-infra04:56
*** thorst has quit IRC04:59
*** kaisers1 has joined #openstack-infra05:04
*** flepied has quit IRC05:06
*** knangia has joined #openstack-infra05:07
*** bexelbie has quit IRC05:09
*** bexelbie has joined #openstack-infra05:10
*** hrubi has quit IRC05:10
*** pgadiya has joined #openstack-infra05:11
*** dharinic has quit IRC05:12
ianw#success migrated to xenial and running on vexxhost05:12
openstackstatusianw: Added success to Success page05:12
*** kaisers1 has quit IRC05:15
*** hrubi has joined #openstack-infra05:15
openstackgerritIan Wienand proposed openstack-infra/system-config master: Update to mention planet01
*** claudiub has joined #openstack-infra05:17
*** rcernin has quit IRC05:18
openstackgerritKota Tsuyuzaki proposed openstack-infra/project-config master: Storlets: Use python-jobs instead of parts indivisually
*** ihrachys has joined #openstack-infra05:21
*** asselin has joined #openstack-infra05:21
*** ihrachys has quit IRC05:22
*** ihrachys has joined #openstack-infra05:22
*** asselin_ has quit IRC05:23
*** dimtruck is now known as zz_dimtruck05:24
*** zz_dimtruck is now known as dimtruck05:24
*** prateek has joined #openstack-infra05:29
*** rcernin has joined #openstack-infra05:31
*** kaisers has joined #openstack-infra05:32
*** dimtruck is now known as zz_dimtruck05:33
*** IRCFrEAK has joined #openstack-infra05:36
*** IRCFrEAK has left #openstack-infra05:37
*** rbuzatu has quit IRC05:39
*** rbuzatu has joined #openstack-infra05:39
*** rbuzatu has quit IRC05:39
*** rbuzatu has joined #openstack-infra05:39
*** kaisers has quit IRC05:41
*** IRCFrEAK has joined #openstack-infra05:43
*** IRCFrEAK has left #openstack-infra05:44
*** e0ne has joined #openstack-infra05:45
openstackgerritLenny Verkhovsky proposed openstack-infra/puppet-openstackci master: added Single Use Slave plugin
openstackgerritTobias Henkel proposed openstack-infra/zuul feature/zuulv3: Change mutex to counting semaphore
*** e0ne has quit IRC05:52
*** adriant has quit IRC05:55
*** thorst has joined #openstack-infra05:55
*** yamahata has joined #openstack-infra05:56
*** kaisers has joined #openstack-infra05:56
*** rbuzatu_ has joined #openstack-infra05:57
*** florianf has joined #openstack-infra05:57
*** thorst has quit IRC05:59
*** rbuzatu has quit IRC06:00
openstackgerritMerged openstack-infra/system-config master: Update to mention planet01
openstackgerritIan Wienand proposed openstack-infra/system-config master: Remove
*** rbuzatu_ has quit IRC06:01
*** kaisers has quit IRC06:03
*** oomichi has quit IRC06:03
*** oomichi has joined #openstack-infra06:04
*** pgadiya has quit IRC06:04
openstackgerritIan Wienand proposed openstack-infra/system-config master: Thats what I'm trying to tell you, kid. It ain't there. It's been totally blown away.
openstackgerritHong Hui Xiao proposed openstack-infra/project-config master: [WIP] Add bgp to dragonflow fullstack job
*** makowals_ has quit IRC06:05
*** sree has joined #openstack-infra06:07
*** gcb has quit IRC06:07
*** yuanying has quit IRC06:07
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Link to Infra Manual for CI problems
*** AJaeger has quit IRC06:10
openstackgerritAndreas Jaeger proposed openstack-infra/infra-manual master: Explain OpenStack CI Status
*** yuanying_ has joined #openstack-infra06:13
*** AJaeger has joined #openstack-infra06:14
*** mkoderer has joined #openstack-infra06:15
AJaegerianw, could you review , please? This moves packages around a bit for dib.06:15
*** kaisers1 has joined #openstack-infra06:19
Dinesh_BhorAJaeger: Hi, do you have some time?06:23
*** e0ne has joined #openstack-infra06:23
AJaegerDinesh_Bhor: what's up?06:23
*** isaacb has joined #openstack-infra06:24
Dinesh_BhorAJaeger: About this: After this patch and it's child got merged I can see patches submitted by bot job to masakari, python-masakariclient.06:24
Dinesh_Bhor AJaeger: but I don't see any patch for masakari-monitors06:24
*** e0ne has quit IRC06:24
Dinesh_BhorAJaeger: could you please help me to debug this?06:25
*** cshastri has joined #openstack-infra06:25
AJaegerDinesh_Bhor: you have to look at the requirements post job.06:25
*** rbuzatu has joined #openstack-infra06:26
AJaegerLast change merged as hash of fffff013eeee72038a29c3c3d4c4ec1b31b7b324, so using the formula  logs.o.o/first two chars of hash/full hash/ gives us
openstackgerritSreeram Vancheeswaran proposed openstack-infra/project-config master: Enable functional test jobs for nova-dpm
AJaegerDinesh_Bhor: and that leads to
AJaegerDinesh_Bhor: your error then is
*** vikrant has quit IRC06:28
*** nadya has joined #openstack-infra06:28
*** dhajare has quit IRC06:28
*** vikrant has joined #openstack-infra06:28
AJaegerDinesh_Bhor: there's an #openstack-requirements channel if you have further questions06:28
Dinesh_BhorAJaeger: ok, thanks a lot for the info06:29
Dinesh_BhorAJaeger: it explains a lot06:29
*** oomichi has quit IRC06:29
*** oomichi has joined #openstack-infra06:30
*** makowals has joined #openstack-infra06:30
*** sputnik13 has quit IRC06:32
*** kaisers1 has quit IRC06:33
*** nadya has quit IRC06:33
*** sputnik13 has joined #openstack-infra06:34
*** tesseract has joined #openstack-infra06:34
*** ralonsoh has joined #openstack-infra06:43
*** dhajare has joined #openstack-infra06:44
*** chem_gon` is now known as chem06:45
*** chem has quit IRC06:45
*** chem has joined #openstack-infra06:45
*** jcoufal has joined #openstack-infra06:46
*** flepied has joined #openstack-infra06:49
*** andreas_s has joined #openstack-infra06:50
*** kaisers1 has joined #openstack-infra06:50
*** martinkopec has joined #openstack-infra06:51
*** thorst has joined #openstack-infra06:56
*** pcaruana has joined #openstack-infra06:57
*** jaosorior has joined #openstack-infra06:57
*** markus_z has joined #openstack-infra06:58
*** yatin_away is now known as yatinkarel06:58
*** bhavik1 has joined #openstack-infra07:00
*** aeng has quit IRC07:00
*** ccamacho has joined #openstack-infra07:00
*** thorst has quit IRC07:00
*** zz_dimtruck is now known as dimtruck07:01
*** kaisers1 has quit IRC07:02
*** bhavik1 has quit IRC07:02
*** neiljerram has joined #openstack-infra07:03
*** jcoufal has quit IRC07:04
*** shardy has joined #openstack-infra07:07
*** dimtruck is now known as zz_dimtruck07:11
*** matrohon has joined #openstack-infra07:11
*** kaisers1 has joined #openstack-infra07:17
*** markmcd has quit IRC07:18
*** neiljerram has quit IRC07:19
*** amoralej|off is now known as amoralej07:20
*** knangia has quit IRC07:31
*** hashar has joined #openstack-infra07:31
*** kaisers1 has quit IRC07:33
*** hashar has quit IRC07:36
*** stakeda has joined #openstack-infra07:37
*** amotoki_ has joined #openstack-infra07:41
*** jascott1_ has quit IRC07:42
*** dizquierdo has joined #openstack-infra07:42
*** hashar has joined #openstack-infra07:42
*** dhajare has quit IRC07:42
*** amotoki has quit IRC07:44
openstackgerritDirk Mueller proposed openstack-infra/project-config master: Start building opensuse based nodepool images
*** kaisers1 has joined #openstack-infra07:49
openstackgerritDirk Mueller proposed openstack-infra/nodepool master: Remove link to modindex
ianwdirk: does current layout work?07:50
*** sshnaidm is now known as sshnaidm|afk07:50
dirkianw: yep, current layout is also fine07:51
*** jascott1_ has joined #openstack-infra07:52
ianwdirk: ok, i'm going to merge the changes to do that and drop my lock when they're in.  if it needs to change, just have to run a manual sync first i'd say07:53
openstackgerritMerged openstack-infra/project-config master: [magnum] Add ocata to gerritbot
openstackgerritMerged openstack-infra/project-config master: [magnum] Fix dib elements path
*** thorst has joined #openstack-infra07:56
openstackgerritMerged openstack-infra/project-config master: Remove gate-novaclient-dsvm-functional-mitaka-ubuntu-trusty-nv
openstackgerritMerged openstack-infra/project-config master: Add missing DEVSTACK_LOCAL_CONFIG+= for opensuse nodepool job
openstackgerritMerged openstack-infra/system-config master: openSUSE mirror : fix paths
*** jascott1_ has quit IRC07:57
openstackgerritMerged openstack-infra/project-config master: Tweak hooks for golang dsvm job
openstackgerritMerged openstack-infra/project-config master: Publish LOCI tarball to tarballs.o.o
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** jpich has joined #openstack-infra08:00
*** flepied has quit IRC08:01
openstackgerritMerged openstack/diskimage-builder master: Fix code-block in README of rhel7 element
*** zz_dimtruck is now known as dimtruck08:02
*** bhavik1 has joined #openstack-infra08:02
*** openstackgerrit has quit IRC08:03
*** dhajare has joined #openstack-infra08:03
*** openstackgerrit has joined #openstack-infra08:05
openstackgerritMerged openstack-infra/project-config master: Enable multinode topology for slow Patrole tests.
openstackgerritMerged openstack-infra/project-config master: Add experimental murano functional test gate.
openstackgerritMerged openstack-infra/project-config master: Add separate ACL/group for castellan
openstackgerritMerged openstack-infra/project-config master: Skip pre-caching /deb- repositories in nodepool images
openstackgerritMerged openstack-infra/project-config master: Uninstall postfix on nodes if it's there
openstackgerritMerged openstack-infra/project-config master: Add a new project sushy-tools
openstackgerritMerged openstack-infra/project-config master: [Manila] Add 2 jobs for manila-image-elements project
openstackgerritMerged openstack-infra/project-config master: Run check-requirements on remaining powervm projects
*** jordanP has joined #openstack-infra08:06
openstackgerritMerged openstack-infra/project-config master: Update ARA's Ansible version with releases for 2.1.x, 2.2.x and 2.3.x
*** e0ne has joined #openstack-infra08:08
openstackgerritMerged openstack-infra/project-config master: Add install-guide-jobs for murano installation guide.
openstackgerritMerged openstack-infra/project-config master: add coverage-jobs to congress
openstackgerritMerged openstack-infra/project-config master: Remove tripleo-test-cloud-rh2 from nodepool
*** shardy has quit IRC08:09
*** rossella_s has joined #openstack-infra08:09
openstackgerritMerged openstack-infra/project-config master: Made ironic-dsvm-standalone voting
openstackgerritMerged openstack-infra/project-config master: Enable Octavia gate with Centos amphora
openstackgerritMerged openstack-infra/project-config master: Enable LXD test on Pike+
openstackgerritMerged openstack-infra/project-config master: Add ocata and pike UCA repos
*** dimtruck is now known as zz_dimtruck08:12
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Allow installing proliant tools from an OS package
openstackgerritColleen Murphy proposed openstack-infra/project-config master: Manage NTP on zuul workers in DIB
openstackgerritColleen Murphy proposed openstack-infra/project-config master: Install more packages with DIB
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Allow installing proliant tools from an OS package
*** alexchadin has joined #openstack-infra08:15
*** thorst has quit IRC08:16
*** dingyichen has quit IRC08:18
openstackgerritMerged openstack-infra/project-config master: Retire os-cloud-config
*** jtomasek has joined #openstack-infra08:22
*** nmathew has joined #openstack-infra08:24
*** alexchadin has quit IRC08:29
ianwdirk / pabelanger : ok ... whatever the opensuse mirror is doing now, it's doing on it's own.  so please keep an eye on it for a while to make sure it's working :)08:30
*** arxcruz|pto is now known as arxcruz08:31
*** shardy has joined #openstack-infra08:31
openstackgerritMasahito Muroi proposed openstack-infra/project-config master: Update Blazar's gate jobs
*** s-shiono_ has quit IRC08:33
*** openstackgerrit has quit IRC08:33
*** s-shiono has joined #openstack-infra08:33
*** s-shiono has quit IRC08:33
*** mjblack has quit IRC08:33
*** openstackgerrit has joined #openstack-infra08:36
openstackgerritFabien Boucher proposed openstack-infra/storyboard-webclient master: Change maxlength to 100 for projects and project_groups name
openstackgerritDebayan Ray proposed openstack-infra/project-config master: Add coverage job to sushy
*** rbuzatu has quit IRC08:37
*** mjblack has joined #openstack-infra08:38
*** bhavik1 has quit IRC08:39
*** derekh has joined #openstack-infra08:42
*** shardy has quit IRC08:46
*** yamahata has quit IRC08:48
openstackgerritFabien Boucher proposed openstack-infra/storyboard master: Increase project and project group name length limitation
*** shardy has joined #openstack-infra08:48
*** rbuzatu has joined #openstack-infra08:50
*** john-davidge has joined #openstack-infra08:51
*** ssbarnea has joined #openstack-infra08:51
*** felipemonteiro_ has joined #openstack-infra08:52
*** hashar_ has joined #openstack-infra08:52
*** mtreinish has quit IRC08:53
*** neiljerram has joined #openstack-infra08:54
*** hashar has quit IRC08:55
*** felipemonteiro has quit IRC08:55
openstackgerritFabien Boucher proposed openstack-infra/storyboard master: Decrease projects and project_groups name MIN length limitation
*** hashar_ is now known as hashar08:56
*** mtreinish has joined #openstack-infra08:57
openstackgerritJuan Antonio Osorio Robles proposed openstack-infra/tripleo-ci master: DO NOT MERGE: Testing TLS everywhere
*** dizquierdo has quit IRC09:02
*** zz_dimtruck is now known as dimtruck09:02
*** neiljerram has quit IRC09:04
odyssey4mehmm, I see that is still returning a 404 page - is that a known issue?09:06
openstackgerritMasayuki Igawa proposed openstack/boartty master: Fix inconsistency name
fricklerodyssey4me: seems to be working fine for me, maybe some caching issue?09:10
*** alexchadin has joined #openstack-infra09:11
*** dimtruck is now known as zz_dimtruck09:12
*** thorst has joined #openstack-infra09:13
openstackgerritNoam Angel proposed openstack/diskimage-builder master: Add option to update all packages early on
*** sshnaidm|afk is now known as sshnaidm09:14
openstackgerritAlexander Evseev proposed openstack-infra/devstack-gate master: Add more varibales to the script
*** thorst has quit IRC09:17
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: testing tempest for oooq
*** electrofelix has joined #openstack-infra09:21
*** szaher has quit IRC09:25
*** amotoki_ has quit IRC09:25
*** sambetts|afk is now known as sambetts09:25
*** cuongnv has joined #openstack-infra09:25
openstackgerritRoman Podoliaka proposed openstack-infra/project-config master: osc-placement: experimental job for running of functional tests
*** tosky has joined #openstack-infra09:27
cuongnvhi guys, I have a quesion related to jenkins jobs, how can I know which OS version jenkins is running for my test?09:27
cuongnvthis is output of my test
cuongnvI can guess it's running xenial, but how can I know exact which version or where jenkins pulls ubuntu from?09:28
odyssey4mecuongnv you'd have to add some sort of patch which shows whatever you're looking for09:29
AJaegercuongnv: our tests use a local mirror of the ubuntu repos09:29
odyssey4mebut the repo used is an up to date ubuntu mirror which has been mirrored in each provider09:30
*** szaher has joined #openstack-infra09:30
openstackgerritNoam Angel proposed openstack/diskimage-builder master: Add option to update all packages early on
odyssey4methe OS images are built regularly - I think perhaps weekly, but not sure09:30
AJaegercuongnv: the log files even point to the mirror so you can check them.09:30
AJaegerodyssey4me: daily builds09:30
AJaegercuongnv: what is your real problem? Why do you ask?09:31
odyssey4meinfra-root I see that is still returning a 404 page - is that a known issue?09:31
cuongnvAJaeger, I'm having a problem related to a shared library with the OS, just to make sure my local env is the same with OS image which jenkins running09:32
AJaegerodyssey4me: I have the same experience as frickler above: It just works for me.09:32
*** szaher has quit IRC09:33
*** prateek has quit IRC09:33
odyssey4meoh really? hmm, I must have something stuck in my browser cache then09:33
odyssey4meI appear to be missing messages - I didn't see frickler's response. :/09:33
cuongnvAJaeger, do you mean this mirror?
AJaegercuongnv: yes09:34
AJaegerodyssey4me: frickler answered 4 minutes after your first question09:35
openstackgerritAnusha Ramineni proposed openstack-infra/project-config master: Enable ironic service in congress gate job
odyssey4meah yes, after clearing my cache it's there - thanks frickler & AJaeger09:37
openstackgerritNoam Angel proposed openstack/diskimage-builder master: Add option to update all packages early on
*** szaher has joined #openstack-infra09:39
AJaegerglad to hear, odyssey4me09:39
cuongnvAJaeger, odyssey4me ok thank you, it seems my vagrant box is quite same with the one used by jenkins09:41
*** prateek has joined #openstack-infra09:47
*** prateek has quit IRC09:47
*** prateek has joined #openstack-infra09:47
*** alexchadin has quit IRC09:51
*** stakeda has quit IRC09:52
chemhi, I've got a strange message from Jenkins in and : "This change depends on a change that failed to merge.", how can I debug that ?09:52
chemcurrently I blindly recheck it09:53
jordanPchem, check that the patch(s) your patch depends on are mergeable09:53
*** jascott1_ has joined #openstack-infra09:53
chemjordanP: well .. but it is
*** alexchadin has joined #openstack-infra09:54
chemjordanP: hi, by the way.  Thanks for replying :)09:55
jordanPhi :), yeah that's weird09:56
*** prateek has quit IRC09:56
*** prateek_ has joined #openstack-infra09:56
jordanPwhat I usually do is isn"t of rechecking, I'd click the "rebase" button09:56
AJaegerjordanP: rebase will not help, recheck was correct09:56
jordanPrebase will recheck, so what's wrong ?09:57
AJaegerchem: this happens when you have a depends-on on another change that has not finished the jobs.09:57
AJaegerjordanP: you might be already completely rebased09:58
chemAJaeger: ah, so it needs jenkins +1 before being able to run, right ?09:58
AJaegerchem: I think so - +1 on the job marked as depending on. But don't remember those details09:59
*** jascott1_ has quit IRC09:59
chemAJaeger: ack, oki, kinda make sense, but the error message could be better :)09:59
chemAJaeger: by the way, in the last commit I only update the commit message and it blanks the Jenkins+1, retriggering a "useless" test, can we fix that in some way ?10:00
*** prateek has joined #openstack-infra10:00
*** szaher has quit IRC10:01
*** prateek_ has quit IRC10:03
*** alexchad_ has joined #openstack-infra10:03
*** zz_dimtruck is now known as dimtruck10:03
*** szaher has joined #openstack-infra10:04
openstackgerritKota Tsuyuzaki proposed openstack-infra/project-config master: Storlets: Use python-jobs instead of parts indivisually
*** kuromagi^ has quit IRC10:05
*** rcernin has quit IRC10:05
*** fandi__ has quit IRC10:05
*** alexchadin has quit IRC10:06
*** prateek has quit IRC10:09
*** szaher has quit IRC10:10
*** dizquierdo has joined #openstack-infra10:11
*** szaher has joined #openstack-infra10:11
*** dimtruck is now known as zz_dimtruck10:13
*** prateek has joined #openstack-infra10:15
*** kuromagi has joined #openstack-infra10:16
chemAJaeger: sorry to ping you over again ... just wondering if you could point me to the right direction to avoid triggering a recheck when only the commit message is updated ?10:19
openstackgerritDebayan Ray proposed openstack-infra/project-config master: Add coverage job to sushy
*** sdague has joined #openstack-infra10:23
*** nicolasbock has joined #openstack-infra10:23
*** rcernin has joined #openstack-infra10:24
*** cuongnv has quit IRC10:26
odyssey4mechem you can't10:27
odyssey4mea commit message change is a patch change, so it will trigger the jobs10:27
chemodyssey4me: ah, hum ... that's too bad.  If only we could detect only depends-on change in the commit message then we could avoid useless jobs.10:28
*** wolverineav has joined #openstack-infra10:31
*** rbuzatu has quit IRC10:31
*** wolverineav has quit IRC10:31
*** rbuzatu has joined #openstack-infra10:31
*** wolverineav has joined #openstack-infra10:32
AJaegerchem: and you might add/remove a depends-on and that changes the environment your job runs in. So, we have to test again.10:33
*** psachin has quit IRC10:34
chemAJaeger: ^ yeap, aggreed, but see my above comment to odyssey4me, it's only for those one that a re-test is necessary AFAICT.10:35
*** pbourke has quit IRC10:35
*** rbuzatu has quit IRC10:35
odyssey4mechem any change to a patch, no matter how minor, could change the validity of the patch10:36
openstackgerritDebayan Ray proposed openstack-infra/project-config master: Add coverage job to proliantutils
odyssey4methe tests are essential10:36
chemodyssey4me: would be if anything was testing the validity of the commit message, so I guess that would make sense to have it configurable.10:37
*** pbourke has joined #openstack-infra10:37
*** sree has quit IRC10:41
*** sree has joined #openstack-infra10:42
AJaegerchem: a commit change is a new revision in gerrit and that's what we test.10:45
AJaegerchem: Yes, you could not run tests but then it might become complicated. This is not as trivial as it sounds.10:46
*** john-davidge has quit IRC10:46
chemAJaeger: odyssey4me ack, it's oki, I just wanted to make sure that I was not missing something here.  Thanks for the discussion.10:48
*** psachin has joined #openstack-infra10:51
*** sree has quit IRC10:57
*** alexchad_ has quit IRC10:58
*** sree has joined #openstack-infra10:58
*** alexchadin has joined #openstack-infra10:58
*** alexchad_ has joined #openstack-infra10:58
*** alexchadin has quit IRC10:58
*** jkilpatr has joined #openstack-infra11:01
*** sree has quit IRC11:02
*** hashar is now known as hasharLunch11:04
*** zz_dimtruck is now known as dimtruck11:04
*** martinkopec has quit IRC11:05
*** arxcruz has quit IRC11:07
*** asselin has quit IRC11:07
*** asselin has joined #openstack-infra11:07
*** adarazs is now known as adarazs_lunch11:10
*** arxcruz has joined #openstack-infra11:12
*** mat128 has joined #openstack-infra11:12
*** mat128 has quit IRC11:13
*** dimtruck is now known as zz_dimtruck11:14
*** mat128 has joined #openstack-infra11:15
*** rbuzatu has joined #openstack-infra11:15
*** neiljerram has joined #openstack-infra11:20
*** lucasagomes is now known as lucas-hungry11:20
*** john-davidge has joined #openstack-infra11:22
*** ccamacho is now known as ccamacho|lunch11:24
*** rbuzatu has quit IRC11:24
*** rbuzatu has joined #openstack-infra11:25
*** john-davidge has quit IRC11:27
*** shaiton_ has joined #openstack-infra11:31
shaiton_Hi there, I would lik11:32
shaiton_Hi there, I would like to signe the CLA on gerrit but I get a server Error "Cannot store contact information"11:33
shaiton_any info on your side?11:33
shaiton_I use a lot git-review and would like to give back some patches11:33
*** thorst has joined #openstack-infra11:34
AJaegershaiton_: This needs your gerrit preferred e-mail address to match a primary e-mail address for a foundation individual member account.11:34
AJaegershaiton_: If you already followed the instructions at and still get that, see for additional troubleshooting tips.11:34
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config master: gnocchi: remove unexisting jobs
shaiton_how thank you AJaeger I missed the "fondation member" part..11:36
*** thorst has quit IRC11:36
AJaegerAh... Just register there, please. Thanks for contributing!11:36
*** thingee has quit IRC11:36
*** ttx has quit IRC11:38
*** ttx has joined #openstack-infra11:38
*** shardy is now known as shardy_lunch11:38
*** dhellmann has quit IRC11:42
*** dhellmann has joined #openstack-infra11:42
*** tuanluong has quit IRC11:42
*** adam_g has quit IRC11:45
*** asselin_ has joined #openstack-infra11:46
*** adam_g has joined #openstack-infra11:47
*** neiljerram has quit IRC11:47
*** asselin__ has joined #openstack-infra11:48
*** thingee has joined #openstack-infra11:49
*** asselin has quit IRC11:49
*** jgr is now known as jgrassler11:49
*** asselin_ has quit IRC11:51
*** hasharLunch is now known as hashar11:52
*** jascott1_ has joined #openstack-infra11:56
*** ccamacho|lunch is now known as ccamacho11:56
*** psachin has quit IRC11:58
*** dprince has joined #openstack-infra12:00
*** thorst has joined #openstack-infra12:00
*** jascott1_ has quit IRC12:01
*** signed8bit has joined #openstack-infra12:03
*** hrubi has quit IRC12:04
*** signed8bit has quit IRC12:04
*** signed8bit has joined #openstack-infra12:04
*** zz_dimtruck is now known as dimtruck12:05
*** jaypipes has joined #openstack-infra12:05
*** trown|outtypewww is now known as trown12:05
*** lucas-hungry is now known as lucasagomes12:07
*** HeOS has joined #openstack-infra12:07
openstackgerritAndy McCrae proposed openstack-infra/project-config master: Promote Upgrade jobs to voting for OpenStack-Ansible
*** adarazs_lunch is now known as adarazs12:08
shaiton_it's all good now AJaeger thanks.12:10
*** prateek has quit IRC12:10
*** psachin has joined #openstack-infra12:11
*** ricardoas has left #openstack-infra12:12
*** rhallisey has joined #openstack-infra12:13
*** ldnunes has joined #openstack-infra12:13
*** mat128 has quit IRC12:14
*** dimtruck is now known as zz_dimtruck12:15
*** rbuzatu has quit IRC12:16
*** jcoufal has joined #openstack-infra12:19
*** gouthamr has joined #openstack-infra12:21
*** ssbarnea has quit IRC12:23
jkilpatranyone else seeing this?12:27
*** thingee has quit IRC12:27
jkilpatr2017-04-03 11:51:53.677984 | error: Download error for timed out12:27
jkilpatrdid it twice in a row (recheck)12:27
openstackgerritPavlo Shchelokovskyy proposed openstack-infra/project-config master: Change mogan tempest job to use ipmitool driver
*** ssbarnea has joined #openstack-infra12:29
pabelangerjkilpatr: looks like an issue with pypi mirrors12:31
pabelangerI can take a quick look12:31
pabelangernetwork traffic looks odd12:32
*** rlandy has joined #openstack-infra12:32
pabelangerlook a while to load for me too12:33
*** adam_g has quit IRC12:35
pabelangerload average: 0.49, 12.34, 27.7812:35
pabelangerya, server was underload12:35
pabelangerbut coming back down12:35
pabelangerI wonder if apache2 was waiting on afs again12:35
*** jaosorior is now known as jaosorior_brb12:36
pabelangerApr  3 12:28:14 mirror kernel: [15062560.448790] afs: Waiting for busy volume 536870932 () in cell openstack.org12:36
pabelangerjkilpatr: recheck again12:36
*** adam_g has joined #openstack-infra12:36
jkilpatrpabelanger, this was on a gate, so unworkflow or reworkflow or does "recheck ci" work?12:37
*** aysyd has joined #openstack-infra12:37
jkilpatrand reworkflow*12:37
* dims peeks12:37
pabelangerjkilpatr: recheck should be enough12:37
jkilpatrpabelanger, cool, thanks for the help!12:37
dimspabelanger : same problem on rax too?12:38
dimspabelanger :
pabelangerdims: ya, I think we are ddos'ng afs12:38
dimsack :)12:38
pabelangerthis happened last week too12:38
*** mdrabe has joined #openstack-infra12:39
*** neiljerram has joined #openstack-infra12:39
dimsack thanks for confirming pabelanger12:39
*** signed8bit is now known as signed8bit_Zzz12:40
openstackgerritMonty Taylor proposed openstack-infra/system-config master: Add redirect for shade docs
*** tonytan4ever has quit IRC12:41
*** rbuzatu has joined #openstack-infra12:42
*** Julien-zte has joined #openstack-infra12:42
*** thingee has joined #openstack-infra12:44
AJaegermordred: we have all other redirects in openstack-manuals, what about adding your changes there as well? ^12:45
pabelangerinfra-root: I have an appointment with customs / immigration this morning to complete my NEXUS pass. As such, I'll be offline for most of the morning.12:45
*** shardy_lunch is now known as shardy12:45
mordredAJaeger: oh - neat - I will go look12:47
*** rbuzatu has quit IRC12:50
mordredAJaeger: remote: Add redirect for shade docs         <-- how's that look?12:51
AJaegermordred: reviewed and suggested a better way12:52
*** mjblack has quit IRC12:52
*** vikrant has quit IRC12:53
* mordred learns this morning12:53
*** signed8bit_Zzz is now known as signed8bit12:53
mordredAJaeger: hopefully that one looks better now12:54
*** amoralej is now known as amoralej|lunch12:54
*** mjblack has joined #openstack-infra12:55
*** smarcet has joined #openstack-infra12:56
*** smarcet has quit IRC12:56
*** smarcet has joined #openstack-infra12:56
*** smarcet has quit IRC12:57
*** smarcet has joined #openstack-infra12:57
*** kgiusti has joined #openstack-infra12:57
*** smarcet has quit IRC12:57
openstackgerritMonty Taylor proposed openstack-infra/project-config master: Re-add magnumclient to shade's magnum job
*** smarcet has joined #openstack-infra12:58
*** smarcet has quit IRC12:58
AJaegermordred: add some '()'s12:58
*** smarcet has joined #openstack-infra12:58
*** smarcet has quit IRC12:59
*** smarcet has joined #openstack-infra12:59
*** smarcet has quit IRC12:59
mordredAJaeger: wow. I really am on first coffee aren't I?12:59
*** smarcet_ has joined #openstack-infra12:59
*** baoli has joined #openstack-infra13:00
*** askb has quit IRC13:00
*** smarcet_ has quit IRC13:00
*** smarcet_ has joined #openstack-infra13:00
dhellmannhi, folks! I wanted to check in and make sure things are running smoothly, as far as we know, before I start approving releases. Are we all clear this morning?13:00
*** smarcet_ has quit IRC13:00
*** smarcet_ has joined #openstack-infra13:01
*** florianf has quit IRC13:02
*** tosky has quit IRC13:02
*** tosky has joined #openstack-infra13:02
*** smarcet_ has quit IRC13:02
AJaegermordred: no comments ;)13:03
*** smarcet_ has joined #openstack-infra13:03
*** smarcet_ has quit IRC13:03
*** smarcet_ has joined #openstack-infra13:03
*** zz_dimtruck is now known as dimtruck13:06
*** john-davidge has joined #openstack-infra13:06
*** smarcet_ has quit IRC13:08
*** rfolco has joined #openstack-infra13:09
*** john-davidge has quit IRC13:11
mordreddhellmann: there was a pypi mirror issue a little earlier, but I tihnk it's passed ... pabelanger ?13:11
*** florianf has joined #openstack-infra13:12
*** sekelso has joined #openstack-infra13:13
*** pradk has quit IRC13:14
*** esberglu has joined #openstack-infra13:15
*** dimtruck is now known as zz_dimtruck13:15
*** skelso has joined #openstack-infra13:16
pabelangermordred: dhellmann: yes, we had a busy AFS volume that was causing issues for serving things over apache13:16
pabelangerlooks to be better now13:16
pabelangerhowever, this is about the 2nd / 3rd time it has happened recently13:16
dhellmannpabelanger : ok, it sounds like it's safe to release, but maybe go slowly so we don't end up with a bunch of failures if we do start having issues13:17
*** sekelso has quit IRC13:17
pabelangerdhellmann: seems like a good plan13:18
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Support built packages to be installed in image
*** srobert has joined #openstack-infra13:18
*** mat128 has joined #openstack-infra13:19
*** rook|tower is now known as rook13:19
*** thingee has quit IRC13:23
*** dhellmann has quit IRC13:24
pabelangermordred: we could try bumping out timeout in pip.conf to see if that helps. I am also not sure if we can control the retry attempts either13:24
electrofelixzaro: wonder if you might be able to give a review, I did some work on helping fix it up so not sure I should be approving it myself13:24
*** thingee has joined #openstack-infra13:25
openstackgerritSean Dague proposed openstack-infra/os-loganalyze master: Remove swift support
*** dhellmann has joined #openstack-infra13:25
mordredsdague: awesome. one minor nit, while you're in there13:28
sdaguemordred: sure, I've got the patch up now, so will be easy to rev13:28
*** makowals_ has joined #openstack-infra13:28
openstackgerritSean Dague proposed openstack-infra/os-loganalyze master: Remove swift support
*** baoli has quit IRC13:29
mordredsdague: +213:29
*** makowals has quit IRC13:30
*** baoli has joined #openstack-infra13:30
*** eharney has joined #openstack-infra13:31
*** beekneemech is now known as bnemec13:31
*** jaosorior_brb is now known as jaosorior13:33
*** nt has quit IRC13:34
*** asselin has joined #openstack-infra13:35
*** asselin_ has joined #openstack-infra13:37
*** asselin__ has quit IRC13:38
mnaserhas anyone seen any issues with rdo/centos mirrors today?13:39
*** mattmceuen has joined #openstack-infra13:39
*** sekelso has joined #openstack-infra13:39
mnaseri just saw 2 jobs timeout in kolla13:39
mnaseroraclelinux and centos which both share the rdo repos13:39
mordredmnaser: we had a bit of a slowdown/timeout issue about an hour ago?13:39
mnasermordred sounds like around the same time these failed -
mnaserrecheck is ok or are things still iffy?13:40
_ari_hashar: is you could review this when you have some cycles that would be great
*** gcb has joined #openstack-infra13:41
*** asselin has quit IRC13:41
*** skelso has quit IRC13:41
mordredmnaser: recheck should be ok - it seems to be resolved13:42
*** dtantsur is now known as dtantsur|brb13:42
mnaserthanks mordred13:42
*** mat128 has quit IRC13:45
*** rossella_s has quit IRC13:46
*** rossella_s has joined #openstack-infra13:47
*** jcoufal has quit IRC13:52
*** hongbin has joined #openstack-infra13:53
*** lazyPower is now known as lp|swap13:56
*** knangia has joined #openstack-infra13:56
*** Goneri has joined #openstack-infra13:56
*** dizquierdo has quit IRC13:57
*** sree has joined #openstack-infra13:58
*** makowals_ has quit IRC13:58
*** sree_ has joined #openstack-infra14:00
*** sree_ is now known as Guest6443214:01
*** isaacb has quit IRC14:01
*** makowals has joined #openstack-infra14:01
*** ihrachys has quit IRC14:02
*** rbrndt has joined #openstack-infra14:02
*** jkilpatr has quit IRC14:02
*** nmathew has quit IRC14:02
*** mrunge_ has joined #openstack-infra14:03
*** kgiusti has quit IRC14:04
*** sree has quit IRC14:04
*** amotoki has joined #openstack-infra14:05
*** kgiusti has joined #openstack-infra14:05
fungipabelanger: mordred: looks like it's a very large sync from afs01.dfw to afs02.dfw which started at ~13:10 and is ongoing?
*** Guest64432 has quit IRC14:06
fungioutbound for afs01.dfw:eth0 and inbound for afs02.dfw:eth0 look to match up precisely14:06
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Fix debian-jessie settings for devstack
*** zz_dimtruck is now known as dimtruck14:06
fungiinterestingly, that seems to have started just as the impact cleared up?14:07
openstackgerritMerged openstack-infra/nodepool feature/zuulv3: Remove unused timing constants
*** Julien-zte has quit IRC14:09
AJaegerfungi, ianw set up an opensuse mirror today - not sure what time that was finished. Looked more like 8:30 if I interpret his comment correctly.14:09
openstackgerritThomas Morin proposed openstack-infra/project-config master: networking-bagpipe: add an experimental fullstack job
fungiohh, i wonder if we're riding memory utilization too close on afs01.dfw... we've been running with about 1.5gb paged into swap, and there was a memory bump around 12:00 which seems to have resulted in a sizable swap activity spike14:09
*** florianf has quit IRC14:09
*** sree has joined #openstack-infra14:10
fungiahh, right, we had this conversation the last time we were looking at that14:10
*** mdrabe has quit IRC14:11
*** Julien-zte has joined #openstack-infra14:12
fungidefinite spike in load average and swapping, with the memory bump lasting from around 11:35 to 13:10 (coincidentally ending when the large volume of sync traffic began)14:13
*** smarcet has joined #openstack-infra14:13
*** EricGonczer_ has joined #openstack-infra14:13
*** sree has quit IRC14:14
*** mdrabe has joined #openstack-infra14:17
*** pradk has joined #openstack-infra14:18
*** dimtruck is now known as zz_dimtruck14:19
*** amoralej|lunch is now known as amoralej14:20
*** Julien-zte has quit IRC14:23
openstackgerritMerged openstack-infra/tripleo-ci master: Set REMAINING_TIME for local run
*** ricardoas has joined #openstack-infra14:24
*** florianf has joined #openstack-infra14:24
*** amotoki has quit IRC14:25
*** Julien-zte has joined #openstack-infra14:27
*** Julien-zte has quit IRC14:27
*** Julien-zte has joined #openstack-infra14:27
*** Julien-zte has quit IRC14:27
*** gordc has joined #openstack-infra14:29
*** EricGonc_ has joined #openstack-infra14:30
*** EricGonczer_ has quit IRC14:30
*** beslemon has joined #openstack-infra14:32
*** cshastri has quit IRC14:33
*** zz_dimtruck is now known as dimtruck14:33
smarcetfungi: morning , how are u doing ? already sent u email with hiera variables to update for openstackid :) cheers14:35
*** rbuzatu has joined #openstack-infra14:38
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: WIP: Support built packages to be installed in image
*** nicolasbock has quit IRC14:41
*** chlong has joined #openstack-infra14:45
*** ZZelle_ has quit IRC14:46
*** nicolasbock has joined #openstack-infra14:48
*** baoli has quit IRC14:49
*** marst has quit IRC14:49
openstackgerritMerged openstack-infra/nodepool master: Remove link to modindex
*** dtantsur|brb is now known as dtantsur14:54
*** marst has joined #openstack-infra14:56
openstackgerritIhar Hrachyshka proposed openstack-infra/irc-meetings master: Removed korzen from the list of neutron upgrades meeting chairs
*** mriedem has joined #openstack-infra14:57
*** ihrachys has joined #openstack-infra14:58
*** amotoki has joined #openstack-infra14:59
*** sekelso has quit IRC15:01
*** sekelso has joined #openstack-infra15:01
fungismarcet: received... one concern though15:01
*** xarses has joined #openstack-infra15:02
smarcetfungi; sure tell me15:02
fungismarcet: the reason we enter the trove db hostnames into hiera (and the reason rackspace makes them so random-looking in the first place i expect) is that any tenant with an instance in that region can reach any trove db for any other tenant--there is no network access control there15:03
*** xarses_ has joined #openstack-infra15:03
fungismarcet: so if we instead move that to a dns cname we risk disclosing the ip address of the db and make it easier for attackers to target it with pre-authentication mysql vulnerabilities (which do crop up from time to time)15:04
*** makowals has quit IRC15:04
smarcetfungi: but inst ip address private one ?15:05
fungismarcet: yes, a "private" address in the rfc 1918 sense, but on a network shared by every instance for every tenant in that rackspace region15:06
fungiso the potential set of attackers is of course reduces from "everyone on the internet" to "everyone with a credit card"15:06
*** makowals has joined #openstack-infra15:06
fungier, reduced15:06
*** xarses has quit IRC15:07
smarcetfungi: i got your point15:07
*** john-davidge has joined #openstack-infra15:07
smarcetfungi;: i will speak with jp about it , thx for let me know :)15:07
*** dizquierdo has joined #openstack-infra15:09
*** sree has joined #openstack-infra15:10
gcbfungi:   we moved castellan to oslo  recently in and add separate ACL/group for castellan in
*** john-davidge has quit IRC15:12
gcbfugi:  we decided to both oslo and barbican core reviewers wll be that group, can you add me in,members then I can add others there15:13
*** makowals has quit IRC15:13
*** skelso has joined #openstack-infra15:13
*** sree has quit IRC15:14
*** felipemonteiro has joined #openstack-infra15:16
*** hashar has quit IRC15:19
*** jamesdenton has joined #openstack-infra15:20
*** dave-mccowan has joined #openstack-infra15:20
*** baoli has joined #openstack-infra15:21
*** baoli has quit IRC15:24
*** jascott1_ has joined #openstack-infra15:25
*** adrian_otto has joined #openstack-infra15:26
EmilienMpabelanger: re: puppetlabs mirror. The packages take more than 2GB, would it be ok?15:26
gcbfungi:  this is my info "ChangBo Guo(gcb)",  thanks15:28
*** annegentle has joined #openstack-infra15:28
clarkbgcb: I can do that in a moment15:28
gcbclarkb, thanks15:28
clarkbgcb: done15:29
*** jascott1_ has quit IRC15:29
gcbclarkb, cool :-)15:30
*** nt has joined #openstack-infra15:30
*** tesseract has quit IRC15:31
*** baoli has joined #openstack-infra15:31
fungigcb: sorry, in a meeting right now but any of our ~dozen gerrit admins can do that if it's urgent15:32
*** skelso has quit IRC15:32
*** HoloIRCUser has joined #openstack-infra15:32
gcbfungi, it doesn't matter , clarkb have done for me , thank all the same :-)15:33
*** thorst is now known as thorst_afk15:33
gcbyeah, I should check infra-core firstly15:34
jpmaxmanFungi we can make it more random if it helps15:35
*** rbuzatu has quit IRC15:35
jpmaxmanGiven how often these maintenance events come up just seems easier to be able to control with DNS15:35
fungijpmaxman: yeah, that's probably a fine compromise15:36
rajinirI'm new to using zookeeper with nodepool, how do I clean/delete the images so I can rebuilt them?15:38
clarkbrajinir: same way as pre zookeeper. Just nodepool image-delete them15:38
clarkbrajinir: as an alternative, don't explicitly delete them and instead explicitly build them and the will replace existing onces automatically15:39
*** amotoki has quit IRC15:39
Shrewsrajinir: also,
*** HoloIRCUser2 has joined #openstack-infra15:41
*** HoloIRCUser has quit IRC15:41
*** tonytan4ever has joined #openstack-infra15:43
*** mat128 has joined #openstack-infra15:43
jeblairwell, that's our sysadmin proceduces specific to openstack-infra, and while potentially useful, it doesn't apply to all cases.  if the nodepool docs themselves don't say how to delete or build an image, we should fix that.15:43
rajinir<Shrews,clarkb, thank you will try15:43
*** jordanP has quit IRC15:43
jeblair(the point of that documentation is so that when someone screams at us that an image is broken, any infra-root knows what to do to fix that)15:44
clarkb the command is documented but not general use case stuff15:45
Shrewsjeblair: yes, but i think the info likely would be helpful to rajinir since it's mostly generic in how the builder operates15:45
clarkb(important to note that this hasn't changed as part of zk work, that interface is the same)15:45
jeblairShrews, clarkb: yes.  i think the nodepool docs need improvement to actually narratively describe some of this.  but i think it's important to note that the infra docs linked are specifically geared toward solving one particular problem and only incidentally describe the system.  they recommend doing several steps that may not be necessary for rajinir unless they're running into exactly that situation.15:48
rajinirClarkb,Shrews: I have issued the deletes, the status changed to deleteing.., but I don't see any sign of delete happening , I restarted nodepool,  no sign of nodepool moving along,15:48
clarkbrajinir: do you have a nodepool builder process running?15:51
*** ijw has joined #openstack-infra15:51
clarkbrajinir: it is responsible for monitoring the state of images and taking action on them, building deleting being the big ones15:52
*** ihrachys has quit IRC15:52
*** pcaruana has quit IRC15:53
*** ihrachys has joined #openstack-infra15:53
rajinirclarkb: nodepool 10608     1  0 10:50 ?        00:00:00 /usr/bin/python /usr/local/bin/nodepoold --no-builder -c /etc/nodepool/nodepool.yaml -l /etc/nodepool/logging.conf -p15:53
*** ihrachys has quit IRC15:53
rajinirclarkb it is running, but no activity in nodepool.log, debug.log is saying no image found15:54
rajinirsomeone it looks hosed15:54
*** ihrachys has joined #openstack-infra15:54
SamYaplewhere might i find a history of jobs (specifically logs) from the 'post' queue?15:54
clarkbrajinir: thats --no-builder so there is no builder thread there. You need a separate process running nodepool-builder iirc15:54
*** HoloIRCUser2 has quit IRC15:55
smarcetfungi: i just sent u another email with the final settings, sorry for back and forwards :/15:55
rajinirclarkb: I do this warning: WARNING nodepool.cmd.nodepoold: --no-builder is deprecated and will be removed in the near future. Update your service scripts to avoid a brea15:55
jeblairclarkb, rajinir: yes, you need to start the "nodepool-builder" process as well.15:55
*** andreas_s has quit IRC15:56
*** e0ne has quit IRC15:57
*** ihrachys has quit IRC15:58
openstackgerritEdgar Magana proposed openstack-infra/irc-meetings master: Update for the UC IRC meetings
rajinirclarkb: jebliar: Thanks, restarted both nodepool and nodepool-builder services. Will check if that helped15:59
mordredSamYaple: I feel like someone wrote a little utility to print out log urls for a given change - sdague do you remember that utility?15:59
openstackgerritEdgar Magana proposed openstack-infra/irc-meetings master: Update for the UC IRC meetings
sdaguemordred: I don't off the top of my head16:01
*** cody-somerville has joined #openstack-infra16:02
jeblairwas it dhellmann?16:02
SamYaplehmmm ok well then a more direct question, is tehre a way to know if a post job failed?16:02
*** matrohon has quit IRC16:03
jeblairSamYaple: find the git sha for the commit in question (the merge commit if it made one), then its (where AB is the first two chars of the git sha, and ABCD... is the entirety)16:03
*** gcb has quit IRC16:03
clarkbor use openstack health16:03
*** mat128 has quit IRC16:04
fungismarcet: jpmaxman: received! will set those now if you feel it's safe to do so16:04
smarcetfungi: thx a lot :)16:05
*** yamahata has joined #openstack-infra16:05
mordredSamYaple: there's also a build_queue tag in logstash16:05
*** adrian_otto has quit IRC16:05
mordredSamYaple: so like:
mordredjeblair: I tried to find a docs page with that log structure documented but couldn't. do you think it's something we should add to the infra-manual?16:08
*** trown is now known as trown|lunch16:08
clarkbmordred: its there maybe just not organized well?16:09
jeblairi'd really like to switch post to change-merged, but we need the commit sha.  :|16:10
*** psachin has quit IRC16:10
mordredjeblair: ++16:11
SamYaplejeblair: thanks, that worked16:11
fungismarcet: jpmaxman: it's updated on our puppetmaster and will likely be applied on and in the next 15-30 minutes16:11
smarcetok got it16:11
smarcetfungi: thx so much :)16:11
mordredclarkb: cool - let me make a patch that breaks that bit out a little ibt16:11
jpmaxmangreat thanks fungi :)16:13
*** markus_z has quit IRC16:14
fungipabelanger: ianw: not sure which of you added opensuse_keytab to production/fqdn/ in the private yaml on puppetmaster.o.o, but it's not committed yet16:14
fungier, private hiera16:14
rajinirclarkb, jeblair: something is still wrong, the images are deleted from the list, but images rebuild is not happening..16:16
*** mat128 has joined #openstack-infra16:18
jeblairrajinir: is there any information in the nodepool-builder log?16:18
*** jkilpatr has joined #openstack-infra16:18
*** dizquierdo has quit IRC16:19
*** d0ugal has quit IRC16:19
openstackgerritMonty Taylor proposed openstack-infra/infra-manual master: Split out section about finding post-queue logs
mordredclarkb: how does that look? ^^ (also SamYaple)16:20
dhellmannsdague, jeblair, SamYaple :
mordreddhellmann: that's it!16:21
*** rbuzatu_ has joined #openstack-infra16:22
*** rbuzatu_ has quit IRC16:22
*** rbuzatu has joined #openstack-infra16:22
*** jpich has quit IRC16:24
*** mat128 has quit IRC16:27
rajinirjebliar: The last message is delete image one, Deleting image build dpc-0000000009 from local_0116:29
rajinirJebliar: on issuing the command manually to build, I see this exception Exception: Skipping build request for image dpc; paused16:30
clarkbrajinir: that means you are configured to not build that image, it is "paused"16:30
clarkbrajinir: you can remove the pause from your config then it should build16:30
*** mat128 has joined #openstack-infra16:31
rajinirclarb: I'm trying to build the xenial image for first time, which config file has this pause?16:32
rajinirThis is my image config details from the nodepool.yaml16:32
clarkbrajinir: the nodepool.yaml16:32
rajinirclarkb: I see the pause is set to True16:33
*** alexchad_ has quit IRC16:33
clarkbrajinir: yup so its "paused" no new builds will happen while that is set16:33
openstackgerritValeriy Ponomaryov proposed openstack-infra/project-config master: [Manila] Fix building of custom image
jeblairrajinir: did you set it to True earlier?16:34
rajinirclarkb: Is there a standard config , that's documented for zenial?16:34
rajinirclarkb: xenial,16:34
clarkbrajinir: we document the way we build them, but if there is any standard its probably set by ubuntu itself ( and we definiotely don't do all the things they do)16:35
rajinirclarb: I removed the pause and I can see some errors now16:35
*** derekh has quit IRC16:35
clarkbs/document/publish our configs/16:35
*** smarcet has quit IRC16:41
*** smarcet has joined #openstack-infra16:42
*** thorst_afk is now known as thorst16:42
rajinirclarkb: Thanks a bunch16:43
*** mat128 has quit IRC16:43
*** ihrachys has joined #openstack-infra16:46
*** smarcet has quit IRC16:46
openstackgerritMerged openstack/os-client-config master: Docs: add a note about rackspace API keys
*** smarcet has joined #openstack-infra16:47
*** rossella_s has quit IRC16:47
*** sambetts is now known as sambetts|afk16:50
*** beslemon has quit IRC16:50
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_queue_rate_limiting
openstackgerritMiguel Lavalle proposed openstack-infra/project-config master: Add job to Neutron DVR check queue panel
openstackgerritLuz Cazares proposed openstack-infra/project-config master: Add Refstack jobs to publish documentation
SamYaplemy jjb job templates do not have 'gerrit-git-prep', is this the builder responsible for cloning the project to the WORKDIR?16:52
*** annegentle has quit IRC16:53
*** e0ne has joined #openstack-infra16:55
*** aarefiev_afk is now known as aarefiev16:56
*** lucasagomes is now known as lucas-afk16:56
SamYapleah yea. it is16:56
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Add scenario005 job (4nodes)
zaroelectrofelix: will take a look tomorrow16:57
*** mat128 has joined #openstack-infra16:59
openstackgerritSam Yaple proposed openstack-infra/project-config master: Fix LOCI jobs
*** rbuzatu has quit IRC17:03
electrofelixzaro: cheers17:03
*** rlandy is now known as rlandy|biab17:03
*** rbuzatu has joined #openstack-infra17:03
*** dhajare has quit IRC17:04
*** jkilpatr has quit IRC17:05
*** chlong has quit IRC17:05
*** rbuzatu has quit IRC17:07
*** john-davidge has joined #openstack-infra17:08
*** adrian_otto has joined #openstack-infra17:10
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_queue_rate_limiting
*** Apoorva has joined #openstack-infra17:11
cloudnullo/ all, qq do we know of a difference between the images found in the osic for centos7 then other clouds? Asking because in OSA we're seeing erasurecode issue with swift on cent when gating on not the OSIC.17:12
cloudnullpassing gate
cloudnullfailing gate
clarkbcloudnull: they should be the same with the difference being format17:13
cloudnulland it looks to be consistent rax fails, osic passes.17:13
*** john-davidge has quit IRC17:13
clarkbrackspace runs xen and accepts vhd images, osic is kvm and accepts qcow2, but dib produces both from the same "master" image17:13
notmynamecloudnull: please feel free to ask in -swift if there's something we need to look at17:13
odyssey4meit could possibly be an inconsistent mirror too, unless those packages are pre-installed into the image?17:13
cloudnullnotmyname: IDK if it's a swift issues, seems to gate and run just fine on swift just rax cent7 is being angry17:14
clarkbodyssey4me: the mirrors should all be the same too since we use AFS to host them17:14
*** d0ugal has joined #openstack-infra17:14
* cloudnull wow, total English failure.17:15
*** mat128 has quit IRC17:15
cloudnullIDK if it's an issue with swift, seems to gate and run just fine locally, and on the osic for that matter, just rax cent7 is being angry.17:16
AJaegermordred, fungi, speaking about docs, could you review , please?17:16
clarkbcloudnull: the fails I see don't seem to be related to erasurecoding? they are bad status lines on acl check tests17:16
fungicloudnull: i wonder if it's a concurrency difference exposing something racy in some tests?17:17
clarkbcloudnull: which is possibly a https proxy fail? we had a similar issue with apache and neutron api17:17
*** dprince has quit IRC17:17
*** mat128 has joined #openstack-infra17:17
clarkbdue to timing on reused connections being closed as client made requests17:17
openstackgerritClint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Re-enable test_queue_rate_limiting_dependent
_ari_zaro: if you have cycles to review
clarkb(python requests doesn't handle http persistent connection closes cleanly)17:18
fungicloudnull: one of the biggest differences between our providers is the numbers of vcpus, so tests get parallelized into different groupings depending on how many processors are available17:18
fungiwhich can make tests run in different orders, or result in different tests running at the same time, exposing unintended test dependencies and races17:19
*** sekelso has quit IRC17:19
openstackgerritSam Yaple proposed openstack-infra/project-config master: Fix LOCI jobs
cloudnullah, I've been thinking this was due to ec traps because its failing on cross-policy writes (which is because our second policy is an ec policy) but maybe it is just a race issue in tempest.17:21
*** chlong has joined #openstack-infra17:21
_ari_waynr: could you review this change for me when you have the cycles
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Remove enabled_services=tempest from patrole gate jobs.
*** tosky has quit IRC17:23
*** esberglu_ has joined #openstack-infra17:23
openstackgerritFelipe Monteiro proposed openstack-infra/project-config master: Remove enabled_services=tempest from patrole gate jobs.
*** esberglu has quit IRC17:24
*** armax has joined #openstack-infra17:25
pabelangerand back online17:25
*** jerryz has joined #openstack-infra17:25
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Remove ovb updates job from promotion pipeline"
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Remove ovb updates job from promotion pipeline"
*** jaosorior is now known as jaosorior_away17:29
*** yamahata has quit IRC17:32
*** rbuzatu has joined #openstack-infra17:34
pabelangerEmilienM: 2GB for puppetlabs mirror? Should be okay17:34
*** nicolasbock has quit IRC17:35
EmilienMpabelanger: I'll try to find time to write the patches17:35
clarkbpabelanger: apache cache is between 648MB and 43MB on the various mirror hosts17:35
clarkb(osic is still largest)17:36
pabelangerclarkb: ya, I checked osic this morning, was 500MB17:36
pabelangersub 1GB is great IMO17:36
pabelangerclarkb: I'm also going to start on RDO rsync, going to create AFS volumes17:36
clarkbpabelanger: ya, but I am not sure its properly expiring old data17:37
clarkb-rw------- 1 www-data www-data 27488 Mar 30 22:26 @@@UGoA.data17:37
pabelangerclarkb: oh?17:37
clarkbthat is from more than 24 hours ago17:37
pabelangerwhat is the expire value?17:37
clarkbapache should have a default max expiration of 24 hours, thinking maybe we need to explicitly set the value17:37
clarkb(and our default is different because debuntu config or something)17:37
*** rlandy|biab is now known as rlandy17:38
pabelangerCacheMaxExpire ?17:38
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: s/gerrit-git-prep/zuul-git-prep/g
AJaegerfungi, jeblair, clarkb, pabelanger: I think we can retire gerrit-git-prep now, what do you think? ^17:38
clarkbpabelanger: ya17:38
clarkbpabelanger: docs say default is one day but wodnering if thats overridden somewhere by debuntu17:39
clarkbAJaeger: we only needed it for the post and release jobs right?17:39
clarkbAJaeger: but now we don't need it there anymore right?17:39
clarkbpabelanger: oh wait I see what it is that checks the origin server17:40
AJaegerclarkb: that's my understanding. We fixed the problems we had there.17:40
clarkbpabelanger: so its possible that we aren't actually rotating out data like I expected17:40
*** beslemon has joined #openstack-infra17:41
*** chem is now known as chem_gone17:41
clarkbpabelanger: that seems to only affect a resposne if it has already expired17:42
clarkbfor some reason I had thought it was general pruning17:42
pabelangerguess we need that?17:42
*** smarcet has quit IRC17:43
pabelangerI'll give it a test shortly17:44
clarkbI think based on the size of current caches, 2GB should be plenty to start17:44
*** sekelso has joined #openstack-infra17:45
openstackgerritThiago da Silva proposed openstack-infra/project-config master: add /usr/local/lib to library cache
clarkbhtcacheclean -n -p /opt/apache_cache -t -l 2048M17:46
clarkbpabelanger: ^ something like that in an hourly cron?17:46
pabelangerclarkb: ya, think so. Just seeing that the interwebs suggests17:47
*** shardy is now known as shardy_afk17:47
clarkbI'm going to manually install apache2-utils on osic mirror to see if ubuntu install it as a service17:48
clarkbdoesn't look like it so cron seems like good easy mode17:48
clarkbI think we want the -t because these urls change based on git sha117:50
*** smarcet has joined #openstack-infra17:50
clarkbso we won't be reusing directories often once cleaned out17:50
*** jamielennox is now known as jamielennox|away17:50
*** ijw has quit IRC17:51
*** sekelso has quit IRC17:51
*** nicolasbock has joined #openstack-infra17:51
clarkbpabelanger: do you want me to write the patch or are you already working on it?17:51
clarkb(but ya assuming that gets cache size under control I think we can open this up to all the jobs_17:51
pabelangerclarkb: if you want to sure17:52
pabelangeralready started on AFS volumes17:52
clarkbok will get that up shortly17:52
*** armax has quit IRC17:53
clarkbI'm going to run the above command on osic mirror just to test, and will then reduce down to 512M and see if it cleans17:53
*** jmorgan1 has joined #openstack-infra17:54
*** trown|lunch is now known as trown17:56
*** mat128 has quit IRC17:57
*** amoralej is now known as amoralej|off17:57
*** baoli has quit IRC17:57
*** Sukhdev has joined #openstack-infra17:57
*** mat128 has joined #openstack-infra17:57
*** Sihan has joined #openstack-infra17:58
*** d0ugal has quit IRC17:59
*** xyang1 has joined #openstack-infra17:59
clarkbpabelanger: so it seemed to work nicely17:59
pabelangerclarkb: nice18:00
*** aarefiev is now known as aarefiev_afk18:00
*** Sihan has quit IRC18:00
*** ZZelle has joined #openstack-infra18:01
*** electrofelix has quit IRC18:01
*** baoli has joined #openstack-infra18:01
*** aNuposic has quit IRC18:01
*** mat128 has quit IRC18:01
*** dprince has joined #openstack-infra18:02
*** alexchadin has joined #openstack-infra18:03
*** ssbarnea has quit IRC18:04
pabelangerclarkb: mordred: if you have a moment, for the RDO mirror18:05
*** ssbarnea has joined #openstack-infra18:05
pabelangerwill take about 300GB of space18:05
mordredpabelanger: on it18:05
*** markmcclain has quit IRC18:05
openstackgerritJames E. Blair proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks
mordredpabelanger: oh - speaking of mirrors ... hongbin is working on adding consistent support for getting docker installed properly to devstack18:05
mordredpabelanger: the script he's starting from ultimately installs an APT repo - although sdague and I suggested that perhaps having a mirror of that in the infra mirror system would be a better way to go18:06
pabelangermordred: hongbin okay cool, ya. Happy to help with that18:06
mordredespecially if this will be the devstack plugin that everyone using docker uses to get their docker18:06
*** markmcclain has joined #openstack-infra18:07
pabelangershouldn't be had to mirror dockers apt repos18:07
*** reed has quit IRC18:07
*** armax has joined #openstack-infra18:07
openstackgerritClark Boylan proposed openstack-infra/system-config master: Run htcacheclean to clean mirror caches
mordredpabelanger: no - it shouldn't be - I think it's a pretty small repo18:07
*** pvaneck has joined #openstack-infra18:07
pabelangerk, I'll add it to the list18:08
*** reed has joined #openstack-infra18:08
pabelangerclarkb: we should try later this week, and see how image caching works18:08
mordredpabelanger: yah - it's tiny:
mordredpabelanger: I'll be excited if the idea works out!18:09
clarkbpabelanger: ya, probably easiest thing is to just test it locally through a mirror18:09
*** mat128 has joined #openstack-infra18:09
clarkbpabelanger: rather than set up a job to do it18:09
clarkbthen if that works we can push it out to jobs and worry about disk space a lot more :)18:09
pabelangerclarkb: indeed18:10
*** dhellmann has quit IRC18:10
*** dhellmann has joined #openstack-infra18:11
hongbinpabelanger: thanks for helping out about that, just let me know once the mirror is ready :)18:11
openstackgerritMerged openstack-infra/project-config master: Remove enabled_services=tempest from patrole gate jobs.
pabelangerhongbin: sure, do you have a patchset going now?18:11
*** Swami has joined #openstack-infra18:12
*** harlowja has joined #openstack-infra18:12
*** florianf has quit IRC18:12
*** ssbarnea has quit IRC18:12
pabelangerhongbin: thanks18:14
*** tosky has joined #openstack-infra18:14
*** aNuposic has joined #openstack-infra18:15
*** florianf has joined #openstack-infra18:15
*** dtantsur is now known as dtantsur|afk18:15
openstackgerritMerged openstack-infra/tripleo-ci master: Populate logs exclude list with infra requirements
openstackgerritMerged openstack-infra/tripleo-ci master: Remove unused variables for Upgrades
fungii'm probably jumping at shadows, but the mailing lists seem really quiet today18:19
SamYaplequiet fungi!18:20
*** d0ugal has joined #openstack-infra18:20
* bkero knocks on wood.18:20
fungitailing the mailman posts log seems we're getting messages, and they do all seem to get delivered to me if i'm a subscriber, so nothing to worry about i guess18:20
mordredfungi: maybe we shoudl just blame the lovely spring weather april has brought?18:21
fungii should certainly be out catching up on yardwork18:21
mordredfungi: I did that on saturday. I'm still very aware that I did that on saturday18:22
fungiit's the homeowner exercise program18:22
*** rbrndt has quit IRC18:22
*** ralonsoh has quit IRC18:23
mordredit's very effective18:23
*** HeOS has quit IRC18:23
openstackgerritMerged openstack-infra/system-config master: Mirror rdo centos7 repo to AFS
*** annegentle has joined #openstack-infra18:25
*** kzaitsev_mb has joined #openstack-infra18:25
*** mat128 has quit IRC18:26
*** mat128 has joined #openstack-infra18:26
clarkbI dug up a bunch of bamboo18:28
clarkband there is still so much remainig18:28
*** ssbarnea has joined #openstack-infra18:28
*** e0ne has quit IRC18:28
mordredclarkb: by the time you dig up more, the stuff you already dug up will have regrown18:28
fungigive in and start harvesting it as a timber crop?18:30
clarkbits the miniature stuff so its not great timber18:30
fungiit's artisnal portland craft bamboo18:30
clarkbits also really splintery wood, not fun to work with18:30
fungifinest on the planet18:30
clarkbonce I poke through my finger with a sliver of it18:31
*** HoloIRCUser has joined #openstack-infra18:31
*** ssbarnea has quit IRC18:33
*** hrubi has joined #openstack-infra18:34
clarkbhello infra-root, /me is going to ask again for reviews on so that we can continue to make progress on the gerrit upgrade18:38
clarkbthat change is a prerequisite for deploying new gerrit on review-dev then doing testing there18:39
*** cody-somerville has quit IRC18:39
*** cody-somerville has joined #openstack-infra18:39
*** andreww has quit IRC18:41
*** ihrachys has quit IRC18:41
*** xarses has joined #openstack-infra18:41
*** ihrachys has joined #openstack-infra18:41
*** felipemonteiro has quit IRC18:41
*** xarses has quit IRC18:41
clarkbbut note that it affects review.o.o too, hence not single approving it18:41
*** xarses has joined #openstack-infra18:42
*** felipemonteiro has joined #openstack-infra18:42
jeblairclarkb, zaro: is it the case that 1.52 won't work < 2.12?18:42
mtreinishfungi: are there any other services you think would be good/easy to add mqtt support to in the short term?18:43
*** signed8bit is now known as signed8bit_Zzz18:43
mtreinishfungi: I'm trying to think about what else we should flush out before our talk at the summit18:43
clarkbjeblair: that was my understanding, I asked that in an earlier patchset18:44
mordredclarkb: you have 3 +2s now18:44
clarkbjeblair: however it may just be that that is what things are tested with so people don't want to change it?18:44
*** makowals has joined #openstack-infra18:44
mordredit's sad that 1.52 can't work with 2.11 :(18:44
clarkbwe can have zaro double confirm, and then I can approve and watch things18:45
*** signed8bit_Zzz is now known as signed8bit18:46
*** dimtruck is now known as zz_dimtruck18:46
mordredI mean - the patch is fine as is18:46
mordredit could just be simpler if it did work with 2.1118:46
mordredbut I don't think you need to wait for the double confirm18:46
clarkbI'm about to consume lunch but will approve after and keep an eye on review.o.o18:47
clarkbthanks everyone for reviewing18:47
jeblairyeah, that was definitely a +2 comment from me.  :)18:47
*** rbrndt has joined #openstack-infra18:49
*** john-davidge has joined #openstack-infra18:50
*** rbuzatu has quit IRC18:50
*** rbuzatu has joined #openstack-infra18:51
*** rbuzatu has quit IRC18:51
*** rbuzatu has joined #openstack-infra18:51
*** rbuzatu_ has joined #openstack-infra18:52
*** zz_dimtruck is now known as dimtruck18:53
*** john-davidge has quit IRC18:55
*** rbuzatu has quit IRC18:56
*** rbuzatu_ has quit IRC18:57
*** armax has quit IRC19:00
*** mat128 has quit IRC19:00
mtreinishinfra-root: if someone has time I'm curious if running "mosquitto_sub -h localhost --topic '$SYS/broker/bytes/received'" returns anything. I'm trying it from my laptop and not getting any data19:02
mtreinishI checked the config and we should be returning stats every 10secs (the default value)19:02
mtreinishI'm think it might be restricted to localhost (or I might need to add an acl to read that topic)19:02
fungimtreinish: running now19:03
fungiso far... silence19:03
fungiwhat is $SYS?19:03
mtreinishit's a special topic var for mosquitto for stats to be published: (the "Broker Status" section)19:04
*** adrian_otto has quit IRC19:04
*** csomerville has joined #openstack-infra19:05
mtreinishI'm looking into how we can collect metrics on how firehose is being used (if at all :) ) and how much data we're pumping into it19:05
*** florianf has quit IRC19:07
*** csomerville has quit IRC19:07
*** csomerville has joined #openstack-infra19:07
fungiyeah, it's dead jim19:07
*** cody-somerville has quit IRC19:08
mtreinishhmm, I'll have to figure out what I setup wrong. I'm guess it's an acl or something like that then19:08
* mtreinish reads more docs19:09
mordredsilly acls19:09
mordredmtreinish: have you tried turning it off and then back on again?19:09
*** hashar has joined #openstack-infra19:09
mtreinishmordred: heh, no. That actually works a lot for mosquitto in our experience :)19:11
*** sree has joined #openstack-infra19:11
*** alexchadin has quit IRC19:13
openstackgerritMatthew Treinish proposed openstack-infra/puppet-mosquitto master: Add system stats topics to public readonly acl
mtreinishfungi, mordred: ^^^ I think that'll do it. BUt I don't have a vm setup right now with mosquitto to test it19:13
openstackgerritmark mcclain proposed openstack-infra/project-config master: Enable Python3.5 gating for networking-arista
*** beslemon has quit IRC19:15
*** sree has quit IRC19:15
mtreinishfungi, mordred: if we can get that working I'm planning on writing a little script to dump that data into graphite so we can make graphs19:17
fungimtreinish: manually applied that config addition and restarted mosquitto. working over localhost for me now19:17
fungigive it a try?19:17
mtreinishfungi: it's working for me remotely too19:17
fungiawesome. approving19:18
mtreinishoh, looks like someone already wrote that in go:
openstackgerritPaul Belanger proposed openstack-infra/system-config master: Add rdo_keytab to mirror-update.o.o
fungiolaph: i noticed you ended up with two gerrit accounts for the same e-mail address... the ids are 5545 and 6726. i'm guessing just looking in the db that you're not actually using 6726 so happy to mark that account inactive if you're good with that19:20
openstackgerritmark mcclain proposed openstack-infra/project-config master: Add check-requirements to networkings-arista
olaphfungi: sounds good19:23
*** d0ugal has quit IRC19:23
fungiolaph: thanks, i've set 6726 inactive in gerrit now. there was no openid associated with it, so i don't think there's any way you've been logging into it for ages. shouldn't cause you any problems19:25
*** d0ugal has joined #openstack-infra19:25
mtreinishfungi: heh, it failed puppet-lint. I didn't even change any puppet code19:27
olaphif it does, I know where you live!  (well, kinda)19:28
*** makowals has quit IRC19:32
clarkbmtreinish: maybe tool got more strict?19:32
*** liusheng has quit IRC19:32
*** marst_ has joined #openstack-infra19:32
pabelangerclarkb: fungi: do you mind: adds missing rdo keytab file19:32
*** liusheng has joined #openstack-infra19:32
mtreinishclarkb: hmm, I guess. I'll push a fix for that right now19:34
*** marst has quit IRC19:35
*** d0ugal has quit IRC19:36
openstackgerritMatthew Treinish proposed openstack-infra/puppet-mosquitto master: Add system stats topics to public readonly acl
openstackgerritMatthew Treinish proposed openstack-infra/puppet-mosquitto master: Fix puppet-lint issue
mtreinishclarkb, fungi: ^^19:36
*** chlong has quit IRC19:37
*** harlowja has quit IRC19:37
*** xyang1 has quit IRC19:39
*** HeOS has joined #openstack-infra19:39
*** yamahata has joined #openstack-infra19:40
*** xyang1 has joined #openstack-infra19:40
*** dimtruck is now known as zz_dimtruck19:45
mordredolaph: I know where you live19:46
*** armax has joined #openstack-infra19:47
mordredolaph: you and your cow, too19:47
cmurphyclarkb: fungi while you're at it want to push and through so zaro and asselin_ can more forward?19:48
openstackgerritMerged openstack-infra/system-config master: Add rdo_keytab to mirror-update.o.o
*** rbuzatu has joined #openstack-infra19:50
*** john-davidge has joined #openstack-infra19:51
*** chlong has joined #openstack-infra19:51
pabelangerokay, RDO rsync going19:52
openstackgerritMonty Taylor proposed openstack-infra/shade master: Migrate create_image_snapshot to REST
openstackgerritEmilien Macchi proposed openstack-infra/project-config master: jjb: fix number of nodes for tripleo scenario005
*** john-davidge has quit IRC19:55
cmurphythanks fungi20:00
fungithanks for fixing!20:00
*** rbrndt has quit IRC20:00
*** rbrndt has joined #openstack-infra20:00
pabelangerfungi: mind a +3 on
*** rbrndt has quit IRC20:01
pabelangercleans up the mirror caches for reverse proxy20:01
pabelangerclarkb: once we land ^, I think we can have EmilienM update tripleo jobs to start using the proxy20:01
clarkband kolla and whoever else20:02
*** shardy_afk has quit IRC20:02
*** HoloIRCUser is now known as skelso20:02
EmilienMI'm ready like never20:02
EmilienMtripleo patch is ready to use all repos from mirrors, except buildlogs, but no mirror there20:03
openstackgerritMerged openstack-infra/puppet-gerrit master: Fix puppet-lint error
openstackgerritMerged openstack-infra/puppet-openstackci master: Fix puppet-lint errors
openstackgerritMerged openstack-infra/puppet-mosquitto master: Fix puppet-lint issue
*** shardy_afk has joined #openstack-infra20:04
*** jamielennox|away is now known as jamielennox20:05
openstackgerritMerged openstack-infra/puppet-mosquitto master: Add system stats topics to public readonly acl
*** skelso has quit IRC20:07
*** adrian_otto has joined #openstack-infra20:08
*** e0ne has joined #openstack-infra20:08
cmurphybtw puppet changes would be more fun to review if went through20:08
openstackgerritMerged openstack-infra/system-config master: Run htcacheclean to clean mirror caches
openstackgerritMerged openstack-infra/system-config master: Add an example not using websockets
*** e0ne has quit IRC20:09
fungimtreinish: did you see SamYaple's haskell indentation question on 449773?20:10
*** e0ne has joined #openstack-infra20:10
mtreinishfungi: I did not, let me look at that review20:10
clarkbcmurphy: did we just bump ahead to xenial without keeping trusty? I guess at this point probably just make that work then get updated to xenial20:10
*** Goneri has quit IRC20:11
*** sekelso has joined #openstack-infra20:11
*** rwsu has quit IRC20:11
cmurphyclarkb: i wasn't really paying attention when it happened but at some point it changed20:11
cmurphyclarkb: but it seems like it only changed for the puppet-openstackci beaker job and not for all the other ones20:12
fungiseems fine to me20:12
mtreinishfungi: responded, it's actually an rst indent question :)20:12
openstackgerritVasyl Saienko proposed openstack-infra/project-config master: Add ironic-multinode-job to networking-generic-switch
fungimtreinish: aha, it ended up in the quoted haskell, so i assumed the latter. i guess he was asking because it appears in the codeblock in the rendered copy?
*** rbrndt has joined #openstack-infra20:13
mtreinishfungi: that's part of the haskell, at least it didn't work unless I had that line at the top of the file20:14
fungimtreinish: okay, then i guess i have no idea why he was asking about it20:14
*** rwsu has joined #openstack-infra20:15
fungii agree it's rendering fine if that's part of the haskell source code20:15
clarkbmtreinish: fungi right thats like a compiler hint20:15
clarkbit modifies how thinsg are compiled20:15
fungimaybe he thought it was intended to be rst formatting metadata and didn't realize it was supposed to be part of the quotes source20:16
fungier, quoted source20:16
fungisince clarkb likes haskell, i'll just +2 and give him an opportunity to review20:17
openstackgerritMerged openstack-infra/puppet-openstackci master: Fix beaker-rspec tests
openstackgerritMerged openstack-infra/puppet-gerrit master: Add a known_hosts file
*** jamesdenton has quit IRC20:20
*** flepied has joined #openstack-infra20:22
clarkbmtreinish: fungi the only thing I would change glancing at it is qosGranted should be anonymous "_" since its not used anywhere else20:23
clarkbbut this is more verbose so maybe its ebtter20:23
clarkbok brb then approving gerrit change20:24
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: (WIP) Run puppet 4 in beaker tests
*** sekelso is now known as skelso20:24
*** zz_dimtruck is now known as dimtruck20:25
*** baoli has quit IRC20:25
*** jkilpatr has joined #openstack-infra20:25
*** megm has quit IRC20:27
*** e0ne has quit IRC20:28
*** rbrndt has quit IRC20:29
openstackgerritMerged openstack-infra/system-config master: Start firehose doc section for client on port 1883
*** Sukhdev has quit IRC20:30
*** megm has joined #openstack-infra20:30
*** baoli has joined #openstack-infra20:32
*** armax has quit IRC20:34
*** armax has joined #openstack-infra20:34
*** armax has quit IRC20:35
openstackgerritMerged openstack-infra/puppet-gerrit master: Fix placement of ssh public key
*** rbrndt has joined #openstack-infra20:40
*** rbuzatu has quit IRC20:44
*** beslemon has joined #openstack-infra20:45
ianwfungi: argh that's me sorry on the key.  did you commit?20:46
clarkbI have approved the puppet-gerrit change20:46
*** dimtruck is now known as zz_dimtruck20:48
*** zz_dimtruck is now known as dimtruck20:48
fungiianw: i did not since no idea who to attribute to nor what you wanted for commit subject20:48
*** armax has joined #openstack-infra20:48
fungii usually commit specific files when i make hiera edits or use -p if necessary and leave edits i didn't make un-committed while hunting down those who know how they should be annotated20:49
*** aeng has joined #openstack-infra20:50
*** david-lyle has quit IRC20:51
*** esberglu_ is now known as esberglu20:51
*** david-lyle has joined #openstack-infra20:51
*** beslemon has quit IRC20:51
ianwoh cool, pabelanger did. if it's any consolation, i got distracted updating the docs on how to add the key (will push after a reread today)20:52
*** annegentle has quit IRC20:52
*** harlowja has joined #openstack-infra20:53
pabelangerya, added it with rdo keytab20:54
clarkbwhich key?20:54
*** trown is now known as trown|outtypewww20:55
*** rbuzatu has joined #openstack-infra20:55
*** rbuzatu has quit IRC20:55
*** rbuzatu has joined #openstack-infra20:55
*** esberglu has quit IRC20:56
*** esberglu has joined #openstack-infra20:56
*** rbuzatu_ has joined #openstack-infra20:57
*** rockyg has joined #openstack-infra20:57
mtreinishfungi: I think that should enable us to get graphs on mqtt usage20:58
*** rbuzatu__ has joined #openstack-infra20:58
mtreinishfungi: heh, of course after I test it. I'll spin up a vm with mosquitto and statsd when I'm back home later today20:58
*** rbuzat___ has joined #openstack-infra20:59
*** thorst has quit IRC20:59
*** rbuzatu has quit IRC20:59
waynr_ari_: okay, i've reviewed it.21:00
*** dprince has quit IRC21:00
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Perform pre-launch merge checks
*** thorst has joined #openstack-infra21:00
*** d0ugal has joined #openstack-infra21:00
*** rderose has joined #openstack-infra21:01
*** esberglu has quit IRC21:01
*** rbuzatu has joined #openstack-infra21:01
*** umbSublime has joined #openstack-infra21:01
*** rbuzatu_ has quit IRC21:01
*** dprince has joined #openstack-infra21:01
*** rbuzatu_ has joined #openstack-infra21:02
*** rbuzatu__ has quit IRC21:02
*** rbuzatu__ has joined #openstack-infra21:03
*** tonytan4ever has quit IRC21:04
openstackgerritMerged openstack-infra/puppet-gerrit master: Update bouncy castle libraries for Gerrit 2.13
*** rbuzat___ has quit IRC21:04
*** thorst has quit IRC21:04
*** rbuzatu has quit IRC21:05
*** rbuzatu_ has quit IRC21:06
openstackgerritdane-fichter proposed openstack-infra/project-config master: Add barbican-tempest-plugin tests to Cursive
*** rbuzatu__ has quit IRC21:08
*** shardy_afk has quit IRC21:08
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Make tox-linters run the tox/linters play
*** ihrachys has quit IRC21:13
*** dprince has quit IRC21:14
*** annegentle has joined #openstack-infra21:15
*** ldnunes has quit IRC21:16
fungimtreinish: i also haven't reviewed the log processor patch yet until we make sure things go smoothly with the subunit worker (since the log processor code is more heavily used downstream)21:16
*** rfolco has quit IRC21:18
*** srobert has quit IRC21:18
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info role
*** mattmceuen has quit IRC21:20
*** eharney has quit IRC21:20
*** skelso has quit IRC21:21
*** smarcet has quit IRC21:21
*** mattoliverau has quit IRC21:24
*** darrenc has quit IRC21:24
*** darrenc has joined #openstack-infra21:24
*** gouthamr has quit IRC21:24
*** matt6434 has joined #openstack-infra21:25
*** ijw has joined #openstack-infra21:25
*** thorst has joined #openstack-infra21:26
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info role
*** smarcet has joined #openstack-infra21:28
openstackgerritClark Boylan proposed openstack-infra/zuul feature/zuulv3: Cleanup docs builds and error on warnings
*** thorst has quit IRC21:30
openstackgerritMerged openstack-infra/puppet-nodepool master: Rename nodepoold to nodepool-launcher for zuulv3
*** beslemon has joined #openstack-infra21:31
SamYapleive seen alot of zuulv3 stuff in the past few days... how close is that? is it about to land?21:32
clarkbSamYaple: its running jobs against itself, but I think there is still a fair bit of work to be done before we deploy it for all jobs21:33
clarkbin part just to do the transition itself but also sandboxing21:33
*** Goneri has joined #openstack-infra21:33
*** esberglu has joined #openstack-infra21:35
SamYapleah ok cool21:35
*** smarcet has quit IRC21:36
*** smarcet has joined #openstack-infra21:36
pabelanger too21:37
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Add net-info role
*** aysyd has quit IRC21:40
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Make tox-linters run the tox/linters play
*** beslemon has quit IRC21:41
*** ramishra has quit IRC21:42
*** kzaitsev_mb has quit IRC21:44
*** EricGonc_ has quit IRC21:46
*** EricGonczer_ has joined #openstack-infra21:46
*** felipemonteiro has quit IRC21:47
*** thorst has joined #openstack-infra21:47
*** Sukhdev_ has joined #openstack-infra21:47
*** hashar has quit IRC21:51
*** armax has quit IRC21:51
*** smarcet has quit IRC21:51
*** dizquierdo has joined #openstack-infra21:52
clarkbpabelanger: reading up on docker registries, we'll have to whitelist it as insecure (due to no https), but otherwise forward proxies seem to be expected to work so I kidna expect the reverse to work too21:52
*** wolverineav has quit IRC21:53
SamYapleclarkb: whats the issue with docker registries and https?21:53
pabelangerclarkb: nice! Surprising about https, guess we should open a github issue on this21:53
SamYaplei run a registry with https....21:54
*** wolverineav has joined #openstack-infra21:54
clarkbSamYaple: just that you have to explicitly allow non https (because you don't provide a proper url for downloading)21:54
SamYapleoh. thats on the client side though. is that what you are refering to?21:54
pabelangerOh, ya. that makes sense21:55
pabelangerwe do the same with APT21:55
SamYapleyea if thats the thing thats perfectly fine21:55
*** rhallisey has quit IRC21:55
clarkbya that21:55
fungithough for lack of index signing on apt repos, not lack of ssl21:55
fungiapt repos are almost never served over https anyway21:56
clarkbjust trying to reason through if a reverse proxy like we did for rdo would work21:56
clarkband I think it would given what I can read. I guess next step is set it up and test it21:56
SamYapleclarkb: there is a docker specific caching thing fyi21:56
clarkbSamYaple: ya trying to avoid having 50 difference $thing specific daemons for caching21:56
clarkbbecause we already have one21:56
fungithe alternative is to set up a cert for the proxy and explicitly add it to the trust on the workers21:57
*** armax has joined #openstack-infra21:57
SamYaplefungi: i think dockerd may actually ignore the system ca-certs21:58
inc0SamYaple: talking about registry in infra?21:59
SamYapleinc0: registry proxy21:59
SamYaplekolla gates will need to be modified, but i can submit a patch for that22:00
*** rderose has quit IRC22:00
inc0well correct me if I'm wrong but we can also push stuff to registry proxy right?22:00
fungithe proxy would be caching requests to dockerhub itself22:00
SamYapleright, the base layers for things inc022:00
*** wolverineav has quit IRC22:01
SamYapleubuntu/centos:latest kind of thing22:01
*** asselin_ has quit IRC22:01
inc0that's cool, but it's also possible to setup some proxy that we can push to as well, as in, if not found in registry, pull from dockerhub22:01
pabelangeryes, we have a spec going for that22:01
fungiit's a potential stop-gap for things hosted externally which we don't have a good mirroring solution for yet22:01
fungistuff we would push into would be a hosted registry22:02
funginot a caching proxy22:02
jeblairzuul meeting time in #openstack-meeting-alt22:02
*** armax has quit IRC22:02
inc0SamYaple: yeah I know this one, but what I'm thinking about is having docker-registry.openstack.org22:03
*** asselin has joined #openstack-infra22:03
inc0that will act as both cache and pushable place to replace tarballs hack22:03
SamYapleright thats not what we were tlaking about just now22:03
inc0we can kill 2 birds with one stone tho22:04
*** Swami has quit IRC22:04
fungiinc0: right, that's a separate need for things we generate. the current discussion is about improving access to things outside our control22:04
pabelangerinc0: we'd want to write to AFS, once then have a regional registry read from it and serve the image22:04
SamYapleits a bigger discussion though, this will allow us to stop pulling from dockerhub at all22:04
fungiinc0: except we can't kill those two with one stone22:04
inc0what I'm saying, if my memory works well, proxy can do both22:04
fungiinc0: they're only related by the words "docker image"22:04
inc0no, hold on22:04
*** rbuzatu has joined #openstack-infra22:05
inc0caching registry is squid-easy22:05
inc0as dockerhub is just https22:05
fungidoes dockerhub maintain an index? not seeing how to magically inject things into the cached index which aren't actually on dockerhub if so22:05
*** EricGonczer_ has quit IRC22:05
*** ssbarnea has joined #openstack-infra22:06
fungisure we could host local files and do 404 fallback to the proxy or vice versa, but that doesn't address indexing22:06
inc0fungi: easy enough, if exists locally - serve, else check with dockerhub22:06
SamYaplewhat i think we are refering to is starting dockerd --registry-mirror=http://infrahost22:06
SamYaplewhich has nothing to do with pushing22:06
fungiinc0: so you're saying there's no index maintained by dockerhub we'd need to augment/substitute to include locally hosted files?22:07
inc0fungi: not sure about that, frankly I don't know docker registry internals that well22:07
SamYaplethere it is, this cache is pull only. pushing still goes directly to dockerhub22:07
fungithis is orthogonal to running an apache caching proxy, which we already needed to add for other things. if you _don't_ want to use it for dockerhub access, then don't22:08
mordredyah - we started looking at that link late last week22:09
fungiit is being suggested as a (possibly temporary until we work out something better) solution to some of your download instability in jobs22:09
inc0what I'm saying is if we're running registry, we can configure it to make it pushable too22:09
inc0as we *really* need pushable registry22:10
fungiinc0: sure, completely separate discussion from this22:10
inc0don't see why sepeare thoi22:10
fungithis is just apache mod_proxy with caching enabled22:10
fungiit's not "running a registry" in any sense of the word22:10
fungiit's something we already have, which we can easily give you access to leverage if you want to do so22:11
SamYaplefungi: oh if you are using mod_proxy, that is not what I am refering to at all22:11
inc0well we didn't have much issues with dockerhub not working well, it's mostly about conservation of infra networking22:11
mordredinc0: yah - and setting up a registry is enough additional work that so far it has died on the vine every time it's come up for the last 6-9 months22:11
fungiclarkb: pabelanger: this reminds me, what do you think of setting it up to also proxy for tarballs.o.o? i bet that could relieve a lot of load on that server from ci jobs22:12
SamYaplefungi: but this could end up with the same affect i believe22:12
mordredwe _want_ to do it, and we want it ot be good22:12
clarkbfungi: ++22:12
mordredfungi: ++22:12
clarkbfungi: in fact that would be such a simple thing to do we should just do it22:12
inc0mordred: does it still need to be puppet?;)22:12
mordredinc0: yes22:12
mtreinishfungi: ok sure, that's a good way to handle it22:12
pabelangerfungi: ++22:12
*** rcernin has quit IRC22:12
*** marst_ has quit IRC22:13
*** mdrabe has quit IRC22:13
mtreinishfungi: also for the log_processor patch I wasn't sure what we wanted in the message payload. We've got a lot of flexibility there22:13
mordredinc0: and it still needs to not run in a docker container itself - which is totally easy and doable - it just needs to be done - the hardest part is going to be hooking it into the multi-region AFS backend infrastructure - BUT...22:13
inc0I mean last time it died because there is half-functional puppet module for registry somewhere and we didn't want to reimplement it22:13
inc0if you can just trust me on getting this done in infra, I'll get this done..22:13
SamYaplemordred: im not sure multi-region AFS backend is needed22:13
mordredif apache proxying works for docker hubs at all (which this may show us)22:13
SamYapleif everything pulls through a registry mirror (see link above) you could have one per region/cloud/whatever22:13
mordredthen we might be able to just have a single-region registry22:14
inc0mordred: if I can have access to one shared dir, that's all I need22:14
mtreinishfungi: we basically have access to all the metadata from zuul, and other stuff too I think22:14
pabelangerinc0: you can get it done today by proposing a patch to system-config. No special access is needed22:14
inc0 <- just need to pick it back up22:14
mordredthat's why I think great steps are: a) verify apache proxying b) maybe add pull-through registry cache c) implement pushable central registry22:14
inc0I'll do it then22:14
pabelangerOnce I finish up the AFS mirror for rdo, docker. My next task is to try a docker registry with afs22:14
pabelangersee how that all works and stuff22:15
fungiinc0: the puppet-docker_registry module got blocked because the puppet team expressed concerns we were creating a new module which duplicates one in puppetforge rather than working with the authors of that existing module to cover our use case22:15
SamYaplepabelanger: what i was saying above is im not sure AFS is needed22:15
inc0pabelanger: so if we can have single writer and mirror readers across regions, would that be easier?22:15
*** ZZelle has quit IRC22:15
*** Goneri has quit IRC22:15
pabelangerSamYaple: we want AFS, so we can write images once, then release them to all our cloud regions22:15
SamYaplepabelanger: singler writer globally and read-only pull caches per environment22:15
SamYaplepabelanger: yup thats what im saying22:16
*** rfolco has joined #openstack-infra22:16
*** rfolco has quit IRC22:16
mordredtwo different things - although I agree it's possible we may _not_ need an AFS backend for the pushable registry22:16
pabelangerright, so we need to validate we can write to AFS, trigger a 'vos release', and ensure other registry can download image from22:16
pabelangerthe issue is, what happens to the remote registry when the filesystem changes below22:17
inc0pabelanger: nothing22:17
inc0it just uses fs22:17
mordredpabelanger: right - I _think_ the pull-through registries per regoin may solve this for us differently22:17
SamYaplepabelanger: no, thats not what im refering to.22:17
*** mat128 has joined #openstack-infra22:17
inc0mirroring files is all we need22:17
pabelangermordred: agree22:17
openstackgerritColleen Murphy proposed openstack-infra/project-config master: Manage unbound on zuul workers in DIB
SamYaplepabelanger: single writer globally. no file mirroring directly. image pushes to single writer registry. all other regions have local registries with independant storage. everything pulls through the local registries which mirrors things as needed22:18
mordredSamYaple: right. I think that's an excellent option we shoudl consider22:18
pabelangerSamYaple: right, we are experimenting with that now22:18
pabelangeruntil last week, we didn't have a proxy cache22:19
SamYaplecool. then i think everything is laid out and we are all aware of the options. now just to figure out the best path forward22:19
mordredand the best part is - we can consider its siutability by just having independent per-region registries without a writable registry22:19
SamYaplemordred: ++22:19
mordredif the per-region cache registry approach _works_ - then we do not have to engineer how to make our central registry integrate with mutli-regoin replication22:19
mordredif it _doensn't_  - then we do22:20
*** jascott1_ has joined #openstack-infra22:20
inc0ok, so can we do it instead of apache mod_proxy?22:20
*** askb has joined #openstack-infra22:20
SamYaplestill have to write the registry code in puppet inc022:20
SamYaplethats the first stuf22:20
inc0I know, I also had volunteer to help in PTG22:21
fungiinc0: you can do it in addition to or instead of apache mod_proxy potentially, just pointing out that we'll have the apache mod_proxy solution anyway since we needed it for other stuff22:21
*** marst_ has joined #openstack-infra22:21
clarkbright its 3 lines of vhost config at this point to get the mod_proxy solution working22:21
mordredinc0: well, step 1 is to poke a little more at mod_proxy, since it exists and is being used for other things already - so it can be explored in parallel to poking at puppetting docker registries22:21
inc0fungi: if we're doing registry, since I think we agreed it's good thing to do22:21
fungiif you don't want to use the mod_proxy setup we have, it's as easy as ignoring it22:21
clarkbsince we've sorted it out generally otherwise22:21
inc0we might as well just do it22:22
*** matt6434 is now known as mattoliverau22:22
inc0will save us changes to gates of all kolla deliverables22:22
*** jascott1_ has quit IRC22:22
inc02 lines for you, a bit more for us, not much more but still22:22
openstackgerritColleen Murphy proposed openstack-infra/system-config master: Move unbound to openstack_project::server
inc0SamYaple: how about you and me get cracking on puppet module then?22:22
inc0and just get this one done?22:23
SamYapleinc0: i dont do puppet or i would have done it years ago22:23
SamYaplemnaser said he would be working on it22:23
mordredworks for me - fwiw, I'm not convinced we _need_ a whole puppet module - we need to apt-get install one package, and we'll need to install a config file22:23
inc0mordred: right...22:23
inc0thing is, how do we do it in infra?;)22:24
inc0without puppet?22:24
fungiSamYaple: i hadn't seen mnaser talk about working on that, he's got a spec up for publishing release artifacts to dockerhub22:24
mordrednono - it must be in puppet22:24
inc0because it's literally  that easy22:24
SamYaplefungi: talked ot him in a PM22:24
fungioh, cool22:24
mordredI'm saying we don't necessarily need a whole independent _module_ in its own repository22:24
clarkbinc0: its literally 3 lines of config... because I already did all the work for you...22:24
*** baoli has quit IRC22:24
inc0that and setup docker engines in our gates to actually use it...22:25
clarkbinc0: but thats cost regardless22:25
inc0as opposed to local registry which is config change22:25
SamYaplealright fine people. ill own this. ill talk with mnaser and get it done so we can stop going round and round22:25
clarkbits fixed, you can't avoid that22:25
pabelangerwe likely need a server running xenial22:25
* SamYaple grumbles22:25
inc0yeah, what I'm saying it's 2 changes instead of 1 and since we agree we want registry22:25
pabelangerand decided if we are using or OS package22:25
inc0let's just do registy...22:25
mordredinc0: we have _not_ agreed we want a registry running in each mirror host - it's entirely possible that's unneccesary22:26
clarkbinc0: no one is stopping you from doing the registry, we had to do the other thing anyways. So its there and a possible option. If you don't want to use it then don't22:26
inc0thanks SamYaple let me know if there is there is anything I can help with22:26
mordredinc0: but I do agree we _do_ want a central registry at some point - so the puppet work to be able to spin up a registry is not wasted work22:26
mordredso all the work that has been suggested here is valuable22:26
fungiyeah, the last discussion i'd seen was set up a registry pushing updates onto an afs filesystem, and then presenting that from all our per-provider mirrors (direct copies served via http out of read-only afs)22:27
inc0yeah technically mod_proxy apache will work just fine with single writer registry22:27
mordredinc0: ++22:27
inc0yeah, you're right, just central writer is ok22:27
inc0SamYaple: we have this
SamYapleyea im on that review i believe22:28
mordredthat said - if mod_proxy that's on the mirror hosts for some reason winds up not working well and we need to supplement with local pull-through-cache registries in each region, that should not be a hard step to make22:28
SamYapleor rather ive seen it22:28
inc0it needs reviving22:29
SamYaplethanks, gotta go guys!22:29
mordredSamYaple, inc0: thanks!22:29
inc0reviving as a revive not review:P22:29
inc0I'll go call some favors22:29
fungibut it was also pointed out that afs might be a poor fit for caching very large files, so caching http proxy of some centrally-served registry makes sense as an alternative22:29
*** aunnam has quit IRC22:30
inc0yeah, caching proxy+single pusher registry is elegant imho22:30
*** wolverineav has joined #openstack-infra22:30
fungiwhat are the largest files you expect to retrieve?22:30
fungicurious how big those image layers get22:30
*** gordc has quit IRC22:32
inc0fungi: that'd be single base image, think 600mb22:32
fungiokay, we may need to bump the max file size in our current config in that case22:32
inc0I threw this on #tripleo channel to as they might be interested in helping22:33
inc0and are good with puppet22:33
*** andreww has joined #openstack-infra22:35
*** rockyg has quit IRC22:35
*** andreww has quit IRC22:36
*** andreww has joined #openstack-infra22:36
*** chlong has quit IRC22:38
*** xarses_ has quit IRC22:38
*** aunnam has joined #openstack-infra22:39
*** armax has joined #openstack-infra22:39
*** Goneri has joined #openstack-infra22:41
*** jascott1_ has joined #openstack-infra22:42
pabelangerI can start on the central registry22:43
*** jascott1_ has quit IRC22:43
pabelangerwe need a xenial host22:43
inc0pabelanger: thank you good sir22:44
*** ijw has quit IRC22:45
inc0oh one thing tho, we'll need secure access there22:45
inc0that was one of issues afair22:45
*** rbrndt has quit IRC22:45
*** jascott1_ has joined #openstack-infra22:45
inc0I mean it's not hard to setup22:45
*** ijw has joined #openstack-infra22:45
inc0just how to give access to these secrets to gates22:45
pabelangerinc0: it is in the spec22:46
pabelangerand is a good usecase for zuulv322:46
*** Goneri has quit IRC22:46
clarkbpushes you mean?22:46
clarkbthe reads shouldn't matter22:46
*** ijw has quit IRC22:47
inc0yeah pushes22:47
pabelangerto publish we'd do it how it is down today with tarballs22:47
pabelangerupload to tarballs.o.o from unsecure node,22:47
pabelangerthen secure node will download and publish22:47
*** ijw has joined #openstack-infra22:47
mordredyup. pushes will be much better with zuul v322:47
pabelangerthis is why zuulv3 is appealing, it would flex the new secrets system22:47
fungithe secure worker thing was for if you wanted to push releases up to dockerhub (a la mnaser's spec)22:48
*** sshnaidm is now known as sshnaidm|off22:48
fungia central registry, if it's also an http server, doesn't need special credentials unless i'm missing something22:49
mordredit needs credentials to publish to22:49
fungieven if it publishes to itself?22:49
pabelangerdoes it have an ACL, where we can disable from localhost lets say?22:49
mordredit wouldn't publish to itself22:49
mordredI mean - something needs to publish to it22:49
fungioh, i'm picturing a docker registry as being like an apt repo22:49
mordredno - it's a thing with a rest api that you push things to22:50
inc0well technically we could solve it on networking layer22:50
*** xyang1 has quit IRC22:50
jeblair  is almost a change to add a docker server in the way that mordred suggested earlier.22:50
fungicould that rest api be on localhost, with a job we trigger on that node from zuul after any job pushes an image to tarballs.o.o it needs to incorporate?22:50
mordredfungi: quite possibly, yes22:51
pabelangerjeblair: danke!22:51
*** pradk has quit IRC22:51
NobodyCamGood Afternoon Infra folk. Just a off the wall question (as always) is there a order to restart G(errit)/J(enkins)/Z(uul)22:51
NobodyCamI ask because I restarted Jenkins / zuul and now have a bunch of errors like:22:51
NobodyCamERROR zuul.Gearman: Job <gear.Job 0x7fb69c116450 handle: None name: build:zulu-cross-python-check-changes-no-build unique: f1016cdc1b5545d9b59cee663267c307> is not registered with Gearman22:51
mordredfungi: but ultimately I think we wind up doing that much more inefficiently than if we figure out how to allow things to publish to a local docker registry using docker - because otherwise we're re-copying a ton of stuff over and over again22:51
* mnaser sees a lot of talk about the docker repo22:52
*** ijw has quit IRC22:52
mnaseri haven't had a chance to reupdate the spec because of a major concern -- "this can be solved by zuulv3"22:52
jeblairNobodyCam: once nodepool nodes come online and jenkins registers that it can run those jobs, those errors will dissipate22:52
mnaserif we ignore that, we could start moving forward with that.  unless another solution is being proposed right now22:53
mordredfungi: I think the thing you're saying is definitely the thing we can do for now22:53
fungigot it. so it's not just some files you stick in a directory, run a script to generate indices, and then publish with a webserver?22:53
mordredfungi: nope. not even a little bit22:53
NobodyCamjeblair: Thank you.. Will keep an eye on things :)22:53
NobodyCamhey hey g'Afternoon mordred :)22:53
* fungi wonders when/why simple file distribution became so complicated22:53
mordredfungi: because fancy?22:54
mordredfungi: it's hard to build a business model to become a unicorn if you use existing technology in a way that's immediately interoperable with things that existed before22:54
mordredhey NobodyCam !22:54
mnaserif you dont mind, can i get caught up with if we're trying to run mirrors or a repo to push things to (ala tarballs.o.o)22:54
mordredmnaser: yes! I can help with that22:55
fungiahh, makes sense22:55
mordredmnaser: there are a few parallel things - so stick with me for a sec22:55
mordredmnaser: for the effort "make consuming docker images on build slaves faster/more reliable" - clarkb has a mod_proxy entry on each of the webservers in our per-region mirrors to do a caching web proxy22:56
openstackgerritMerged openstack-infra/tripleo-ci master: Revert "Remove ovb updates job from promotion pipeline"
mordredmnaser: so the work there should be "make sure docker points to the per-region mirror URL instead of the dockerhub URL" and that should be good to go22:56
clarkb(assuming its actually just http)22:57
mnaser(i'll keep my thoughts for the end about these things)22:57
mordredyah. if the mod_proxy approach does not work, then the fallback there is to run a registry in pull-through-cache mode on each per-region mirror22:57
mordredwe're starting with mod_proxy because we're already using that for some other things22:57
*** sdague has quit IRC22:57
inc0clarkb: https22:58
mordredmnaser: for the "we want a registry we can push to" - we're reviving the puppet-docker_registry module and some combinatoin of SamYaple, you and folks from #tripleo are going to work on getting the registry module ready22:58
*** sree has joined #openstack-infra22:58
mordredso that we can install a docker registry machine to be a place we can publish images to22:59
mordredwe still need to sort out exact workflow around actually publishing images to it - but it's most likely that the first versoin of that is going to be VERY similar to how we do other things today22:59
*** marst_ has quit IRC23:00
pabelangerjust left comments on 40100323:00
mordredwhich is that jobs that build images will upload them to tarballs.o.o and then we will have a trusted job that will download those and inject them into the registry23:00
fungii gather that would mostly be for things like updating branch-tip images for in-developent versions of stuff23:00
mordredfungi: yah23:00
funginot for final release artifacts you'd want to be more globally useful23:00
inc0fungi: ideally on merge on stable branches and master23:01
mordredfinally there is "publish to dockerhub" - which can be also done as part of the same system that downloads form tarballs.o.o and uploads to the local registry - it can just as easily upload to dockerhub23:01
inc0totally awesome: also push tagged stuff and allow people of world around to pull from it23:01
mordredbut also we expect that process to work better once zuul v3 is out23:01
fungitagged stuff you want the world to consume would presumably go to dockerhub anyway23:01
*** adriant has joined #openstack-infra23:01
mordredinc0: I believe we want to publish actual tagged releases to dockerhub and ratherthan running a 'production' registry that people's deployments might point at23:02
mordredfungi: yah. that23:02
inc0either way works23:02
mordredI don't want to be on the hook for running a dockerhub that might directly break mnaser's cloud :)23:02
inc0just throwing it out there, I don't have big incentive to use our infra for it23:02
inc0if we can get autopushed done properly23:02
mordredyup. I think we can get pushes to dockerhub and done in the fulllness of time - and our caching/proxy approach should allow those to be consumed23:03
*** sree has quit IRC23:03
*** annegentle has quit IRC23:03
mordrednow - I _do_ think we should run a registry and publish to it just for completeness- just like we publish tarball releases to tarballs.o.o - even though we also publish them to pypi23:03
*** ijw has joined #openstack-infra23:03
*** ijw has quit IRC23:03
mordredjust because having more than one publication point is good for allowing people to double check content validity and stuff23:04
fungi(speaking of the fullness of time, please review mnaser's spec about automated publishing to dockerhub if you have thoughts on the matter)23:04
inc0and have backup source if dockerhub dies in fire for various reasons23:04
*** ijw has joined #openstack-infra23:04
mnaserso to summarize work items of what mordred explained: 1) infra sets up some sort of per-region reverse cache for dockerhub to reduce internet pulls for things from dockerhub.  b) infra+consumers of docker registries work together to build something that can stand up a docker registry, 3) the spec regarding publishing ( ) could be revised to include 'publish to local' and 'publish to dockerhub'23:05
jeblairbtw, i'm not planning on updating  i just wrote the first patchset for inc0 to fill in the blanks; if we use it, i expect someone else to update it23:05
fungiright, like with sdists/wheels and pypi.p.o vs tarballs.o.o23:05
pabelangerjeblair: I was planning on pushing something up23:05
*** rbuzatu has quit IRC23:05
mnaserthat's a nice start23:05
fungimnaser: that sounds like what i got from the discussion, yes23:06
mnaseri would be happy to finish that stuff up as i've done too much puppet for my own good23:06
inc0mnaser: if we get local registry in infra23:06
mnaserid just have to ask if there's a simple way to test these on a local vms (or something infra has used to easily bootstrap things locally in a vm)23:06
inc0I think we can table autopusher to dockerhub for zuulv323:06
mnaserquite honestly, i've been working on our defcore stuff all day and i cant stand looking at tempest a second more23:07
mnaserso i dont mind toying around with this for now23:07
mnaseri dont expect it to take very long23:07
jeblairmnaser: yeah,  has a quick how-to local test infra puppet changes23:07
inc0mnaser: right, if you make docker registry work in infra, I owe you beer23:07
*** mat128 has quit IRC23:07
mnaseron a super side note, would it be okay to install docker + run registry in docker?23:07
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Collect test-results for tox jobs
mnaserin my experience, its a bit easier to work with and much better tested overall (and we can easily stay on newer releases)23:08
mnaserrather than rely on os deployed ones23:08
inc0well registry is glorified http server23:08
mnaserthank you jeblair and inc0 i'll get on this today23:08
inc0so not sure if we want to add complexity of installind and configuring docker daemon23:08
mnaserill take your word for it inc023:09
*** ijw has quit IRC23:09's a golang bin which we can, quite literally, curl down and run23:09
inc0or do apt-get install23:09
pabelangermnaser: I am going to push up an update to 401003 shortly, we can iterate on that if you'd like23:09
bkeroI don't think golang advocates using distro packages.23:09
mnaserlets see what release its running, i know docker registry performance improved a lot lately so we'll have to check23:10
mnaserplease pabelanger -- in the meantime i'll do the little work needed here to stand up a local env23:10
inc0mnaser: registry 2 is better....a lot better...23:10
mnaserand i think distros still dont ship 2 afaik23:10
inc0for what its worth, project already exists in project.config
mordredbkero: I do not care23:11
inc0in this case I'd go for curled binary23:11
inc0mordred :D23:11
mordredin this case I would go for apt-get installing it23:11
mordredsince it's in ubuntu23:11
mordredand there is no reason not to use it23:11
bkeromordred: I'd care if we needed to call on them to support one day and they told us to pound sand.23:11
inc0mordred: if apt doesnt have v223:11
mnasermordred id do that, but i think its best we verify the version available23:12
inc0then we really want v2...23:12
mnaserthe old registry was very bad and slow23:12
mordredbkero: I have zero expectatoins that they will ever support it anyway23:12
mordredand ubuntu xenial has v223:12
mordredso we're fine23:12
inc0indeed we are23:12
mnaseroh cool23:12
inc0cool mnaser thanks for help with this on23:13
inc0I'll get to full release upgrade gates when we have registry...23:13
*** marst has joined #openstack-infra23:13
inc0(as in...gates that will deploy stable and upgrae to cool is that?:D)23:13
mordredVersion: 2.3.0~ds1-123:13
mordredis what's in ubuntu23:13
mnaserinc0 awesome.  np.  ill do my best to take this as far as i can23:13
inc0mnaser: let me know if you need any help23:14
* bkero remembers this whole thread about it
mnaserif i can get feedback circled quickly, it would make it easier so hopefully ill base my work on what small things jeblair  and pabelanger started off23:14
*** andreww has quit IRC23:14
mordredbkero: that thread is FUD23:14
inc0bkero: after spending last week + weekend with golang23:15
inc0I don't care at all what they do or do not propse23:15
inc0because this is bag of bull...23:15
bkeromordred: about the only thing I took away is that person claiming that distro packages change paths23:15
bkeroWhich I wouldn't doubt, but it's likely not to matter for our use.23:15
mordredbkero: yah. that's what I get too23:15
mordredalso - distro package managers don't change paths when upstream paths are in the right place :)23:16
bkeroHaha really23:16
mordredwe've been putting files in places for a LONG TIME - it's not rocket science23:16
bkeroLet's talk about the LSB23:16
mordredunless you just flat don't care23:16
*** ssbarnea has quit IRC23:16
clarkbalso thats about installing Go, not about installing things that are compiled from Go23:16
* bkero peers into /usr/local23:16
*** aunnam has quit IRC23:17
mordredto be fair - I have go installed not-from-packages - but I have it installed in ~/.local/bin/go too23:17
bkeroIf distro-ifying other package managers is anything to go by, it'll also disable auto-update mechanisms for itself and whatever it considers $GOBIN/$GOHOME and accompanying libraries installed system-wide.23:17
mnaserfor sometime23:17
mnaseri had to work a lot with smartos, stuff created of joyent from whatever was left of sun23:17
mordredoh wow23:18
mnaserif you like things going in the right place, you'd love it23:18
clarkbmnaser: you are reminding me of my days using blitz23:18
openstackgerritPaul Belanger proposed openstack-infra/system-config master: [WIP] Create
pabelangermnaser: untested ^23:18
pabelangershould get us started23:18
mordredbkero: at least go binaries only need GOHOME at build time and not at runtime23:18
mnaserpabelanger sweet, i'll cherry pick and see where i go23:18
pabelangerthings like HTTPs need to be solved23:19
pabelangerand user auth23:19
pabelangerneed to read up more on it23:19
pabelangerfor now, just added htaccess username / password23:19
NobodyCamjeblair: Fyi- I had to edit a slave to get the jobs reregistered with gearman23:19
*** arxcruz has quit IRC23:19
*** aunnam has joined #openstack-infra23:19
mordredfungi: fwiw, reading the docs, I believe we _could_ write a thing to just write image layers to filesystem locations and serve them with apache23:19
mordredfungi: it would be completely custom development on our part23:19
mordredfungi: but it does not appear that there are any things in the docker-registry API that cannot be handled by just putting files on disk in the right layout23:20
inc0well yes, we can make docker registry from a http server23:20
inc0that just navigates filesystem and serves content23:20
mordredinc0: do you know if there is already a tool to explode a docker image into the right files on filesystem?23:20
inc0well, we do it today with tarballs23:21
*** yamahata has quit IRC23:21
inc0we simply run registry in gate, push to localhost and tarball what comes out on the other end23:21
*** rbuzatu has joined #openstack-infra23:21
*** arxcruz has joined #openstack-infra23:21
bkerohehe oh man /usr/lib/go/bin/go23:21
inc0go bin go23:21
mordredbkero: I can't IMAGINE why distros would move that23:21
bkeromordred: actually that's an ubuntuism23:22
mordredbkero: wat?23:22
bkeroupdate-alternatives --remove go /usr/lib/go/bin/go23:22
bkerofrom the preinst rules file23:22
mordredoh. alternatives23:22
mordredof course23:22
fungimight more accurately be a debianism23:22
* bkero is down the rabbit hole of apt-get source golang-go23:22
mordredinc0: ok - I'm going to look at that and learn a little bit more23:22
*** doffm has left #openstack-infra23:22
bkeroCould be. It's a debian rules script23:22
openstackgerritIan Wienand proposed openstack-infra/system-config master: Updates to adding mirror documentation
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Collect test-results for tox jobs
mordredinc0: might not be worth anything - but it would be good to konw all th ethings23:23
*** wolverineav has quit IRC23:23
inc0mordred: I'd still be more comfortable with having registry running23:23
mordredinc0: I just like to know what the fallback options are23:23
clarkbzaro: ok, so next step is pushing a change to upgrade review-dev, and we'll have to follow our rough steps for doing a normal upgrade. Any chance you have time some time this week to run through that with me?23:23
bkerofungi: Although the changelog is addresses23:23
inc0mordred: sure, good hacking project23:23
*** signed8bit is now known as signed8bit_Zzz23:23
inc0as far as I know it's just glorified file server23:23
mordredinc0: you know, for when docker's VC folks tell them to remove their source code form the internets and turn it all proprietary23:23
fungibkero: even on -1 package revs?23:23
mnaseralright almost getting that local machine running and then the puppet apply fun will start soon23:23
ianwnobody reviewed my comedy changelog yet23:24
inc0well, if that happends I'm starting project called vac23:24
bkerofungi: yes, and control says 'Maintainer: Ubuntu Developers'23:24
inc0and build rockets there23:24
*** signed8bit_Zzz is now known as signed8bit23:25
*** rbuzatu has quit IRC23:25
fungibkero: for the golang compiler? wacky that they aren't importing that from debian23:26
fungibut i've learned to stop being surprised there23:26
mnaserpabelanger: if that does install things successfully (still getting a sample machine setup).  what would be the next step?23:26
clarkbbkero: is there something that dictates the binary path? I guess I don't grok what the issue is23:26
clarkbthen again I had an nfs mounted /usr/local/bin/ghc for things so maybe I am one of thoes terrible people the golangers don't like23:27
pabelangermordred: I'd be okay with mirroring docker infrastructure without using docker things. Hosting a registry, I have hopes :)23:27
mordredinc0: can you point me to the place where you do the "run local docker registry, push to it, then tar what comes out hte other end" part?23:27
mordredpabelanger: ++23:27
pabelangermnaser: we'd bring a server online for the control plane23:28
pabelangermnaser: actually, -1 on the review. We need to make sure afs is also installed23:28
pabelangerI think server.pp has a flag23:28
mnaserafs => true23:29
*** reed_ has joined #openstack-infra23:29
bkero has a few gems though. "Remove Plan9 rc(1) scripts"23:30
bkeroDoesn't look like there's anything nefarious in there though.23:31
openstackgerritPaul Belanger proposed openstack-infra/system-config master: [WIP] Create
bkerolots of touching23:31
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Collect test-results for tox jobs
mnaserpabelanger are we okay with storing the encrypted value in hiera for auth or should we encrypt the plain value in puppet so that the password is visible in hiera23:32
pabelangermnaser: we have encrypted things in hiera already23:33
bkeroand the S390 port-in-a-patch, and the passing -no-pie to the linker23:33
pabelangerso, that should be fine23:33
mnaserokay cool23:33
pabelangermnaser: I think the better question is, is htpasswd the right auth to use23:33
pabelangerI see token support23:34
mnaserpabelanger token is a bit more complicated, it pretty much relies on an external api for auth so it involves running another piece of software23:34
*** dizquierdo has quit IRC23:34
mnaserkinda like keystone so to speak23:34
pabelangerah, ya. we don't want that23:34
mnaseryou can setup things like this:
*** reed_ has quit IRC23:35
pabelangerwe should also consider wrapping the registry behind apache23:35
clarkbI get /usr/bin/go on tumbleweed23:36
clarkbwhich is symlink for alternatives to /usr/lib64/go/bin/go23:36
bkeroNo GOVER in there?23:38
*** adrian_otto has quit IRC23:38
*** armax has quit IRC23:38
bkeroUbuntu definitely involves /usr/lib/go-${GOVER}/23:38
bkeroWhich is apparently the GOROOT23:38
*** dingyichen has joined #openstack-infra23:39
clarkbno /usr/lib64 doesn't hve a bunch of go versions23:41
clarkbnot sure why they use alternatives23:41
clarkbprobably because as a distro /usr/bin/ is in your path and they don't want to force users to add /usr/lib64/go/bin to their paths23:41
*** mat128 has joined #openstack-infra23:43
mnaserpabelanger do we want to do the https inside apache then?23:43
*** signed8bit is now known as signed8bit_Zzz23:44
pabelangermnaser: ya, lets bind docker to 8080 localhost, then we can https apache on 80 / 443 for now23:45
*** s-shiono has joined #openstack-infra23:45
mnaserpabelanger cool, on it23:45
pabelangerthen we can copy pasta vhost configuration for ssl23:46
*** Hal has quit IRC23:46
*** asselin has quit IRC23:46
*** asselin has joined #openstack-infra23:47
*** tosky has quit IRC23:47
inc0mordred: <- this is registry starting23:48
inc0pushing is just kolla-build push23:48
inc0sorry, wrong place23:48
*** s-shiono_ has joined #openstack-infra23:49
inc0and then we tar contents of /tmp/kolla_registry23:49
*** s-shiono has quit IRC23:51
*** asselin_ has joined #openstack-infra23:54
*** asselin has quit IRC23:57
*** mat128 has quit IRC23:58

Generated by 2.14.0 by Marius Gedminas - find it at!