Friday, 2016-05-06

kjorgensenthe message saying "Starting check jobs" and the 'status_url' from /etc/zuul/zuul.conf00:00
anteayaand is the job currently stuck in the queue on your jenkins?00:01
kjorgensenwell it doesn't even show in the jenkins web interface00:02
*** dizquierdo has quit IRC00:02
anteayalets see if we can find out what the gearman server thinks is happening00:02
anteayacan you find logs for the gearman server?00:02
kjorgensenthey are in /var/log/zuul/gearman-server.log right?00:03
anteayato be honest I don't know00:04
anteayaI personally don't run these services00:04
anteayaI'm just really practiced at asking questions00:04
*** tbachman has joined #openstack-infra00:04
anteayabut if there is a file at that path, let's see what it says00:04
*** chlong has quit IRC00:05
*** kzaitsev_mb has quit IRC00:06
*** rbrndt has quit IRC00:07
kjorgensenok :) yea i'm not 100% sure either but I assume it's the one. Strangely, that file is mostly empty except for a cryptic error message
*** yamamoto has joined #openstack-infra00:08
anteayakjorgensen: I'm going to conclude that your gearman server is not running00:08
anteayakjorgensen: are you able to assess if it is running or not?00:09
anteayaah you are using python2.600:09
anteayais there a reason you are using python2.6?00:09
anteayacould you possibly convinced to try running it with python2.7?00:09
anteayaand what version of gear are you running?00:10
*** sflanigan has joined #openstack-infra00:11
*** sflanigan has joined #openstack-infra00:11
kjorgensenhaha yea we're kinda behind00:11
kjorgensenand by "kinda" I mean "very"00:11
anteayacurrent version of gear is 0.5.9
anteayawell we may have found a potential issue00:12
anteayagood news is the path forward is very clear00:12
anteayaI'd say check all your versions and upgrade things then give it another shot00:13
anteayaalso we have a puppet module:
anteayathat has instructions for how folks can set up their own ci00:13
*** ddieterly has joined #openstack-infra00:13
anteayathe use case for this module is that folks set up their own testing to interact with our gerrit00:14
anteayabut you can set up your own gerrit and have your ci point to that00:14
*** vhosakot has joined #openstack-infra00:14
kjorgensenso my installed versions are gear==0.5.9 and zuul==2.1.000:14
anteayawell that is the latest gear00:15
asselin_kjorgensen, anteaya the published docs for that repo are here:
anteayaand the latest zuul:
anteayathanks asselin_00:16
anteayakjorgensen: any thoughts on trying it with a newer python release?00:16
anteayaasselin_: I missed you at summit00:16
kjorgensenanteaya: I can look into that00:16
anteayakjorgensen: sounds like it might be worth investigating00:16
asselin_yeah, unfortunately I couldn't make it00:16
anteayaasselin_: what python version do you run for your tooling?00:17
kjorgensenanteaya: thanks for trying to help00:17
anteayaI always try00:18
kjorgensenwe actually use chef for our CI deployments, I'm guessing there isn't docs to set up zuul with chef? ;)00:19
anteayawell our team uses puppet00:19
bkeroYou could probably adapt the puppet modules00:19
anteayabut chef does have cookbooks for openstack service deployment00:20
anteayaI don't know as they have written any cookbooks for deploying our infra tools though00:20
anteayabut you can look00:20
anteayais the home of all our repos00:20
anteayaclarkb: thank you this is a really good talk:
asselin_pabelanger has some in ansible I believe.....00:21
anteayaI really enjoyed it00:21
anteayadims: I recommend you taking a look at and consider adding it to your blog post page00:21
anteayaI think it is a really well done talk00:21
kjorgensenthanks anteaya, yea I can look at the puppet code and try to adapt it00:22
anteayakjorgensen: you are welcome to ask questions as you go along00:22
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add test and update documentation for mailer plugin
*** dtantsur has joined #openstack-infra00:22
anteayamany folks are in the same boat as yourself, so have different thoughts on how to get you to where you want to go00:23
anteayakjorgensen: asselin_ has set up a number of ci systems, has helped a ton of folks to set up theirs and wrote the puppet-openstackci module and documentation00:23
kjorgensenyou were saying that you don't think my gearman server was running, but I think it is, because I can run 'telnet localhost 4730' and then get the 'status' from my gearman server00:25
anteayaI'm glad you confirmed00:25
anteayais your gearman worker running?00:26
*** jamesmca_ has joined #openstack-infra00:26
*** salv-orlando has joined #openstack-infra00:26
kjorgensenare you asking if my jenkins slave node shows up in the 'workers'?00:27
anteayaif you have an answer to that question, let's say that's the question00:27
asselin_I would like to know if you're using jenkins slaves or jenkins master00:28
*** madhu_ak has quit IRC00:28
anteayaasselin_: thanks for jumping in :)00:28
kjorgensenwell our setup has 1 jenkins master and 1 slave node00:29
kjorgensenmy test jenkins job is configured to execute only on the slave00:29
*** jamesmca_ has quit IRC00:29
asselin_ok, btw, there was a bug to run on jenkins master...but should have been fixed & released. Slaves works fine.00:29
kjorgensenso earlier the status looked like this00:30
kjorgensenbut now it is like this00:30
asselin_what's the full output of status?00:30
kjorgensenso the workers seem to have disappeared00:30
asselin_you should see the job name as part of the output00:30
kjorgensenright, I do see the job name in the full output00:31
asselin_can you paste it?00:31
dimsanteaya : done!00:32
anteayadims: awesome thanks!00:32
*** salv-orlando has quit IRC00:32
kjorgensenasselin_: our org has many jenkins jobs and they all show up in the output, but I am only interested in the one00:33
*** earlephilhower has quit IRC00:33
asselin_can you paste the one?00:33
openstackgerritMerged openstack-infra/project-config: Implement periodic job to update Puppet OpenStack constraints
kjorgensenit's what I mentioned before00:34
kjorgensen(I just changed from our company names to generic names)00:34
asselin_what's the job name and the slave name?00:34
asselin_i see, job name is 'job', and slave name is 'SLAVE'00:35
kjorgensenyea, basically00:35
asselin_so that 1 means you have  1 in the queue waiting to run, right?00:36
asselin_the last 0 means you don't have any slaves capabile of running that job00:36
asselin_is that slave offline?00:37
asselin_ok, toggle it offline, and bring it backonline and check the status again00:38
kjorgensenwe've seen this strange behavior as I've been testing, and to fix it we reset the connection between the jenkins master and the zuul server00:39
*** mriedem has quit IRC00:39
asselin_i don't recall ever needing to do that...00:39
asselin_also, do you have the zuul merger running?00:40
kjorgensenwhen you say "toggle" which service are you referring to?00:40
asselin_the jenkins slave00:40
*** akshai has joined #openstack-infra00:40
asselin_in the jenkins master UI, you can take the slave offline and bring it back online00:41
*** akshai_ has joined #openstack-infra00:42
kjorgensenah, unfortunately I don't have privileges to do that00:42
kjorgensenand yes, zuul-merger is running00:42
*** piet has joined #openstack-infra00:42
asselin_consider tyring this out and setup your own test environment:
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin
openstackgerritMadhumitha Viswanathan proposed openstack-infra/jenkins-job-builder: Add partial support for maven-release-plugin
kjorgensenasselin_: thank you! I will take a look at those instructions00:44
kjorgensenand see if I can apply it to our environment00:45
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin
*** akshai has quit IRC00:45
anteayaat the very least it gives you some freedom to figure out how the tools interact00:46
anteayaso you have a better understanding when you don't have permissions over the entire system00:46
*** mriedem has joined #openstack-infra00:46
*** kjorgensen has quit IRC00:46
anteayamriedem: I have to say nice work on the recaps00:47
anteayaand it is good to have in print that you are willing to address the nfv issues but lack any interest from anyone willing to do the work00:47
anteayaI'll be pointing folks to that post if it comes up in my earshot00:48
*** whoops has quit IRC00:48
*** sflanigan is now known as sflanigan_afk00:48
*** dgonzalez has quit IRC00:48
*** signed8bit_Zzz is now known as signed8bit00:49
*** whoops has joined #openstack-infra00:49
*** frickler has quit IRC00:49
*** tdasilva has quit IRC00:50
*** frickler has joined #openstack-infra00:50
*** dgonzalez has joined #openstack-infra00:50
*** salv-orlando has joined #openstack-infra00:51
*** mtanino has quit IRC00:52
*** tdasilva has joined #openstack-infra00:52
*** bpokorny_ has joined #openstack-infra00:53
*** bpokorny_ has quit IRC00:54
*** bpokorny_ has joined #openstack-infra00:55
*** kjorgensen has joined #openstack-infra00:55
*** salv-orlando has quit IRC00:56
*** bpokorny has quit IRC00:57
*** kjorgens_ has joined #openstack-infra00:58
*** kjorgensen has quit IRC00:59
*** bpokorny_ has quit IRC00:59
*** bpokorny has joined #openstack-infra01:00
*** piet has quit IRC01:01
*** kjorgens_ has quit IRC01:02
*** cody-somerville has joined #openstack-infra01:03
*** kzaitsev_mb has joined #openstack-infra01:03
*** gyee has quit IRC01:03
*** xarses has joined #openstack-infra01:04
*** kzaitsev_mb has quit IRC01:08
*** gongysh has joined #openstack-infra01:09
*** baoli_ has quit IRC01:09
*** baoli has joined #openstack-infra01:10
*** zhurong has joined #openstack-infra01:15
*** thorst_ has joined #openstack-infra01:20
*** salv-orlando has joined #openstack-infra01:22
*** thorst_ has quit IRC01:24
*** aeng has quit IRC01:25
*** aeng has joined #openstack-infra01:26
mriedemanteaya: thanks01:26
*** Jeffrey4l has joined #openstack-infra01:29
*** xyang1 has quit IRC01:29
*** salv-orlando has quit IRC01:30
*** chlong has joined #openstack-infra01:31
*** kengo_sakai has joined #openstack-infra01:31
*** ddieterly has quit IRC01:31
openstackgerritMatt Riedemann proposed openstack/requirements: Bump u-c oslo.versionedobjects to 1.9.0
*** yamahata has quit IRC01:43
*** cody-somerville has quit IRC01:47
*** bpokorny has quit IRC01:48
*** Goneri has joined #openstack-infra01:48
*** Swami has quit IRC01:49
*** banix has quit IRC01:51
*** jyuso has quit IRC01:52
*** bpokorny has joined #openstack-infra01:54
*** jcoufal has joined #openstack-infra01:57
*** salv-orlando has joined #openstack-infra01:57
*** baoli has quit IRC01:57
*** camunoz_ has quit IRC01:59
*** bpokorny has quit IRC02:00
*** savihou has quit IRC02:00
*** kjorgensen has joined #openstack-infra02:00
*** savihou has joined #openstack-infra02:01
*** jcoufal has quit IRC02:03
*** kzaitsev_mb has joined #openstack-infra02:04
*** anteaya has quit IRC02:04
*** kjorgensen has quit IRC02:04
*** sflanigan_afk is now known as sflanigan02:04
*** hdd has quit IRC02:05
*** woodster_ has quit IRC02:08
*** rfolco has quit IRC02:08
*** kzaitsev_mb has quit IRC02:09
*** tonytan4ever has quit IRC02:09
*** salv-orlando has quit IRC02:09
openstackgerritSteve Martinelli proposed openstack-infra/elastic-recheck: add query for bug 1578866
openstackbug 1578866 in OpenStack Identity (keystone) "test_user_update_own_password failing intermittently" [Undecided,New]
*** Volundr has joined #openstack-infra02:10
*** kjorgensen has joined #openstack-infra02:11
*** whoops has quit IRC02:11
*** shashank_hegde has quit IRC02:14
*** apoorvad has quit IRC02:16
*** kjorgensen has quit IRC02:17
*** piet has joined #openstack-infra02:20
*** vhosakot has quit IRC02:25
*** vhosakot has joined #openstack-infra02:31
*** newbie|2 has joined #openstack-infra02:32
*** vhosakot has quit IRC02:33
openstackgerritMerged openstack-infra/elastic-recheck: add query for bug 1578866
openstackbug 1578866 in OpenStack Identity (keystone) "test_user_update_own_password failing intermittently" [High,Confirmed]
*** mriedem has quit IRC02:36
*** jyuso1 has joined #openstack-infra02:36
*** piet has quit IRC02:37
*** kjorgensen has joined #openstack-infra02:38
*** kjorgensen has quit IRC02:42
*** vhosakot has joined #openstack-infra02:47
*** kjorgensen has joined #openstack-infra02:53
*** piet has joined #openstack-infra02:56
*** ccamacho has quit IRC02:57
*** ccamacho has joined #openstack-infra02:58
*** kjorgensen has quit IRC02:58
*** binbincong has quit IRC03:04
*** daemontool_ has quit IRC03:04
*** kjorgensen has joined #openstack-infra03:05
*** kjorgensen has quit IRC03:09
*** maishsk has quit IRC03:10
*** kjorgensen has joined #openstack-infra03:13
*** piet has quit IRC03:13
*** akshai_ has quit IRC03:15
*** tlian has quit IRC03:16
*** kjorgensen has quit IRC03:17
*** maishsk has joined #openstack-infra03:20
*** markvoelker has quit IRC03:20
*** salv-orlando has joined #openstack-infra03:21
*** hdd has joined #openstack-infra03:24
*** kjorgensen has joined #openstack-infra03:28
*** binbincong has joined #openstack-infra03:30
*** salv-orlando has quit IRC03:30
*** kjorgensen has quit IRC03:32
*** julim has quit IRC03:33
*** amit213 has quit IRC03:33
*** julim has joined #openstack-infra03:33
*** amit213 has joined #openstack-infra03:33
*** signed8bit has quit IRC03:36
*** asettle has quit IRC03:42
*** asettle has joined #openstack-infra03:42
*** salv-orlando has joined #openstack-infra03:44
*** kjorgensen has joined #openstack-infra03:45
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Get translation jobs into healthcheck
*** salv-orlando has quit IRC03:46
*** asettle has quit IRC03:47
openstackgerritMerged openstack-infra/irc-meetings: Remove the API WG 0000 UTC meeting
*** kjorgensen has quit IRC03:50
*** sflanigan has quit IRC03:50
*** sflanigan has joined #openstack-infra03:51
openstackgerritMerged openstack-infra/puppet-kibana: Remove support for Kibana2
*** esker has quit IRC03:55
*** DevBox has joined #openstack-infra03:55
AJaegerproject-config cores, is now ready - retiring of two docs repos. Could you review this, please?03:55
*** kjorgensen has joined #openstack-infra03:57
*** DevBox has quit IRC03:58
openstackgerritMerged openstack-infra/puppet-meetbot: Indentation of the parameters is refactored
*** DevBox has joined #openstack-infra04:01
*** links has joined #openstack-infra04:02
*** kjorgensen has quit IRC04:02
*** kzaitsev_mb has joined #openstack-infra04:05
*** hamzy has joined #openstack-infra04:06
*** kzaitsev_mb has quit IRC04:11
*** zz_dimtruck is now known as dimtruck04:12
openstackgerritMerged openstack-infra/irc-meetings: Adding a meeting for Operators Telco and NFV Working Group
*** shashank_hegde has joined #openstack-infra04:20
*** markvoelker has joined #openstack-infra04:21
*** piet has joined #openstack-infra04:21
*** markvoelker has quit IRC04:26
*** salv-orlando has joined #openstack-infra04:28
*** psachin has joined #openstack-infra04:30
*** salv-orlando has quit IRC04:32
*** maishsk has quit IRC04:33
*** Nikolay_St has quit IRC04:34
*** maishsk has joined #openstack-infra04:35
*** hdd has quit IRC04:39
*** maishsk has quit IRC04:41
*** maishsk has joined #openstack-infra04:43
*** dimtruck is now known as zz_dimtruck04:46
*** asettle has joined #openstack-infra04:47
*** cody-somerville has joined #openstack-infra04:54
*** flwang1 has quit IRC04:54
*** kzaitsev_mb has joined #openstack-infra04:56
*** kengo_sa_ has joined #openstack-infra05:00
*** salv-orlando has joined #openstack-infra05:01
*** fedexo has joined #openstack-infra05:02
*** shashank_hegde has quit IRC05:04
*** salv-orlando has quit IRC05:04
*** oanson has joined #openstack-infra05:14
*** salv-orlando has joined #openstack-infra05:17
*** shashank_hegde has joined #openstack-infra05:21
*** markvoelker has joined #openstack-infra05:22
*** salv-orlando has quit IRC05:24
*** kengo_sa_ has quit IRC05:25
*** markvoelker has quit IRC05:26
*** piet has quit IRC05:27
*** netsin has quit IRC05:30
*** netsin has joined #openstack-infra05:31
*** vhosakot has quit IRC05:32
*** Nikolay_St has joined #openstack-infra05:32
*** ianychoi has joined #openstack-infra05:32
*** salv-orlando has joined #openstack-infra05:35
*** jerryz has quit IRC05:36
*** salv-orlando has quit IRC05:39
*** gildub has joined #openstack-infra05:44
*** kzaitsev_mb has quit IRC05:52
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Fix validation for stable branch
*** yolanda has joined #openstack-infra05:55
*** mikelk has joined #openstack-infra05:56
*** sridhar_ram has joined #openstack-infra05:57
*** sridhar_ram has quit IRC06:04
*** shashank_hegde has quit IRC06:07
*** shashank_hegde has joined #openstack-infra06:14
*** jkraj has joined #openstack-infra06:17
*** kengo_sa_ has joined #openstack-infra06:19
*** javeriak has joined #openstack-infra06:20
*** zul has quit IRC06:21
*** kengo_sakai has quit IRC06:22
*** markvoelker has joined #openstack-infra06:23
*** itsuugo has quit IRC06:25
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** markvoelker has quit IRC06:27
*** asettle has quit IRC06:28
*** fawadkhaliq has joined #openstack-infra06:31
openstackgerritMerged openstack-infra/project-config: Add experimental-project-config-layout for ubuntu-precise
AJaegerI'd like an infra-root to cleanup and . Please delete the *.pot files in the directory itself. You can also delete the complete directories, the content is updated after each merge and these files are placed in the wrong location.06:34
*** fedexo has quit IRC06:36
*** e0ne has joined #openstack-infra06:39
*** salv-orlando has joined #openstack-infra06:46
*** javeriak has quit IRC06:46
*** yolanda has quit IRC06:46
*** yolanda has joined #openstack-infra06:47
yolandaAJaeger, done06:47
AJaegerthanks, yolanda ! and good morning!06:49
*** yaume has joined #openstack-infra06:49
AJaegerNow we're ready for - I'll remove WIP now...06:50
*** javeriak has joined #openstack-infra06:50
yolandagood morning06:50
*** yaume has quit IRC06:50
*** yaume has joined #openstack-infra06:50
*** itsuugo has joined #openstack-infra06:50
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Retire operations-guide and ha-guide (2/2)
*** e0ne has quit IRC06:53
AJaegeryolanda: could you review the project-config reviewing guidelines again, please?
AJaegerand is there a meeting going on right now? If not, could you +A , please?06:55
*** salv-orlando has quit IRC06:57
openstackgerritMerged openstack-infra/project-config: Add experimental centos-7 jobs to check-tripleo-jobs
*** e0ne has joined #openstack-infra06:59
*** e0ne has quit IRC07:00
*** yolanda has quit IRC07:02
openstackgerritMerged openstack-infra/project-config: stop puppet-unit 3.3 and 3.4 jobs starting from newton
*** permalac has joined #openstack-infra07:06
openstackgerritMerged openstack-infra/system-config: Remove statusbot from meeting channels
*** yolanda has joined #openstack-infra07:06
*** yottatsa has joined #openstack-infra07:07
*** shashank_hegde has quit IRC07:08
*** fawadkhaliq has quit IRC07:14
openstackgerritMerged openstack-infra/project-config: Set image build in magnum to periodic
openstackgerritMerged openstack-infra/project-config: use 24hours everywhere in grafana/neutron.yaml
openstackgerritMerged openstack-infra/project-config: Retire operations-guide and ha-guide (1/2)
*** e0ne has joined #openstack-infra07:16
*** pahuang has quit IRC07:16
*** ifarkas has joined #openstack-infra07:21
*** yolanda has quit IRC07:21
*** arxcruz has joined #openstack-infra07:21
openstackgerritMerged openstack-infra/project-config: Set TEMPEST_CONCURRENCY to 1 for grenade-ironic
*** zhurong_ has joined #openstack-infra07:23
*** markvoelker has joined #openstack-infra07:23
*** nadya has joined #openstack-infra07:24
*** oanson has quit IRC07:24
*** zhurong has quit IRC07:26
*** esikachev has joined #openstack-infra07:26
*** salv-orlando has joined #openstack-infra07:26
openstackgerritMerged openstack-infra/project-config: Make gate-murano-python34-db voting; add gate job
*** markvoelker has quit IRC07:28
*** markus_z has joined #openstack-infra07:29
*** yottatsa has quit IRC07:31
*** zhurong_ has quit IRC07:31
*** yolanda has joined #openstack-infra07:32
openstackgerritMerged openstack-infra/project-config: Use m1.large for centos-7 on tripleo-test-cloud-rh1
*** zhurong has joined #openstack-infra07:32
*** fawadkhaliq has joined #openstack-infra07:33
AJaegeryolanda: could you put on your review queue as well, please? Thanks!07:35
openstackgerritSylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project
*** oanson has joined #openstack-infra07:36
openstackgerritMerged openstack-infra/project-config: Integrate shade+magnum neutron job
*** openstackstatus has quit IRC07:38
*** openstackstatus has joined #openstack-infra07:39
*** ChanServ sets mode: +v openstackstatus07:39
*** hdaniel has joined #openstack-infra07:39
*** cody-somerville has quit IRC07:40
*** salv-orlando has quit IRC07:40
yolandaAJaeger, sure07:40
AJaegerThanks, yolanda ! I'd like to wrap up loose ends before my vacation starts in a week...07:42
yolandaoh, vacation ... enjoy :)07:43
AJaegerlooking forward to it ;) But first another week...07:45
openstackgerritSylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project
*** nadya has quit IRC07:46
openstackgerritvenkatamahesh proposed openstack/requirements: Remove kolla-mesos from projects.txt
*** jlanoux has joined #openstack-infra07:47
*** kzaitsev_mb has joined #openstack-infra07:47
*** hdaniel has quit IRC07:49
*** cody-somerville has joined #openstack-infra07:52
*** daemontool has joined #openstack-infra07:53
*** chlong has quit IRC07:54
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Retire operations-guide and ha-guide (2/2)
AJaegeris the proposal slave offline? I don't see it moving forward any translation jobs...07:58
*** zzzeek has quit IRC08:00
*** gildub has quit IRC08:01
*** nadya has joined #openstack-infra08:02
*** zzzeek has joined #openstack-infra08:02
*** edwarnicke_ has joined #openstack-infra08:04
*** edwarnicke has quit IRC08:07
*** edwarnicke_ is now known as edwarnicke08:07
*** e0ne has quit IRC08:07
*** andrewbogott_ has joined #openstack-infra08:08
yolandaAJaeger, checking08:08
*** dmk0202 has joined #openstack-infra08:08
*** pilgrimstack1 has joined #openstack-infra08:09
*** pilgrimstack has quit IRC08:09
yolandaAJaeger, it is08:09
*** andrewbogott has quit IRC08:09
*** andrewbogott_ is now known as andrewbogott08:09
yolandathere may be some job stopping it08:09
*** BobBall_AWOL is now known as BobBall08:11
*** lucas-afk is now known as lucasagomes08:15
*** salv-orlando has joined #openstack-infra08:19
yolandaAJaeger, i'm checking the latest jobs now08:19
*** e0ne has joined #openstack-infra08:20
yolandalooks as the puppet constraints job set proposal offline08:21
*** kushal has joined #openstack-infra08:23
*** mhickey has joined #openstack-infra08:23
AJaegerargh ;(08:23
AJaegerok, will check and sent a patch...08:23
AJaegerthanks , yolanda08:24
*** salv-orlando has quit IRC08:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Setup propose-puppet-openstack-constraints properly
AJaegeryolanda: is that regex fine? ^08:27
*** sflanigan has quit IRC08:27
*** shardy has joined #openstack-infra08:28
yolandaAJaeger, going to double check...08:28
AJaegerplease do ;)08:28
yolandalooks good08:29
*** jaosorior has joined #openstack-infra08:32
*** jistr has joined #openstack-infra08:32
*** jyuso1 has quit IRC08:33
*** javeriak has quit IRC08:33
*** jyuso1 has joined #openstack-infra08:34
*** pcaruana has joined #openstack-infra08:36
*** dizquierdo has joined #openstack-infra08:37
*** derekh has joined #openstack-infra08:38
*** e0ne has quit IRC08:40
*** jang has joined #openstack-infra08:40
openstackgerritMerged openstack-infra/project-config: Setup propose-puppet-openstack-constraints properly
*** gongysh has quit IRC08:40
*** rossella_s has joined #openstack-infra08:41
*** binbincong has quit IRC08:41
*** binbincong has joined #openstack-infra08:42
*** e0ne has joined #openstack-infra08:42
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Fix validation for stable branch
*** salv-orlando has joined #openstack-infra08:44
*** gongysh has joined #openstack-infra08:44
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add VirtualBMC to gerritbot
*** salv-orlando has quit IRC08:48
*** zeih has joined #openstack-infra08:53
*** rossella_s has quit IRC08:54
*** fawadkhaliq has quit IRC08:54
*** rossella_s has joined #openstack-infra08:54
*** ihrachys has joined #openstack-infra08:54
*** jordanP has joined #openstack-infra08:55
*** tesseract has joined #openstack-infra08:56
*** tesseract is now known as Guest4293008:56
*** gongysh has quit IRC08:59
*** alex_xu has quit IRC09:03
*** nadya has quit IRC09:04
*** alex_xu has joined #openstack-infra09:05
*** javeriak has joined #openstack-infra09:06
*** electrofelix has joined #openstack-infra09:07
buntinglifeless: I was wondering if upper constraints worked on the clients?09:11
*** cody-somerville has quit IRC09:12
*** zeih has quit IRC09:12
*** sambetts|afk is now known as sambetts09:14
*** javeriak has quit IRC09:20
*** dizquierdo has quit IRC09:30
*** javeriak has joined #openstack-infra09:31
*** chem has joined #openstack-infra09:33
*** asettle has joined #openstack-infra09:35
*** yottatsa has joined #openstack-infra09:36
*** jyuso1 has quit IRC09:38
*** asettle has quit IRC09:39
*** salv-orlando has joined #openstack-infra09:41
*** maishsk has quit IRC09:44
*** maishsk_ has joined #openstack-infra09:44
*** zhurong has quit IRC09:46
*** mhickey has quit IRC09:47
*** tosky has joined #openstack-infra09:47
openstackgerritSagi Shnaidman proposed openstack-infra/system-config: Refresh log files for tripleo project
openstackgerrityolanda.robla proposed openstack-infra/project-config: Move ansible-dev magnum shade to non-voting
*** salv-orlando has quit IRC09:49
*** gildub has joined #openstack-infra09:49
*** zhurong has joined #openstack-infra09:51
*** pfallenop has quit IRC09:53
*** jkraj has quit IRC09:57
*** sshnaidm has quit IRC09:57
*** jaosorior is now known as jaosorior_lunch09:59
*** pfallenop has joined #openstack-infra10:00
*** mhickey has joined #openstack-infra10:02
*** ihrachys has quit IRC10:04
*** zhurong has quit IRC10:06
*** maishsk_ has quit IRC10:08
*** ekarlso has quit IRC10:08
*** zhurong has joined #openstack-infra10:08
kozhukalovteam, could you please merge this ?10:10
openstackgerritAndrea Frittoli proposed openstack-infra/project-config: Add gate-tempest-dsvm-neutron-full-ssh to periodic
openstackgerrityolanda.robla proposed openstack-infra/shade: Add bay certificates to shade
*** zhurong has quit IRC10:11
ZaraHi, I'm in the process of patching to add my credentials; does realname map to a name used anywhere else in the system? I'm asking as I'm often just 'Zara' sans surname.10:12
nibalizerZara: totally fine10:13
*** zhurong has joined #openstack-infra10:13
*** zhurong has quit IRC10:13
rcarrillocruzit's too early for you nibalizer , where are thou?10:14
yolandakozhukalov, approved. The depends-on already landed so is fine now10:14
nibalizerrcarrillocruz: :)10:14
nibalizerstill in portland10:14
Zaranibalizer: so it's okay for me to use my surname or not as I wish? :)10:14
rcarrillocruzbtw nibalizer10:16
rcarrillocruzinitial designate ansible goodness coming10:16
*** electrofelix has quit IRC10:20
*** dizquierdo has joined #openstack-infra10:20
*** dims has quit IRC10:20
*** chuck__ has joined #openstack-infra10:25
*** markvoelker has joined #openstack-infra10:26
*** sdague has joined #openstack-infra10:28
*** dizquierdo has quit IRC10:29
*** markvoelker has quit IRC10:30
*** electrofelix has joined #openstack-infra10:32
*** kjorgensen has joined #openstack-infra10:33
*** alaski is now known as lascii10:36
*** maishsk has joined #openstack-infra10:36
*** kjorgensen has quit IRC10:37
*** dizquierdo has joined #openstack-infra10:44
*** maishsk has quit IRC10:46
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Decouple Shotgun project from Fuel
openstackgerritVladimir Kozhukalov proposed openstack-infra/project-config: Decouple Shotgun project from Fuel
*** pavel_bondar has joined #openstack-infra10:51
*** Goneri has quit IRC10:52
*** nadya has joined #openstack-infra10:55
*** dmk0202 has quit IRC10:57
*** dmk0202 has joined #openstack-infra10:57
*** pavel_bondar has quit IRC10:57
*** oanson has quit IRC10:57
*** pavel_bondar has joined #openstack-infra10:58
*** Hal has joined #openstack-infra10:58
*** Hal is now known as Guest9869010:59
*** asettle has joined #openstack-infra10:59
*** asettle has quit IRC10:59
*** jtomasek has quit IRC10:59
openstackgerrityolanda.robla proposed openstack-infra/shade: Add bay certificates to shade
*** kushal has quit IRC11:00
*** dims has joined #openstack-infra11:02
*** maishsk has joined #openstack-infra11:03
openstackgerritVitaly Gridnev proposed openstack/requirements: [sahara] remove sahara-tests from projects.txt
*** chuck__ has quit IRC11:06
*** mhickey has quit IRC11:07
*** esikachev has quit IRC11:07
*** yamamoto has quit IRC11:08
*** oanson has joined #openstack-infra11:10
sylwesterBFolks, could you check it out? if everything is done?11:11
*** jaosorior_lunch has quit IRC11:11
*** jaosorior_lunch has joined #openstack-infra11:12
*** jaosorior_lunch is now known as jaosorior11:12
*** javeriak has quit IRC11:19
*** mhickey has joined #openstack-infra11:20
*** Na3iL has joined #openstack-infra11:20
*** maishsk has quit IRC11:21
*** maishsk has joined #openstack-infra11:24
*** dizquierdo has quit IRC11:25
ansiwenAJaeger: how long does it take until the requirements land in nove? any idea?11:25
*** esikachev has joined #openstack-infra11:26
*** markvoelker has joined #openstack-infra11:27
openstackgerritZara proposed openstack-infra/system-config: Add Zara to users
Zarawell, that took ages; got distracted by email11:27
*** kushal has joined #openstack-infra11:27
AJaegeransiwen: let's ask dhellmann or sdague to review 311761 as first step11:27
*** chuck__ has joined #openstack-infra11:28
*** kengo_sakai has joined #openstack-infra11:28
*** javeriak has joined #openstack-infra11:28
*** ccamacho is now known as ccamacho|lunch11:30
sdagueAJaeger: +A on the requirements review11:31
*** markvoelker has quit IRC11:31
openstackgerritJianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool
openstackgerritJianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool
*** kengo_sakai has quit IRC11:35
AJaegerthanks, sdague ! ansiwen, once that one is in,  a change against nova will be proposed and than one needs to merge.. - and then needs rebasing and can merge...11:36
openstackgerritJianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool
sdagueAJaeger: 311753 can also change tox.ini as well, right?11:37
AJaegersdague: what should be changed?11:38
AJaegerWe use the venv environment for the translations11:38
AJaegersdague: speaking about translations: does not store .pot files anymore in repos, if you want to review it...11:39
AJaegeryeah, that could be removed - it's done by infra scripts now11:39
AJaegeransiwen: ^11:40
ansiwenAJaeger: btw, how are the po files generated from the pot files?11:40
*** tbachman has left #openstack-infra11:40
sdagueAJaeger: right, but we put that into the tox.ini so that we'd get end user fail11:40
*** alex_xu has quit IRC11:42
*** jang has quit IRC11:42
AJaegeransiwen: our scripts generate everything and push to repos - there shouldn't be a need to have pot files in there.11:42
*** baoli has joined #openstack-infra11:42
*** dizquierdo has joined #openstack-infra11:42
AJaegeransiwen: po files are done on translation server11:42
*** DevBox has quit IRC11:42
*** tlian has joined #openstack-infra11:42
AJaegersdague: we only recently put those tox.ini lines into the infra scripts11:43
ansiwenAJaeger: that was my question... since also doesn't support several domains for updateing po files from the pot files, I assumed the po files are generated elsewhere11:43
*** yolanda has quit IRC11:43
AJaegeransiwen: we generate pot files in CI, push to zanata after each merge - and once a day check for translated pot files in zanata and download them and propose to projects, see,n,z for current proposals11:44
*** ldnunes has joined #openstack-infra11:44
ansiwenok, I see...11:45
*** penguinolog has quit IRC11:46
*** maishsk has quit IRC11:46
ansiwenAJaeger: because I realized I'm not very happy with my babel change, because it's not consistent. compile_catalogs (po->mo) supports several domains now, but init_catalogs and update_catalogs (pot->po) doesn't support several domains. feels a bit ugly.11:46
*** zhurong has joined #openstack-infra11:47
AJaegerprojects don't need init_catalogs and update_catalogs in the OpenStack CI environment11:47
ansiwenfor openstack it doesn't matter, because update_catalogs is not used11:47
*** alex_xu has joined #openstack-infra11:47
amrithis something the matter with zuul? the web UI appears to be unresponsive (
*** baoli_ has joined #openstack-infra11:47
AJaegeransiwen: exactly. But if it's used outside, it's indeed different.11:47
ansiwenAJaeger: right, but it's ugly in general11:48
*** yolanda has joined #openstack-infra11:48
*** baoli has quit IRC11:50
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/shade: Amend the valid fields to update on recordsets
*** yamamoto has joined #openstack-infra11:56
*** salv-orlando has joined #openstack-infra11:58
*** psilvad has joined #openstack-infra11:58
*** yamamoto has quit IRC12:03
*** yamamoto has joined #openstack-infra12:05
*** rfolco has joined #openstack-infra12:07
*** psilvad has quit IRC12:07
*** asettle has joined #openstack-infra12:07
*** rhallisey has joined #openstack-infra12:07
*** aysyd has joined #openstack-infra12:09
*** lucasagomes is now known as lucas-hungry12:11
*** dprince has joined #openstack-infra12:11
*** dingyichen has quit IRC12:12
*** markvoelker has joined #openstack-infra12:14
*** maestro has joined #openstack-infra12:16
*** yamamoto has quit IRC12:17
*** Jeffrey4l has quit IRC12:17
*** trown|outtypewww is now known as trown12:17
*** yamamoto has joined #openstack-infra12:18
*** psilvad has joined #openstack-infra12:18
*** ekarlso has joined #openstack-infra12:18
*** kgiusti has joined #openstack-infra12:19
*** ddieterly has joined #openstack-infra12:19
*** yamamoto has quit IRC12:19
*** yamamoto has joined #openstack-infra12:20
*** ddieterly is now known as ddieterly[away]12:20
AJaegermorning, pabelanger !12:20
*** gordc has joined #openstack-infra12:21
openstackgerritAnant Patil proposed openstack-infra/project-config: Add experimental magnum job to heat
*** alex_xu has quit IRC12:22
*** binbincong has quit IRC12:23
*** binbincong has joined #openstack-infra12:24
*** alex_xu has joined #openstack-infra12:25
openstackgerritKiall Mac Innes proposed openstack-infra/project-config: Add experimental job for Designate's knot2 agent backend
pabelangerSo, I think somebody patch bombed review.o.o12:27
pabelangeropenstack/monasca-analytics seems to have uploaded 175 patches, in a row12:27
*** salv-orlando has quit IRC12:28
*** oanson has quit IRC12:28
*** Jeffrey4l has joined #openstack-infra12:28
AJaegerI just checked status.o.o/zuul and was surprised to see a long stack at the top12:28
pabelangerYa, zuul is doing its best to handle it12:28
pabelangerbut status.o.o/zuul is slow12:29
AJaeger - 177 related changes. 177!12:29
*** amrith is now known as _amrith_12:29
*** rodrigods has quit IRC12:29
*** rodrigods has joined #openstack-infra12:29
openstackgerritAndrea Frittoli proposed openstack-infra/project-config: Add gate-tempest-dsvm-neutron-full-ssh to periodic
AJaegerthat series looks really wrong, looking at titles12:29
*** yamamoto has quit IRC12:29
AJaeger"Fix test and addressed comments." or "Fix pep8 new checks."12:30
*** psilvad has quit IRC12:30
AJaegerA lot should be squeezed together...12:30
StevenKwin 2212:30 is the committer12:31
AJaegerwho will send him a link to the Developer's Guide? ;)12:31
* AJaeger has coffee break now...12:31
*** psilvad has joined #openstack-infra12:31
*** regXboi has joined #openstack-infra12:32
*** regXboi has quit IRC12:33
pabelangerYa, zuul is trying to noop them all12:33
pabelangerwhich is just taking time12:33
*** regXboi has joined #openstack-infra12:33
pabelangerthat explains the lack of movement on nodepool.o.o too12:33
*** kushal has quit IRC12:34
*** yamamoto has joined #openstack-infra12:35
*** kengo_sa_ has quit IRC12:35
*** lhinds has joined #openstack-infra12:35
*** signed8bit has joined #openstack-infra12:36
*** kengo_sakai has joined #openstack-infra12:36
lhindshello all, where would be the right place to report an issue with the wiki - launchpad OpenID integration?12:36
lhindsI keep getting 'Verification of the OpenID URL failed. Error message: "Nonce already used or out of range"'12:37
lhindsyet it works fine for gerrit12:37
pabelangerlhinds: you are in the correct spot.12:38
*** xyang1 has joined #openstack-infra12:38
pabelangerlhinds: what is your wiki.o.o username?12:38
lhindslhinds: wiki is lhinds12:38
*** savihou has quit IRC12:39
pabelangerlhinds: There is no user by the name "lhinds". Check your spelling.12:40
*** Guest98690 has quit IRC12:40
pabelangerhave you logged into the wiki before?12:40
lhindspabelanger, good question, I don't think I have12:40
lhindsbeen trying via launchpad12:40
pabelangerya, so that is the issue. We've disabled new user creation on wiki.o.o12:40
pabelangerbecause of spamming issues12:40
pabelangerlet me see if I can manually create your account12:41
lhindsthanks pabelanger12:41
*** kushal has joined #openstack-infra12:41
lhinds Full name: Luke Hinds12:41
lhinds Username: lhinds12:41
lhinds Email address: lhinds@redhat.com12:41
*** kengo_sakai has quit IRC12:41
*** mat128 is now known as mat128|afk12:42
*** Goneri has joined #openstack-infra12:44
*** Na3iL has quit IRC12:46
*** ddieterly[away] is now known as ddieterly12:46
Nikolay_StAJaeger: fungi can you help me with fail of this patch: ? fuel-web guys said that it can be infrastructure issue12:47
*** tpsilva has joined #openstack-infra12:47
*** javeriak has quit IRC12:47
pabelangerlhinds: Hmm, not sure I can manually create it. Let me talk with fungi to see what we need to do.  But keep in mind, we are actually deprecating the wiki so depending on the content, there may be a better place to place it today12:48
*** yamamoto has quit IRC12:48
*** ayoung has quit IRC12:48
lhindsthanks pabelanger - I need to put up a security note track number, I can see if someone else can make the edit for me.12:48
*** asettle has quit IRC12:49
lhindsplease do check with fungi as well still12:49
*** |-paul-| has joined #openstack-infra12:49
pabelangerlhinds: okay, ya. It is not the ideal thing, but likely solves the current issue.  We'll ping you once we figure it out12:49
AJaegerNikolay_St: I see a failing request - no idea where that comes from. You need to dig deeper and debug this yourself12:49
*** Na3iL has joined #openstack-infra12:50
*** yamamoto has joined #openstack-infra12:50
*** gildub has quit IRC12:50
AJaegerpabelanger: did you email the monasca person? Otherwise I'll do it now...12:50
pabelangerAJaeger: fungi: So, I wonder if we need to dump the check queue for openstack/monasca-analytics or just let zuul continue to process the changes.  At this rate is it going to be a few hours before zuul clears out12:51
pabelangerAJaeger: I have not12:51
pabelangerAJaeger: I was actually looking for their IRC handel12:51
*** hrybacki|gone is now known as hrybacki12:51
AJaegerIt's #openstack-monasca12:52
pabelangerokay, heading there12:52
AJaegerGood idea, let's ask there - on my way...12:52
*** ccamacho|lunch is now known as ccamacho12:52
*** akshai has joined #openstack-infra12:53
*** mat128|afk is now known as mat12812:55
pabelangerAJaeger: fungi: Ah, I keep forgetting, the only way to dump a queue is to restart zuul12:55
*** jheroux has joined #openstack-infra12:55
*** ddieterly has quit IRC12:56
*** mhickey has quit IRC12:57
*** thiagop-afk has joined #openstack-infra12:58
*** thiagop-afk is now known as thiagop12:58
*** sdake has joined #openstack-infra13:00
*** ifarkas has quit IRC13:01
*** nadya has quit IRC13:01
*** jaypipes has joined #openstack-infra13:02
AJaegerpabelanger: back to normal business ;)13:02
AJaegerpabelanger: seems that other channel is just idle ;(13:02
*** yolanda has quit IRC13:02
*** matt-borland has joined #openstack-infra13:02
*** yottatsa has quit IRC13:02
*** amitgandhinz has joined #openstack-infra13:04
*** davidSubirosPere has joined #openstack-infra13:04
AJaegerLet's see whether we get any reaction...13:05
*** anteaya has joined #openstack-infra13:06
davidSubirosPerehi, I'm David, I was asked to join this IRC channel to talk about some code I pushed that contains too many commits13:06
*** psachin has quit IRC13:06
*** piet has joined #openstack-infra13:06
anteayahi davidSubirosPere can you share the url of the gerrit change?13:06
*** yolanda has joined #openstack-infra13:07
davidSubirosPereI work in Hewlett Packard Enterprise, and we are releasing some code as open source, I wanted to keep the git history, but it seems there are too many commits13:08
*** yottatsa has joined #openstack-infra13:08
anteayapabelanger: ^^ regarding your email13:08
AJaegerdavidSubirosPere: The proper way to keep history is to import it at repo setup time, not afterwards.13:09
AJaegerNow somebody has to manually merge 178 changes via the webui ;(13:09
AJaegerdavidSubirosPere: thanks for showing up!13:09
mescanefhello infra team:-) please review some puppet code: and
*** jsavak has joined #openstack-infra13:12
*** yamamoto has quit IRC13:13
AJaegerdavidSubirosPere: by accident you did a denial-of-service attack on Zuul - the many changes with the long dependencies of these changes keep our CI really busy...13:13
*** |-paul-| has quit IRC13:13
EmilienMpabelanger: o/13:14
davidSubirosPeresorry about that :( is there a way to stop the jobs?13:14
AJaegerdavidSubirosPere: Not that I'm aware of - just don't resend the whole series, please!13:14
*** savihou has joined #openstack-infra13:15
AJaegerwe currently have 11050 changes in the check queue - new record ;)
dhellmannAJaeger : if he abandons the first patch in the series would that kill all of the jobs behind it? and then we could import the history some other way?13:15
davidSubirosPereok, I thought only a single job would be generated, from now on I will not send so many commits :)13:15
*** yamamoto has joined #openstack-infra13:15
AJaegerdhellmann: no idea13:16
Kialldhellmann: updating the first one on the chain will cause the others to be aborted13:16
Kialleven a commit message update would do it13:16
dhellmannKiall : and they wouldn't just be restarted?13:16
AJaegerdavidSubirosPere: We test each patch on its own. So, if you submit two changes, we test the first one on its own, then the second with the first one applied...13:16
Kialldhellmann: Correct, they won't be restarted13:16
AJaegerKiall: if you edit the commit message, you restart. Abort might work - no idea13:16
Kiall(not without a recheck)13:16
KiallWell - the first patch will restart13:17
Kiallbut the 176 dependant's will not13:17
*** mtanino has joined #openstack-infra13:17
AJaegerKiall: yeah, you might be right since they now would all be not dependent - so merge conflcits basically13:17
dhellmanncool, so that might be a thing to do -- edit the message using the gerrit ui, then wait for the other jobs to die and abandon those patches13:17
AJaegerfungi, are you around to help?13:17
davidSubirosPereshall I abandon the change?13:17
dhellmanndavidSubirosPere : don't do anything yet, let's make sure we have consensus on what will happen :-)13:18
*** yamamoto has quit IRC13:18
AJaegerdavidSubirosPere: ++13:18
*** lucas-hungry is now known as lucasagomes13:18
pabelangeranteaya: davidSubirosPere: Thanks for joining13:18
AJaegerI agree with dhellmann, let's spend some time on figuring out best solution13:18
*** piet has quit IRC13:19
*** yamamoto has joined #openstack-infra13:19
anteayaI'm not buying changing the first commit message aborts jobs on the other patches13:19
pabelangerif we could go back in time, we could just import the project with jeepby but not sure that will work13:19
pabelangercouldn't a infra-root do a force push at this point?13:19
*** piet has joined #openstack-infra13:20
dhellmannfirst priority should be unclogging the gate, then we can figure out the import13:20
dhellmannunless you think a force push would somehow unclog the gate?13:20
*** mriedem has joined #openstack-infra13:20
anteayaI only see 32 patches in the gate13:20
dhellmannanteaya : sorry, "check queue"13:21
*** yamamoto has quit IRC13:21
anteayaokay with you now13:21
AJaegeranteaya: we speak about check queue - 10k jobs13:21
sdagueAJaeger: but they are all noop13:21
anteayahas zuul stopped working?13:21
AJaegerI mean: 10k changes...13:21
pabelangerI think Kiall is right, if we update the first patch in the stack, zuul should know that and kill off the other changes in the check queue13:21
anteayapabelanger: I'm not convincing13:21
AJaegersdague: Yes, but Zuul still needs to process them and that seems to take exponential time ;(13:21
dhellmannpabelanger : the question is whether they die, or get restarted13:22
anteayaI think they will all get restarted13:22
sdaguepabelanger: I'm not super convinced of that, they may just get restarted13:22
dhellmannmaybe someone can write an abandon script?13:22
* AJaeger doesn't know - and Zuul is too busy to test anything ;/13:22
pabelangerdhellmann: sdague: right, so we need a merge conflict between patch 1 or patch 2.13:22
sdaguewhat happens if these get approved13:22
Kiallanteaya: I'm 99.9% sure they will not be restarted.13:22
KiallUpload 2 dependant changes somewhere and try it..13:22
sdagueKiall: you can't try anything right now13:23
pabelangerwe can also dump both check and gate queue, restart zuul, then load gate again13:23
*** yamamoto has joined #openstack-infra13:23
sdaguebecause it's all jammed up13:23
pabelangerasking people in check to recheck their patches13:23
jrollchange the first patch to be 'git rm -rf *' ? that'll merge conflict for sure13:23
anteayapabelanger: yeah I think that is the way to go13:23
KiallI can confirm that behaviour on our internal Gerrit/Zuul?13:23
anteayasave the gate queue and clear the check queue13:23
*** banix has joined #openstack-infra13:24
anteayadavidSubirosPere: and don't do anything with your patch series until we discuss a plan please13:24
sdaguezuul is definitely processing these though, I do kind of wonder if it's better to just script approve them13:24
anteayaI'm only seeing 250 in check btw13:24
sdagueit will take some time to chug through it13:24
anteayaaccording to
*** rhochmuth has joined #openstack-infra13:24
anteayagraphana counts all the dependencies13:24
pabelangersdague: right, at this point it is just a waiting game13:24
*** pcaruana has quit IRC13:25
anteayaso 10k in the check queue in grafana doesn't mean 10K patches are getting tested13:25
AJaegeranteaya: Zuul needs to handle those 10k internally.13:25
sdagueright, but if there is an exponential effect then as things merge, they speed up13:25
sdaguebecause zuul doesn't have to process all the chain13:25
anteayaright so that was my early question, has zuul stopped?13:25
AJaegeranteaya: Yes, those are 178 changes stacked on each other - plus those submitted by others - that's why we have 250 now13:25
anteayais zuul processing events or has it ceased?13:25
pabelangeranteaya: no, zuul is still working, just slowly13:25
sdagueno infra-root around?13:25
anteayapabelanger: great thank you13:26
pabelangerI am13:26
sdagueah, ok13:26
anteayasdague: pabelanger is root13:26
sdagueso, I would see about approving the first 10 patches in the stack, and see how long they take to get through13:26
dhellmannso does someone have a script for approving lots of patches? :-)13:26
sdaguegate queue has priority13:26
*** jcoufal has joined #openstack-infra13:26
sdagueif that moves ok, then that's probably a good approach13:26
anteayawhy have we moved to approving?13:27
yolandahi, i am around as well, but my connection is so bad13:27
anteayaI'm not following that logic?13:27
*** yamamoto has quit IRC13:27
anteayathe world is not on fire, zuul is just slow right now13:27
AJaegerthe PTL should approve! davidSubirosPere is not the PTL for that...13:27
anteayawe are not in a release week13:27
*** yamamoto has joined #openstack-infra13:27
jrollworth noting is that this repo has only noop-jobs configured13:27
anteayaI'm inclined to let zuul do it's job13:27
sdaguejroll: right, so it's only zuul provessing time13:27
*** mtanino has quit IRC13:27
jrollya, no node waste13:27
sdagueanteaya: right, that's what I'm saying13:27
rhochmuthAjaeger: I was just expecting one commit. Is there a way to cancel all those reviews?13:27
*** ewilson has joined #openstack-infra13:28
rhochmuthfor monasca-analytics13:28
*** ddieterly has joined #openstack-infra13:28
*** ddieterly is now known as ddieterly[away]13:28
anteayasdague: ah okay, sorry I got confused13:28
AJaegerrhochmuth: manually abort all of them...13:28
jroll+1 for let zuul do it's thing13:28
sdagueAJaeger: but that doesn't really cancel them13:28
jrolland ptl or someone approve them all13:28
*** yottatsa has quit IRC13:28
anteayarhochmuth: please don't do anything yet13:28
*** tonytan4ever has joined #openstack-infra13:28
dhellmannrhochmuth : at this point we're considering just letting you approve them all to push them in, but as anteaya says don't do anything until we agree on a plan13:28
*** salv-orlando has joined #openstack-infra13:28
*** _amrith_ is now known as amrith13:29
AJaegerLet's just all call it an early weekend ;)13:29
sdagueanteaya: so, on the let zuul do it's job front, rhochmuth approving the first 10, figure out if they merge in any reasonable time13:29
sdagueif they do, approving them all through is fine13:29
sdagueif they don't, because things are super gummed up, then something more drastic can be done13:29
AJaegerBut would rhochmuth approve them? He said "I was expecting one commit"...13:29
dhellmannI like that. I might start with fewer than 10, though13:29
sdagueAJaeger: I hope so, they are his commits from his repo13:30
sdaguethe whole point was to keep history13:30
rhochmuthso this is a new repo and I don't have +2 privs on it13:30
anteayaI'm not following the approve them approach as that just keeps us gummed up rather than letting the check queue subside, they still have to finish in check13:30
*** ddieterly[away] is now known as ddieterly13:30
sdagueanteaya: most of them are done in check13:30
sdaguethey are all noop13:30
*** salv-orlando has quit IRC13:31
*** thorst_ has joined #openstack-infra13:31
anteayagreat, then I'm for just leave them be for now and let this subside13:31
AJaeger,members has no members ;(13:31
sdague jenkins +113:31
anteayanot send them back in with approvals13:31
dhellmannpabelanger : can you fix the gerrit permissions on this new repo? ^^13:31
sdagueanteaya: the reason zuul is slow in check is because it's processing the 177 dependencies on each of these13:31
AJaegerrhochmuth, davidSubirosPere : What every you do: please don't push the series again...13:32
sdagueas the number of dependencies goes down13:32
sdaguethey get faster13:32
pabelangerdhellmann: sure, I just need a gerrit id13:32
anteayasdague: wonderful13:32
sdaguehence the "merge some stuff that jenkins already +1ed"13:32
davidSubirosPereAJaeger, I won't, sorry again13:32
anteayayes zuul stats in cacti confirm zuul is recovering13:32
davidSubirosPerefor now I don't do anything, let me know if I have to abort13:32
davidSubirosPeresorry, I mean abandon13:32
sdaguedavidSubirosPere: definitely don't abandon at this point13:33
AJaegerrhochmuth: what's your email - so that pabelanger can add you as initial member to the repo?13:33
sdagueit looks like it's +1ed at least the first 100 changes13:33
*** nadya has joined #openstack-infra13:33
rhochmuthAJaeger: roland.hochmuth@hp.com13:33
sdagueso doing the first 10 will give an idea of how that would proceed13:33
pabelangerAJaeger: rhochmuth added13:34
AJaegerthanks, pabelanger . rhochmuth you're setup now...13:34
*** hdaniel has joined #openstack-infra13:34
rhochmuthcan you also add me on monasca-transform too?13:34
*** BigWillie has joined #openstack-infra13:34
AJaegerrhochmuth: URL to members?13:34
AJaegersdague: indeed, first 100+ jobs are done, so let's wait...13:35
*** javeriak_ has joined #openstack-infra13:35
anteayaI'm for waiting13:35
sdagueAJaeger: I don't understand the waiting13:35
sdaguewhat are you waiting for?13:35
rhochmuthAJaeger: not sure what you mean?13:35
anteayaaccording to cacti zuul can't process anything right now13:35
*** thorst_ has quit IRC13:35
dhellmannrhochmuth : please let's fix this one before dealing with the other repo. can you approve those first 10 patches/13:35
pabelangerWhat I am curious about, if patch B is in check (it has already been handled my zuul-merger), if patch A then is +A and merged, I don't think patch B will be affected if already in the check queue. And still carry the initial dependency chain.13:36
AJaegersdague: Not abort or anything - let zuul do it's job...13:36
*** yamamoto has quit IRC13:36
sdagueAJaeger: right, I agree not aborting13:36
sdaguemy suggestion was to start approving those patches13:37
AJaegerrhochmuth: >13:37
AJaegerrhochmuth: >13:37
*** thorst_ has joined #openstack-infra13:37
AJaegerrhochmuth: sorry ^ let me try again to paste...13:37
pabelangerbut like anteaya said, I'm leaning to get zuul do its job and process the patches.  Since we are week 1 after summit.13:37
sdaguepabelanger: right, it will, unless there are hicups and it restarts the check for whatever reason13:37
AJaegerrhochmuth:,members  is the URL to the monasca-analytics core group. What's the URL for the other group?13:37
sdagueanyway, I need to drop off. Good luck folks.13:38
*** hdaniel has quit IRC13:39
*** yaume has quit IRC13:39
anteayaI think approving will just increase the load on zuul, not decrease it13:39
*** yamamoto has joined #openstack-infra13:40
anteayaI'm -1 on approving patches at the moment13:40
*** Hal has joined #openstack-infra13:40
*** Hal is now known as Guest7821513:40
*** thorst_ has quit IRC13:41
wznoinskhi infra, is there a zuul layout.yml check I could do except for yaml(lint) ? it looks like zuul doesn't start pipelinemanager at all when i have a mistake in the layout.yml (but I don't know what the mistake is after a lot of changes)13:41
AJaegerpabelanger: ^13:43
*** ifarkas has joined #openstack-infra13:43
AJaegerwznoinsk: tox -e zuul in project-config parses the complete file...13:43
*** sdague has quit IRC13:43
*** yamamoto has quit IRC13:44
wznoinskAJaeger: thanks a life saver13:44
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Add a table for story permissions
*** signed8bit is now known as signed8bit_Zzz13:45
pabelangerIt does seem there is a downside to letting zuul process openstack/monasca-analytics, the next patch in the check queue is 230434,22 and has been idle for 3hrs4mins. Meaning, nodepool has not launched any jobs for that patch.13:45
pabelangerSo, any new patch uploaded into zuul is blocked it seems13:46
*** signed8bit_Zzz is now known as signed8bit13:46
*** javeriak_ has quit IRC13:46
AJaegerpabelanger: I'm fine with a #status notice "Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend!" ;)13:47
* AJaeger wouldn't object to other solutions either - I just don't know enough to advise here...13:47
*** yamamoto has joined #openstack-infra13:47
*** sigmavirus24_awa is now known as sigmavirus2413:48
*** ddieterly is now known as ddieterly[away]13:48
anteayaI'm in favour of a status notice13:49
mescanefjhesketh: hello. regarding issue!/story/2000532 - in my env adding keepalive to paramikos transport in gerritwatcher solves the issue. related change is on review prepared by someone already but not yet merged:
anteayaand we should have a few roots with some more experience here show up in the next hour so I'm in still in favour of waiting13:49
dhellmannare we still going to approve some of the patches to see how that goes?13:49
pabelangerAJaeger: anteaya: sure, let me see if status will pick it up13:49
*** jaypipes has quit IRC13:50
pabelanger#status notice Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend!13:50
anteayadhellmann: I'm not in favour of that approach as I think that will just put more load on zuul, not decrease any load13:50
dhellmannanteaya : how so?13:50
anteayayou are just asking zuul to process them again13:50
*** trown is now known as trown|outtypewww13:50
pabelangeranteaya: dhellmann: see my comment above about new patches being blocked13:50
*** thorst_ has joined #openstack-infra13:50
*** yamamoto has quit IRC13:51
anteayait processed 100 patches, according to someone who posted in backscroll, zuul now doesn't have to process those patches, if you approve them you are asking zuul to care about them again13:51
pabelangerAJaeger: guess I don't have permission13:51
dhellmannanteaya : ok. so we wait for them to pass check, then abandon them?13:51
dhellmannif this is preventing folks from getting any work done I think we need to take stronger action. if we have to wait a bit for more experience to come online, that's fine13:51
anteayapabelanger: right that is what I'm saying, putting patches into the gate queue are blocked13:51
anteayazuul isn't processing new patches at this time13:52
anteayadhellmann: I don'tknow what the step is after they clear check13:52
anteayaafter they clear check I see no advantage in abandoning them13:52
anteayazuul doesn't care about them after they have left the check queue13:52
anteayaso abandoning them serves no purpose as far as un gumming zuul13:53
dhellmannI suppose at that point if they are approved one at a time it wouldn't cause a problem13:53
anteayaI haven't even thought about how to fix the repo13:53
anteayaI'm trying to figure out the best path for clearing zuul13:53
anteayaand zuul has been great at chugging along through things, it doesn't tend to fall over13:54
pabelangerRight, but knowing that now, I'm thinking maybe we do need stronger action like dhellmann is suggesting. Since zuul is just CPU pegged doing noop checks, I'm not sure letting the check queue back up is in our best interest13:54
anteayaand according to cacti it has lots of free memory13:54
anteayapabelanger: okay, I'm still of a mind fungi or clarkb will show up soon13:54
dhellmannif we have a 3+hr wait for a new job to even start, we're going to pretty quickly get to a point where nothing happens today13:55
anteayabut if you would like to move ahead, which direction are you leaning?13:55
anteayadhellmann: I also recognize that currently pabelanger is the only root right now13:55
*** edmondsw has joined #openstack-infra13:55
*** yamamoto has joined #openstack-infra13:55
dhellmannsure. I'd like to have a concrete proposal to put to fungi or clarkb when they come online13:55
pabelangeranteaya: I'd like to get input from fungi / clarkb too.13:55
anteayaand he is doing a great job but still would benefit from consultations with peers if they show up soon13:55
anteayapabelanger: me too13:56
anteayaokay so what's on the table?13:56
anteayawhat do folks want to do?13:56
*** ddieterly[away] is now known as ddieterly13:56
pabelangerI think we have 2 options, either dump the check / gate queue, restart zuul. then requeue only gate. Or +A a patch to see how zuul response13:56
dhellmannI think I agree with anteaya that approving patches doesn't buy us anything right now13:56
AJaegerpabelanger: are you authenticated?13:56
*** dansmith is now known as snotdan13:56
anteayaof those two options, I'm in the restart zuul camp, save the gate queue13:57
pabelangerThere could be more options, but I don't know them at this point in time13:57
pabelangerAJaeger: thought I was13:57
anteayapabelanger: have you restarted zuul before?13:57
AJaegerpabelanger: yes, no permissions - see system-config/hiera/common.yaml13:57
*** chuck__ is now known as zul13:57
*** akshai has quit IRC13:57
dhellmannwe did discuss a few options for causing these items to drop out of check without restarting zuul, but I think we ruled all of those out as not effective?13:57
*** xarses has quit IRC13:58
anteayadhellmann: like abandoning patches?13:58
pabelangeranteaya: I'd be following
dhellmannabandoning, or changing their commit message13:58
anteayadhellmann: I don't believe abandoning patches will affect check at all13:58
*** jkraj has joined #openstack-infra13:58
anteayapabelanger: right so that's no, you haven't restarted it before13:58
anteayapabelanger: which is fine13:58
*** snotdan is now known as superdan13:58
anteayaand I'm fine with restart as a way forward13:59
pabelangeranteaya: sorry, I have. With fungi, pleia2. but I would be following the steps in the document13:59
anteayabut I do think that waiting another 40 minutes for some backup for pabelanger is in our best interest13:59
*** akshai has joined #openstack-infra13:59
anteayapabelanger: ah okay thanks13:59
*** jtomasek has joined #openstack-infra13:59
anteayapabelanger: your call then13:59
anteayaI'm leaning towards waiting13:59
anteayaother than blocking normal friday work, there is no critical release14:00
*** vhosakot has joined #openstack-infra14:00
anteayathis is not a make or break moment14:00
anteayajust so you have some backup on the restart pabelanger14:00
anteayabut up to you14:00
*** sdake_ has joined #openstack-infra14:00
*** Goneri has quit IRC14:01
pabelangerwe're this far into the delay, I have no issue waiting for fungi or clarkb14:01
anteayapabelanger: I support that decision14:01
pabelangergoing to see if I have an out of band contain for either from the summit14:01
anteayadhellmann: thanks :) glad you have your help here14:01
anteayapabelanger: okay14:02
*** yamamoto has quit IRC14:02
*** yamamoto has joined #openstack-infra14:02
*** maestro has quit IRC14:03
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu binary gate on liberty branch
*** sdake has quit IRC14:04
*** javeriak has joined #openstack-infra14:04
*** psilvad has quit IRC14:06
senkzuul seems really fucked up :D14:06
*** ayoung has joined #openstack-infra14:07
senkshould also start plan how to avoid that in the future..14:07
*** ramishra has quit IRC14:07
*** ramishra has joined #openstack-infra14:07
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run cinderclient functional tests in Identity v3-only
anteayasenk: can you find better ways to express yourself please?14:08
senkanteaya: i'm currently thinking about how that desaster could have been avoided.14:09
anteayaI'm currently thinking that there is enough stress in channel without you swearing14:09
*** mtanino has joined #openstack-infra14:09
anteayaI find your choice of language offensive14:09
*** jang has joined #openstack-infra14:09
*** sridhar_ram has joined #openstack-infra14:10
senkanteaya: sorry about that :-/ it was not my intention to offend anyone14:10
pabelangerAJaeger: thanks, adding it now14:10
anteayawell work on that goal14:10
openstackgerritZane Bitter proposed openstack/gertty: Support SyncQueriedChanges batching in Gerrit >=2.9
*** jkraj has quit IRC14:11
*** nadya has quit IRC14:12
*** sridhar_ram1 has joined #openstack-infra14:14
*** pabelanger has quit IRC14:14
*** pabelanger has joined #openstack-infra14:14
pabelanger#status notice Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend!14:14
openstackstatuspabelanger: sending notice14:14
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow the creation of private stories
*** Goneri has joined #openstack-infra14:14
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Add a table for story permissions
*** sridhar_ram has quit IRC14:14
pabelangerAJaeger: I needed to authenticate with freenode14:15
* mordred would like to tip his hat to zuul for dealing with this number of changes, even if slowly14:15
-openstackstatus- NOTICE: Zuul is currently recovering from a large number of changes, it will take a few hours until your job is processed. Please have patience and enjoy a great weekend!14:15
anteayazuul io seems to have come back down to pre-spike levels:
*** salv-orlando has joined #openstack-infra14:15
anteayamordred: morning14:15
openstackgerritGraham Hayes proposed openstack-infra/project-config: Retire openstack/designate-msdnsagent repo
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Add Emilien Macchi blog to TripleO blogs feed
AJaegerand you were listed in system-config, not sure how I could miss that ;(14:15
anteayamordred: isn't zuul a trooper!14:16
mordredanteaya: right?14:16
* regXboi watches and reads, waiting for patches behind the elephant to finally make it into the python14:16
anteayait is a sight to behold14:16
anteayaregXboi: hey there, some good reading14:16
regXboianteaya: ack - I wasn't going to say anything unless things started to go off the rails :)14:16
*** yaume has joined #openstack-infra14:17
openstackstatuspabelanger: finished sending notice14:17
anteayamordred: so the plan is pabelanger restarts zuul and saves the gate queue and not the check queue14:17
* regXboi sighs14:17
anteayamordred: are you available to be his backup as he does this?14:17
*** links has quit IRC14:17
anteayaregXboi: ha ha ha, ah thank you14:17
* regXboi prepares rechecks on patches waiting in queue14:17
*** vgridnev has joined #openstack-infra14:17
anteayaregXboi: let zuul take a breath before you send them all back in?14:18
*** xarses has joined #openstack-infra14:18
anteayaregXboi: just for a moment?14:18
regXboianteaya: it's only two - I haven't pushed any more and I was going to wait until after my phone call... which could take who knows how long :)14:18
anteayaah the phone call14:18
anteayafair enough, thanks14:18
regXboithe *dreaded* *unscheduled* *massively delayed* phone call14:18
*** jlibosva has joined #openstack-infra14:18
anteayawooo the best kind of phone call14:19
regXboibut who's counting :)14:19
anteayamordred: what say you?14:19
anteayayou can hand pabelanger cats helpfully14:19
*** javeriak has quit IRC14:19
*** vhosakot has quit IRC14:19
*** jaypipes has joined #openstack-infra14:20
mordredanteaya: I am happy to hand cats14:21
anteayapabelanger: are you willing to proceed with the zuul restart with mordred as backup cat distributor?14:22
pabelangersure, we all agree, dump check / gate queues, restart zuul, restore only gate14:22
* mordred hands pabelanger a few cats that are only slightly ravenous14:22
anteayaso yes save the gate queue before zuul is stopped14:23
*** zz_dimtruck is now known as dimtruck14:23
* regXboi mutters ... speaking of ravenous ... time to grab something for breakfast14:23
anteayaregXboi: happy breakfast14:23
pabelangerokay, fetching documentation14:23
anteayapabelanger: very good14:23
openstackgerritJulien Danjou proposed openstack/requirements: Remove Ceilometer from projects
AJaegerpabelanger: the top two jobs in the gate are nearly finished, let's merge those first...14:24
pabelangerokay, zuul stopped14:25
pabelangerdeleted unused nodes in nodepool14:25
AJaegerthen let's restart them - no problem...14:25
anteayaAJaeger: yeah only so many things he can juggle at a time14:25
*** ccamacho has quit IRC14:25
*** r-daneel has joined #openstack-infra14:26
pabelangerzuul restarted14:26
AJaegeranteaya: bad timing;) No problem...14:26
anteayaAJaeger: :) good intentions though14:26
openstackgerritMerged openstack-infra/tripleo-ci: Increase timeout and logging for crm_resource
openstackgerritMerged openstack-infra/tripleo-ci: Use megabytes for overcloud image size
anteayalooks good from status.o.o/zuul14:28
pabelangerokay, gate requeued14:29
pabelangerchecking nodepool14:29
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only
*** ccamacho has joined #openstack-infra14:29
*** javeriak has joined #openstack-infra14:30
pabelangerokay, nodepool building nodes14:31
pabelangerI think we are good14:31
anteayawell done14:31
anteayamordred: thanks for being available14:31
AJaegerthanks, pabelanger !14:31
pabelangerjust confirming we dont14:31
pabelangerhave any NOT_REGISTERED jobs14:31
anteayagood call14:31
AJaegerpabelanger: please send out a #status notice and ask for recheck of check jobs14:32
senkgreat work pabelanger !14:32
fungiguess i picked a fine time to take a late morning. catching up on the situation now14:32
anteayafungi: pabelanger has it under control14:32
mordredpabelanger: nicely done14:33
anteayafungi: the tl;dr is someone pushed a 177 patch chain which slowed down zuul14:33
* mordred hands pabelanger a fluffy cat that has been recently fed14:33
openstackgerritJianghua Wang proposed openstack-infra/nodepool: Add namespace for nodepool
anteayafungi: after some discussion we agreed the best way forward was to restart zuul saving only the gate queue14:33
pabelangerI don't see any NOT_REGISTERED, so think we are in the clear14:33
anteayapabelanger: thanks for checking14:33
pabelangeranteaya: AJaeger: mordred: senk: thanks14:33
*** psilvad has joined #openstack-infra14:34
jrollpabelanger: ^5, thanks for the help14:34
rhochmuththanks everyone14:34
mordredanteaya, pabelanger: in the future there is a way that infra-core people can help with importing 177 patch chains without triggering tons of zuul, in case that's helpful to know14:34
*** kushal has quit IRC14:35
mordredrhochmuth: if it makes you feel any better, the first person to do what you did was vish14:35
pabelangerso, I think moving forward, rhochmuth will need to +A the current +1 patches14:35
anteayarhochmuth: okay so, were you part of the decision to submit a 177 patch chain?14:35
fungialso, glad to see this was not an emergency, just zuul working as expected14:35
pabelangermordred: agreed!14:35
mordredrhochmuth: which is to say - it happens to the best of us14:35
rhochmuthmordred: that helps, thanks14:35
anteayarhochmuth: if yes, did anyone involved read any documentation prior to that decision?14:35
fungian excellent test of its ability to handle very, very long patch series14:35
mordredfungi: right?14:35
*** rbrndt has joined #openstack-infra14:35
anteayafungi: yes exactly14:35
mordredI'm not sure I think we need to +A those patches14:35
mordredrhochmuth: this is import of history, yeah?14:36
rhochmuthi'm not sure it is14:36
rhochmuthi'll have to work with davidSubirosPere14:36
anteayarhochmuth: I'm trying to understand if changing documentation would do any good, which wouldn't if you didn't look at any14:36
*** thorst_ has quit IRC14:36
rhochmuthi was only expecting on e commit14:36
*** tojuvone_ has quit IRC14:36
rhochmuthnot the entire history too14:36
anteayarhochmuth: you just thought this would end up as one patch?14:36
mordredrhochmuth: gotcha. so - if the patches in question are correct14:36
AJaegerrhochmuth: next time: If you create a repo, you can import an existing one. So, if there's one you want to import with history, it's trivial at repo creation time...14:37
*** dprince has quit IRC14:37
mordredrhochmuth: and we do want to land them - I suggest that we have an infra-core person just push them to the branch, which will close all of the reviews without making zuul run any gate jobs for them14:37
* fungi is surprised git review didn't throw a warning like "you are about to submit 176 changes..."14:37
rhochmuththe repo wasn't in public github to import14:37
pabelangerfungi: I was thinking that too14:37
mordredrhochmuth: which is easy to do14:37
pabelangerfungi: maybe we need clippy to pop up asking are you sure14:38
mordredso, basically, let's just connect once you have an idea of what you want the end-state to be and we can help you get there14:38
fungidoesn't have to be github either. i just temporarily chuck the git repo up onto my personal webserver until it gets imported, then delete it14:38
anteayapabelanger: no, no clippy, no clippy14:38
rhochmuthdavidsupirospere is new to the project from hp labs14:38
jrollmordred: fwiw, there's no jobs here, just the explosion of events14:38
rhochmuthbut will discuss options with him14:38
AJaegerrhochmuth: happy mentoring!14:38
mordredjroll: it's all no-op jobs?14:38
jrollmordred: yep14:38
rhochmuthi'm not a good mentor14:38
rhochmuthdavidsubirospere are you still there14:39
AJaegerrhochmuth: then best find one ;)14:39
mordredjroll: k. well, still - one git push is still way less work than someone pressing +A 176 times14:39
mordredeven with gertty14:39
jrollmordred: totes, just a heads up :)14:39
pabelangerI guess we should send a status notice telling people zuul was restarted and check queue lost14:39
pabelangerand to recheck patches14:39
jrollmordred: shocking that there's no mass approve scripts out there14:39
fungijroll: actually... gertty has batch operations14:40
anteayapabelanger: agreed14:40
*** thorst_ has joined #openstack-infra14:40
fungiflag the dependent series of reviews, batch approve/verify+2/review+2/submit14:40
*** MarkAtwood has joined #openstack-infra14:40
* fungi can't remember if batch submit is implemented yet though14:41
*** hdd has joined #openstack-infra14:41
anteayaI thought it was14:41
* dhellmann makes a note to give gertty another try14:41
anteayaI thought jeblair batch approved something a month or so ago14:41
dhellmannI assume after the restart it's safe to do things like tag releases?14:42
anteayadhellmann: yes, you may operate as normal now14:42
dhellmannanteaya : thanks for confirming14:42
fungireview, abandon/restore, edit topic14:42
fungino batch submit (yet)14:42
anteayadhellmann: thanks for your help here14:42
davidSubirosPererhochmuth yes I'm here14:43
anteayaah, sorry it was batch review I was referencing14:43
*** e0ne has quit IRC14:43
funginot that we use submit very often since it bypasses ci completely, so maybe not critical to have a batch version (also maybe even a little dangerous)14:43
anteayadavidSubirosPere: out of curiosity, did you reference any documentation prior to submitting that patch chain?14:43
openstackgerrityolanda.robla proposed openstack-infra/shade: Add magnum services call to shade
*** timsim has joined #openstack-infra14:43
*** e0ne has joined #openstack-infra14:44
anteayadavidSubirosPere: I ask wondering if changing our documentation would have made any difference here14:44
*** Na3iL has quit IRC14:44
rhochmuthdavidSubirosPere: everything is restored, we should talk offline about doing a single review14:44
fungiand anyway, batch submit could just be accomplished with a push to the branch instead of refs/for/refs as mordred suggested14:44
fungiby one of the gerrit admins14:44
davidSubirosPereI checked the documentation to create the account and submit a git review, but maybe I should have checked more documentation before pushing all those commits14:44
pabelanger#status notice Zuul has been restarted. As a results, we only preserved patches in the gate queue. Be sure to recheck your patches in gerrit if needed.14:45
openstackstatuspabelanger: sending notice14:45
rhochmuthfungi: the repo is private, so not sure can do a batch push14:45
fungirhochmuth: well, it's not intended to be private, would just need to get a copy of it14:45
*** yottatsa has joined #openstack-infra14:46
-openstackstatus- NOTICE: Zuul has been restarted. As a results, we only preserved patches in the gate queue. Be sure to recheck your patches in gerrit if needed.14:46
fungidavidSubirosPere: we might could also make the multiple changes warning/confirmation git-review displays a little more verbose/scary14:46
sc68callol love that status update14:46
anteayasc68cal: you have pabelanger to thank for that14:46
davidSubirosPereyes, I think it would be a good idea14:46
anteayasc68cal: as well as fixing zuul14:46
*** mixos has joined #openstack-infra14:46
davidSubirosPereso people know that a job will be created for each commit14:47
sc68calsorry, meant the previous one that said "Have a great weekend"14:47
*** vgridnev has quit IRC14:47
anteayasc68cal: ah, I think that was AJaeger's composition14:47
openstackstatuspabelanger: finished sending notice14:47
sc68calI mean it is true, it's going to take a loong time for zuul to right itself, so might as well have an early friday ;)14:47
*** yamamoto has quit IRC14:47
fungialso, in the future, it's possible to filter the exported pipeline contents to filter out unwanted changes. for example, could just | grep -v problem_projectname14:47
*** psilvad has quit IRC14:48
anteayadavidSubirosPere: have you taken a look at our developer's guide yet? it might interest you:
mordredfungi: we can get a copy of it with git review -d14:48
anteayasc68cal: well pabelanger restarted it14:48
pabelangerfungi: Agreed. Was thinking of updating our documentation about that14:48
fungimordred: good point, now that it's in gerrit14:48
davidSubirosPerethanks! I had a quick look, I will have another look14:48
mordredfungi: so all a gerrit admin needs to do is git review -d the end of the series, then git push gerrit master14:48
anteayasc68cal: so pick your status update of choice14:48
sc68calI'll take the one where I get to start my weekend early :)14:48
mordredsc68cal: well chosen14:49
fungisc68cal: i thought i had started my weekend early, then i made the mistake of looking in irc ;)14:49
* mordred on the other hand is going to go to the driver's license office14:49
*** nadya has joined #openstack-infra14:49
mordredfungi: NO VACATION FOR YOU14:49
anteayadavidSubirosPere: that is the documentation that should discuss our testing system and that each patch is tested individually14:49
sc68calfungi: know that feeling. :)14:49
fungimordred: always makes me think of "dmv" by primus14:49
*** ewilson has quit IRC14:50
mordredanteaya: I think it might be useful to add something to the import-a-project docs14:50
davidSubirosPere I see it's explained there14:50
anteayamordred: fair enough14:50
mordredanteaya: that covers the "what if the repo you want to import currently only exists in a private location"14:50
anteayadavidSubirosPere: ah lovely, glad you found it14:50
mordredsince that is what I believe tripped up davidSubirosPere and rhochmuth14:50
anteayamordred: I see, yes14:50
mordredand I think we can make it clear that that is supported and there is a way to deal with it14:51
anteayato be honest this wasn't a usecase I personally had considered14:51
fungianyway, what was the project and which change was intended as the topic branch tip?14:51
mordredwhich is why I think our docs do a bad job of helping people on it :)14:51
anteayabut agreed, making it clear is helpful14:51
anteayafungi: monasca-analytics14:51
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in keystone
anteayaI don't know the branch tip change14:51
fungii should be able to find it easily enough14:51
anteayathank you14:52
mordredfungi: now, if only gerrit had a good way of finding the tip14:52
fungigertty does14:52
mordredthere it is14:52
fungigerrit always confuses me as to which end is which14:52
mordredit actually wasn't as bad in gerrit as I thought14:52
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
fungiheh... Related Changes (177)14:52
mordred(it helps that commit #1 had the subject "initial commit")14:52
mordredthe related changes field holds up14:53
anteayainitial commit is a helpful initial commit title14:53
mordredfungi: how is gertty's display of it?14:53
mordredfungi: it might be a fun pathological test case for jeblair for gertty ui14:53
anteayaha ha ha14:53
fungii have to wait for it to sync since i wasn't subscribed yet14:54
fungiit's pulling them down pretty quickly though14:54
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only
dhellmanndoes anyone have any idea why a patch to openstack/releases would be showing up in the check-tripleo queue in the zuul web ui? there are no jobs listed in the little box on the web page
anteayaoh while we are here, /home/gerrit2 on review.o.o is looking a bit getting-full:
*** esikachev has quit IRC14:55
*** jaypipes has quit IRC14:55
mordreddhellmann: that is a fascinating question14:56
fungidhellmann: i don't see it... on the page14:56
*** vgridnev has joined #openstack-infra14:56
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
dhellmannfungi : it disappeared as soon as I asked the question14:56
anteayadhellmann: I see 311123 in the check queue of status.o.o/zuul, 4 jobs all queued14:56
fungidhellmann: oh14:56
fungimaybe a rendering error?14:56
anteayatoday seems to be the day14:56
dhellmannmaybe that's all it was -- there were no jobs, so it seemed extra odd.14:56
dhellmannif it shows up again I'll mention it again14:57
anteayawhat happens when /home/gerrit2 on review.o.o fills up?
*** eharney has joined #openstack-infra14:57
*** sdake_ has quit IRC14:57
fungianteaya: badnez14:57
anteayashall we wait and see how bad14:57
anteayaor shall we work it into a todo list?14:58
mordredanteaya: I vote on waiting and seeing what breaks14:58
*** andrewbogott has quit IRC14:58
*** andrewbogott has joined #openstack-infra14:58
anteayamordred: that is to be expected14:58
anteayamordred: since I believe you will be discussing driver license things when that happens14:58
fungii'm looking now to see what's eating it all14:59
anteayafungi: you are so wonderful14:59
anteayathank you14:59
fungiunsurprisingly, it's pretty much all in the git repos15:01
*** akshai has quit IRC15:01
anteayaso make more space?15:01
*** jaosorior has quit IRC15:02
fungiyeah, but i'm digging deeper first15:02
*** smcginnis_ has joined #openstack-infra15:02
anteayafair enough15:02
*** Guest42930 has quit IRC15:05
*** david-lyle has joined #openstack-infra15:06
*** ociuhandu has joined #openstack-infra15:07
*** banix has quit IRC15:08
anteayaAJaeger: thanks for creating 31092915:09
anteayaAJaeger: I just added a comment about confirming source code prior to repo creation15:09
funginova's git repo is now up to 30gb on disk. i'm comparing that against the copy jeblair made back in early january15:09
*** vgridnev has quit IRC15:09
fungilooking at the disk utilization graph, we started a steady climb right around the gerrit 2.11 upgrade15:10
*** sridhar_ram1 has quit IRC15:10
anteayaoh my15:10
fungileading me to believe gerrit 2.11 is stashing lots of new something in there 2.8 didn't15:10
fungiwhich may also explain why they added a background git gc task15:11
*** jaypipes has joined #openstack-infra15:11
amrithhi, I saw the note earlier about zuul being backed up, are we past that now?15:11
anteayathe mystery deepens15:11
fungiamrith: yes15:11
amrithhi fungi ... I've tried to recheck a review a couple of times and I see nothing in the zuul queues (see 303671)15:11
amrithah wait15:12
*** vgridnev has joined #openstack-infra15:12
amrithI guess it is taking a while ...15:12
amrithnow it shows up, thanks15:12
amrithfor a while I didn't even see an openstack/trove queue15:13
*** vgridnev has quit IRC15:13
*** r-mibu has quit IRC15:13
*** jamesmca_ has joined #openstack-infra15:13
*** r-mibu has joined #openstack-infra15:13
*** yamamoto has joined #openstack-infra15:14
*** mixos has quit IRC15:14
*** jlibosva has quit IRC15:15
*** sdake has joined #openstack-infra15:16
*** vgridnev has joined #openstack-infra15:16
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
*** esikachev has joined #openstack-infra15:17
*** dizquierdo has quit IRC15:17
AJaegersc68cal: especially if you're in Europe ;)15:18
*** arxcruz has quit IRC15:19
*** zhurong has quit IRC15:20
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add project-config reviewing guidelines
AJaegeranteaya: ready to merge now? ^15:21
*** dynarro has joined #openstack-infra15:21
AJaegerfungi, I'd like to get my bonus points for not storing pot files in git anymore: Do you or any other project-config core time to review, please?15:22
anteayaAJaeger: I'm happy, thank you15:22
AJaegerAny other project-config cores that like to review our reviewing guidelines, please?
AJaegerthanks, anteaya !15:22
anteayathanks for writing it15:23
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow the creation of private stories
fungiokay, so the size of the nova repo on review.o.o was 5gb on january 8, and is 30gb now five months later15:24
AJaegerhow much of that is due to pot source files?15:24
anteayathat seems not related to nova contributors15:24
AJaegerwe could also switch to weekly imports ;)15:24
anteayaAJaeger: did you want to wait for fungi on 313114?15:24
fungiagreed. i think this is some sort of cruft building up in there15:24
*** akshai has joined #openstack-infra15:25
*** vhosakot has joined #openstack-infra15:25
*** yamamoto has quit IRC15:25
fungiprobably not potfile updates15:25
*** vgridnev has quit IRC15:25
AJaegeranteaya: he mentioned in the prerequisite for this, the bonus points ;) So, I'm fine with merging it, just wanted it reviewed ;) fungi, do you want to review?15:25
fungiAJaeger: if it has enough reviews already, then not necessary15:26
anteayaI can merge it15:26
AJaegerfungi, it has now - thanks to anteaya !15:26
fungiworking on adding a second volume to review.o.o for now to buy us time to investigate the out of control growth in its git repos15:26
anteayasounds good15:26
AJaegerthe next translation updates will be large - they will remove pot files ;) But afterwards they will be significantly smaller...15:27
pabelangerclarkb: I am confused why nodepool didn't build new images at 10am UTC today15:28
fungipabelanger: no logs?15:28
*** jamesmca_ has quit IRC15:28
fungipabelanger: or started at 14:14 instead?15:29
pabelangerfungi: from what I see, they started at last night15:29
pabelangerfungi: we updated it to start at 10:34 now15:29
pabelangerbut don't see builds at that time15:30
fungiand restarted the builder daemon after that landed?15:30
pabelangercurrently uploading an image to rax-iad, since it failed before15:31
pabelangerbut, will restart again to be sure15:31
fungiyikes, our gerrit trove db is also almost out of space15:31
*** jordanP has quit IRC15:31
anteayafungi: oh no15:31
anteayafungi: glad you checked, is that on cacti anywhere?15:31
fungiit was 10gb and 90% full, so i'm upping it to 50gb15:32
*** sdake has quit IRC15:32
fungilooks like it's supposed to be able to do that online15:32
fungiso hopefully non-impacting15:32
*** dmk0202 has quit IRC15:33
*** jtomasek has quit IRC15:33
*** esikachev has quit IRC15:33
openstackgerritMerged openstack-infra/project-config: Do not store source translation files in git
pabelangermordred: when you have a moment, can you look into /var/log/nodepool/nodepool-builder.log on nodepool.o.o I am seeing a lot of Manager rax-ord running task ImageTaskGet spamming the lgs15:34
*** dmk0202 has joined #openstack-infra15:34
*** mixos has joined #openstack-infra15:34
pabelangercurious if that is to be expected15:35
*** bpokorny has joined #openstack-infra15:35
*** dtantsur is now known as dtantsur|afk15:35
*** jaypipes is now known as leakypipes15:36
openstackgerritMerged openstack-infra/project-config: Add pyghmi from source non-voting test
*** oanson has joined #openstack-infra15:38
*** markus_z has quit IRC15:38
*** jcoufal has quit IRC15:38
*** mikelk has quit IRC15:38
openstackgerritChris Denneen proposed openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options
*** dmk0202 has quit IRC15:39
*** nadya has quit IRC15:40
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
*** rossella_s has quit IRC15:40
fungii've added a new 200gb ssd volume to review.o.o as a replacement for the current 100gb ssd volume. once i'm comfortable that things are still stable after the trove volume resize, i'll pvmove the extents from the old cinder volume to the new one and then extend the lv/fs to 200gb15:41
*** rossella_s has joined #openstack-infra15:41
fungiafter which time i can take the old volume out of service and delete it15:41
*** jsavak has quit IRC15:42
*** flepied has quit IRC15:42
fungi#status log the trove instance for was 10gb and 90% full, so i'm upping it to 50gb (which is supposed to be a non-impacting online operation)15:42
openstackstatusfungi: finished logging15:42
fungi#status log the current 100gb /home/gerrit2 on is 95% full, so i've added a new 200gb ssd volume to review.o.o as a replacement for the current 100gb ssd volume. once i'm comfortable that things are still stable after the trove volume resize, i'll pvmove the extents from the old cinder volume to the new one and then extend the lv/fs to 200gb15:43
openstackstatusfungi: finished logging15:43
*** salv-orlando has quit IRC15:43
*** jcoufal has joined #openstack-infra15:44
*** nadya has joined #openstack-infra15:44
anteayafungi: thank you15:46
fungianteaya: thanks for spotting the fs growth15:46
*** salv-orlando has joined #openstack-infra15:47
fungiwe may need to bite the bullet on enabling the background git gc, but i'll bring it up again when more people are around to brainstorm15:47
anteayaI do hope someone has the time to dig deeper and see if this is related to the memory leak gc issue15:47
anteayafungi: sounds like a good plan15:47
*** nadya has quit IRC15:48
fungidoubtful that it would be related to the jvm gc issues we're encountering, but you never know (if this is stuff egtting written temporarily to disk _and_ cached in memory)15:48
*** nadya has joined #openstack-infra15:48
fungionce we have more space on there, i'll do a normal git gc on a copy of the repo and see if i can tell what it ends up cleaning up15:48
*** ddieterly is now known as ddieterly[away]15:49
AJaegerteam, it would be nice to get another +2 on the project config reviewing guidelines... Thanks!15:50
*** shashank_hegde has joined #openstack-infra15:50
* AJaeger now signs off for the day15:50
*** nadya has quit IRC15:51
*** thorst_ has quit IRC15:51
*** yottatsa has quit IRC15:52
fungiheh, rackspace has a kb article on "upgrade a cloud databases instance from mysql 5.1 to mysql 5.6" but what it describes is creating a new trove instance and then using mysqldump to copy your data out of the old instance and applying it in the new one. misleading title--got my hopes up15:52
openstackgerritAnita Kuno proposed openstack-infra/infra-manual: Ensure project retirement removes dot files
anteayafungi: :/15:54
*** apuimedo has joined #openstack-infra15:54
anteayaAJaeger: enjoy your weekend15:54
*** jsavak has joined #openstack-infra15:55
*** ddieterly[away] is now known as ddieterly15:55
apuimedolifeless: question about pbr. How can I do it so that after I do `python install --prefix / --root /` the installed software does not depend on git for the version15:55
apuimedo(on kuryr, which uses pbr)15:56
*** lakshmiS has joined #openstack-infra15:56
*** davideagnello has joined #openstack-infra15:57
*** jlanoux has quit IRC15:57
*** thorst_ has joined #openstack-infra15:58
*** tphummel has joined #openstack-infra15:58
*** thorst_ has quit IRC16:01
anteayapabelanger: can you look at and confirm it is needed still or abandon?16:01
*** thorst_ has joined #openstack-infra16:01
sigmavirus24Is!/project/760 still the place for shade bugs?16:02
*** inc0 has quit IRC16:02
pabelanger#status log OSIC leads 21 FIPs, they have been deleted manually.16:02
openstackstatuspabelanger: finished logging16:03
pabelangeranteaya: checking16:03
anteayasamueldmq: here is the list of folks on the project team guide team:,members16:03
sigmavirus24apuimedo: can you expound on that? The version *should* be generated once16:03
*** yolanda has quit IRC16:03
pabelangeranteaya: thanks, abandoned16:03
anteayasamueldmq: they are the folks who can approve
anteayapabelanger: thank you16:03
anteayasigmavirus24: yes16:03
sigmavirus24Thanks anteaya :)16:04
anteayasigmavirus24: welcome, thanks for asking16:04
*** ams_ has joined #openstack-infra16:04
*** kjorgensen has joined #openstack-infra16:04
anteayanow the follow up to that is that mordred is one of the main shade folks and as far as I know rarely reads bug reports16:04
anteayabut storyboard is the shade bug tracker16:04
*** kjorgensen has quit IRC16:05
*** kjorgensen has joined #openstack-infra16:05
apuimedosigmavirus24: sure16:05
apuimedoI am making a container16:05
pabelangerinfra-root: we might have an issue with osic-cloud1:
apuimedoin it, I clone the git repo16:05
clarkbpabelanger: sorry feel half dead this morning but if zuul was restarted the image builds in gearman would go away16:05
apuimedoand do python install --root / --prefix /16:06
pabelangerseeing a lot of connection reset by peer exceptions for osic-cloud116:06
apuimedobut when I run the project16:06
sigmavirus24anteaya: Yeah I was looking for a bug related to the new betamax tests that shade is adding and wanted to contribute to the discussion :)16:06
*** derekh has quit IRC16:06
apuimedoI get the following error16:06
clarkbsince zuul's geard us used by nodepool for image builds16:06
pabelangerclarkb: okay, let me confirm that.16:06
anteayasigmavirus24: awesome, thank you16:06
*** kushal has joined #openstack-infra16:06
*** nadya has joined #openstack-infra16:06
anteayasigmavirus24: did you find anything useful yet?16:06
*** dynarro has quit IRC16:06
anteayamorning zaro16:06
anteayazaro: so we have a thing16:06
*** davidSubirosPere has quit IRC16:07
sigmavirus24anteaya: no, that's why I asked (to confirm that storyboard was still the place)16:07
*** senk has quit IRC16:07
*** roxanaghe has joined #openstack-infra16:07
pabelangerI cannot remember who are contact for osic is to confirm we are seeing an error16:07
fungiclarkb: pabelanger: yeah, if you restart zuul you also need to delete nodes in nodepool16:07
fungiat the moment16:07
anteayazaro: our gerrit /home/gerrit2 is very full very fast16:07
pabelangerfungi: Yup, did that already16:07
sigmavirus24apuimedo: go on :) (preferably using ;))16:07
clarkbpabelanger: fungi should have their emails in his inbox somehwere16:07
anteayazaro: it seems to have changed the rate of storage after the upgrade to 2.1116:07
anteayasigmavirus24: ah, makes sense16:08
fungipabelanger: what error/behavior are you seeing?16:08
apuimedosigmavirus24: yes, yes. Always paste openstack org16:08
fungioh, you pasted it above16:08
pabelangerfungi: see my pastebin from nodepool16:08
*** kzaitsev_mb has quit IRC16:08
anteayaclarkb: go back to bed16:08
pabelangerfungi: Ya, that is one of a few execptions I am seeing16:08
*** johnthetubaguy has quit IRC16:08
fungipabelanger: looks like fip quota exceeded probably?16:08
*** johnthetubaguy has joined #openstack-infra16:08
anteayaclarkb: get rest, drink water, consider a visit to a doctor16:08
fungipabelanger: we likely leaked all the fips for all the nodes which were running there when we restarted16:09
pabelangerfungi: running the neutronclient commend I no longer see any leaked FIPs16:09
pabelangerfungi: we had 21, but I deleted them16:09
sigmavirus24apuimedo: can you try using `pip install --root/ --prefix/` instead? setuptools + pbr might not be doing the right thing there with those options and I'm not sure why16:10
*** dmellado is now known as dmellado|off16:10
fungipabelanger: but still getting that traceback in the logs?16:10
pabelangerfungi: yes16:10
apuimedosigmavirus24: pip install?16:10
*** psilvad has joined #openstack-infra16:10
pabelangersorry for the incoming spam16:10
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add ubuntu-precise to bindep-fallback
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace bare-precise with ubuntu-precise for system-config
openstackgerritPaul Belanger proposed openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise
openstackgerritPaul Belanger proposed openstack-infra/project-config: Set min-ready 0 for bare-precise
openstackgerritPaul Belanger proposed openstack-infra/project-config: Disable bare-precise in nodepool
openstackgerritPaul Belanger proposed openstack-infra/project-config: Delete bare-precise
sigmavirus24apuimedo: no, just `pip install ...`16:10
*** senk has joined #openstack-infra16:10
*** flepied has joined #openstack-infra16:10
sigmavirus24apuimedo: ah yeah, you'll need `pip install --root / --prefix / .`16:10
sigmavirus24You'll need the `.` to mean "install the package in this directory"16:11
*** piet has quit IRC16:11
pabelangerAJaeger: anteaya: if you have time: is what is needed to remove bare-precise from the gate16:11
*** roxanaghe has quit IRC16:12
apuimedosigmavirus24: does that automatically go through the requirements.txt?16:13
* apuimedo was not familiar that pip had the install functionality built it16:13
apuimedo*built in16:13
sigmavirus24apuimedo: pip is magical ;) but pbr reads in requirements.txt to generate the proper data for pip to understand that it needs to install those dependencies16:13
anteayapabelanger: thank you16:13
*** apoorvad has joined #openstack-infra16:14
*** oanson has quit IRC16:14
apuimedosigmavirus24: is there someway for having setup.cfg mandate a oslo-config-generator run, so the installation includes the config example?16:15
sigmavirus24apuimedo: what version of pip are you running?16:15
*** sdake has joined #openstack-infra16:15
sigmavirus24and no, there's no way (I know of) to have oslo-config-generator run on install and generate a sample config16:16
sigmavirus24(and I'd lobby against that anyway)16:16
apuimedo/ # pip --version16:16
apuimedopip 8.1.1 from /usr/lib/python2.7/site-packages (python 2.7)16:16
*** woodster_ has joined #openstack-infra16:16
pabelangerfungi: asking in #osic if they are away of any outages / issues16:16
apuimedosigmavirus24: why would you be against it?16:16
fungipabelanger: cloudnull has offered to help troubleshoot backend issues in osic in the past, so if he's around he might be able to check why our neutron create floating ip calls there are throwing keystone session errors16:16
*** yaume has quit IRC16:16
apuimedoit's nice to distribute sample conf files16:17
sigmavirus24fungi: I don't think cloudnull is around this morning16:17
*** johnny has joined #openstack-infra16:17
apuimedo(with everything commented)16:17
pabelangerfungi: ack16:17
*** johnny is now known as Guest3479816:17
sigmavirus24It's a shame more #osic people don't hang out in here to start with :/16:17
*** yaume has joined #openstack-infra16:17
fungiapuimedo: what would make more sense is to have the sample config generator run at sdist/wheel build time rather than install time16:17
sigmavirus24fungi: I agree16:17
apuimedofungi: yes, either of those would be fine16:18
sigmavirus24apuimedo: downstream linux distributions would shoot us if we ran it at install16:18
apuimedobut I didn't fine a way to specify prefix on sdist16:18
apuimedoand I *need* to have things put in /usr/lib/docker/ forcefully16:18
*** Nikolay_St has quit IRC16:18
apuimedonot in some relative venv location16:19
*** piet has joined #openstack-infra16:19
*** piet has quit IRC16:19
fungiapuimedo: you may need a custom unpacker/installer of some sort? pip is designed to put things into places it considers runnable for a given environment (be it global or virtual)16:19
apuimedosigmavirus24: it would look okay with archlinux package generation, I don't really see what the problem would be with rpm specfile16:20
apuimedofungi: currently I use data_files16:20
apuimedoin pbr / setuptools to place that absolute files16:20
apuimedoand that works16:20
apuimedobut obviously I can't do it when doing sdist16:20
apuimedoonly when I do install16:20
*** ewilson has joined #openstack-infra16:21
apuimedosigmavirus24: any idea, about the crash?16:21
fungiwow, the trove volume resize for our gerrit db is still going16:21
sigmavirus24apuimedo: no I'm not sure what's going on there :(16:21
* sigmavirus24 hasn't tried setting up kuryr before16:22
apuimedosigmavirus24: I have a dockerfile that builds it, if you want16:22
sigmavirus24pbr should know how to get the version info from a PKG-INFO file but I'm not sure if `python install` is generating that and if maybe it's having problems because of the `--root/--prefix` options your'e using16:23
*** e0ne has quit IRC16:23
fungiapuimedo: where did .../kuryr-0.1.0.dev393-py2.7.egg-info/installed-files.txt end up?16:23
sigmavirus24apuimedo: might as well log a bug for it16:23
apuimedolet me see where it ended up16:23
fungithat might tell us where the effective root for installing the egg-info was16:24
*** ruhe has quit IRC16:24
fungi(which is obviously not where it then ended up looking for it)16:24
anteayafungi: :/16:25
*** shashank_hegde has quit IRC16:25
*** warpc has quit IRC16:25
*** dprince has joined #openstack-infra16:25
*** dabukalam has quit IRC16:25
*** salv-orlando has quit IRC16:25
*** beisner has quit IRC16:25
*** dabukalam has joined #openstack-infra16:25
*** yamamoto has joined #openstack-infra16:25
*** grauzikas has quit IRC16:26
*** grauzikas has joined #openstack-infra16:26
*** warpc has joined #openstack-infra16:26
apuimedofungi: ^^16:26
*** vipul- has quit IRC16:26
*** vipuls has joined #openstack-infra16:27
fungiapuimedo: got it. so it's looking for the file in a relative path, but not relative to / apparently16:27
fungiso cwd must be elsewhere at that point16:27
openstackgerritPaul Belanger proposed openstack-infra/project-config: Replace bare-precise with ubuntu-precise in infra-puppet-apply-jobs
openstackgerritPaul Belanger proposed openstack-infra/project-config: Delete bare-precise
openstackgerritPaul Belanger proposed openstack-infra/project-config: Disable bare-precise in nodepool
openstackgerritPaul Belanger proposed openstack-infra/project-config: Set min-ready 0 for bare-precise
openstackgerritPaul Belanger proposed openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise
*** SotK has quit IRC16:28
apuimedofungi: I run the uwsgi server from /16:28
*** mrodden has quit IRC16:28
*** SotK has joined #openstack-infra16:28
fungiyeah, that makes it all the more strange16:28
*** devananda has quit IRC16:28
*** mtreinish has quit IRC16:28
anteayapabelanger: reviewed the stack, two comments on commit messages16:28
fungiso something is changing the working directory prior to attempting to open that file16:29
*** thorst_ has quit IRC16:29
*** stevebaker has quit IRC16:29
*** devananda has joined #openstack-infra16:30
openstackgerritPaul Belanger proposed openstack-infra/project-config: Delete bare-precise
openstackgerritPaul Belanger proposed openstack-infra/project-config: Disable bare-precise in nodepool
openstackgerritPaul Belanger proposed openstack-infra/project-config: Set min-ready 0 for bare-precise
fungiapuimedo: oh! i misread16:30
openstackgerritPaul Belanger proposed openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise
pabelangeranteaya: ack, updated16:30
apuimedoI'll put a print to see where on earth pbr tries to find it16:30
fungiapuimedo: so something is more likely failing to create installed-files.txt actually16:30
*** yamamoto has quit IRC16:30
*** ruhe has joined #openstack-infra16:31
fungigiven that kuryr-0.1.0.dev393-py2.7.egg-info exists and has files under it, but not the installed-files.txt file16:31
fungion even closer inspection, it's attempting to open for write16:32
anteayapabelanger: thank you16:32
*** yaume has quit IRC16:32
*** beisner has joined #openstack-infra16:32
*** inc0 has joined #openstack-infra16:32
* apuimedo running again16:32
fungiso i guess that does mean it's not finding the directory (implying it's looking in the wrong place after all)16:32
*** SergeyLukjanov has quit IRC16:32
*** SergeyLukjanov has joined #openstack-infra16:33
*** mrodden has joined #openstack-infra16:33
*** thorst_ has joined #openstack-infra16:33
fungiso yes, need to know what inst_files_path is at that point16:33
*** stevebaker has joined #openstack-infra16:33
*** bpokorny_ has joined #openstack-infra16:34
*** mtreinish has joined #openstack-infra16:35
*** pblaho has quit IRC16:35
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
*** jsavak has quit IRC16:36
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add openstack/pyghmi to the $PROJECTS variable
*** ddieterly is now known as ddieterly[away]16:37
apuimedothis is the output from the install16:37
anteayajesusaur: what are your thoughts on
anteayajesusaur: it is a dusty one of yours16:38
*** HeOS has quit IRC16:39
*** bpokorny has quit IRC16:39
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin
jesusauranteaya: it depends on which is also dusty16:39
anteayajesusaur: dusty dusty16:39
jesusaurI was waiting to see if Kiall would get that working, and then forgot about both16:39
anteayajesusaur: what do you want to do with the two of them16:40
anteayajesusaur: yeah I can understand16:40
*** ddieterly[away] is now known as ddieterly16:40
Kialloh, theres a patch I haven't thought about it a while16:40
*** dmk0202 has joined #openstack-infra16:40
fungiapuimedo: ahh, i thought you were going to add a print(inst_files_path) before /usr/lib/python2.7/site-packages/pip/req/ line 928 and then retry to pip install the kuryr source again16:41
* Kiall should abandon... 16:41
*** madhu_ak has joined #openstack-infra16:41
*** thorst_ has quit IRC16:41
fungiapuimedo: current theory is that after install completes, pip attempts to create installed-files.txt but that path is ending up not relative to /16:42
*** sdake_ has joined #openstack-infra16:42
*** tosky has quit IRC16:43
*** nadya has quit IRC16:43
apuimedofungi: I'm now with the debugger on pbr code to see where it looks for it16:43
apuimedowith python installation, not pip one16:43
*** thorst_ has joined #openstack-infra16:44
fungiwell, i know we've expressly said in the past that we don't support install by itself, and rely on additional magic in pip16:44
apuimedofungi: really? For all of pbr?16:45
fungiyep, i'm looking for that section of the pbr docs now16:45
*** tosky has joined #openstack-infra16:45
*** sdake has quit IRC16:45
fungioh, right there in the preamble16:46
fungi"Note that we don’t support the easy_install aspects of setuptools: while we depend on setup_requires, for any install_requires we recommend that they be installed prior to running install - either by hand, or by using an install tool such as pip."
pabelanger#status log osic-cloud1 is down at the moment, #osic is looking into the issue. Will update shortly.16:46
openstackstatuspabelanger: finished logging16:46
*** jsavak has joined #openstack-infra16:46
fungiapuimedo: so i guess install is expected to work, but only if you manually install any install_requires16:46
anteayapabelanger: thanks for looking into osic16:47
apuimedofungi: I always do a two step16:47
pabelangeranteaya: np16:47
fungiapuimedo: and if you use pip install instead, then you don't need to install the install_requires first16:47
anteayajesusaur: whatever you and Kiall decide on that patch series is great16:47
apuimedofirst pip install -r requirements :P16:47
apuimedofungi: truw16:47
anteayaI have to go for a walk now, apparently one of my neighbours wants something16:48
anteayaback in a while16:48
*** thorst_ has quit IRC16:48
*** piet has joined #openstack-infra16:49
*** dimtruck is now known as zz_dimtruck16:49
*** electrofelix has quit IRC16:50
*** thorst_ has joined #openstack-infra16:50
fungiapuimedo: so i'm probably in need of a reqind to the original problem statement. you were saying something about needing to install a particular file into /usr/lib/docker, and i do see there is says "copying etc/kuryr.spec -> /usr/lib/docker/plugins/kuryr"16:50
*** burgerk_ has quit IRC16:50
apuimedoyes, that works, because I use the --prefix / --root /, if I use sdist or bdist (which would probably fix the missing pbr version) I can't have that file, since those don't have prefix nor root16:52
*** gyee has joined #openstack-infra16:52
pabelangersuccess, #osic was able to fix the issue:
pabelangerthey are still debugging the problem however16:53
zaroanteaya: ohh wow, yeah.  that looks bad16:54
fungianteaya: looks like your friends in the scientific working group have reiterated their question, this time to the infra ml, on whether they can use storyboard?16:54
*** burgerk has joined #openstack-infra16:54
zaroanteaya: probably can be temporarily relieved by running git gc16:54
*** flepied has quit IRC16:55
*** thorst_ has quit IRC16:55
*** sridhar_ram has joined #openstack-infra16:55
zarofungi: anteaya pointed out that we have pretty much run out of disk space on gerrit (or on /home/gerrit2 volume).16:55
*** dmk0202 has quit IRC16:55
fungizaro: yes, there was much subsequent discussion16:56
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Make ironic-{pxe,agent}_ipmitool jobs voting
* zaro checks16:56
*** thorst_ has joined #openstack-infra16:56
fungizaro: we're also almost out of space in the db, so i'm attempting to rectify that first and then i'll fix the server filesystem space issue16:56
pabelanger#status log osic-cloud1 is coming back online. Thanks for the help #osic16:56
openstackstatuspabelanger: finished logging16:56
*** roxanaghe has joined #openstack-infra16:57
fungipabelanger: what was the issue, ultimately?16:57
*** dmk0202 has joined #openstack-infra16:57
pabelangerfungi: they made some changes to their F5 load balancer it looks like.  So reverting the changes fixed our issue.16:57
pabelangerfungi: they are still hunting for the root cause16:58
*** rhochmuth has quit IRC16:58
fungigot it. thanks!16:58
pabelangerfungi: They are discussing the issue in #osic now16:58
zarofungi, anteaya : i was planning to verify gerrit's git gc but didn't have time this week.  i'll put move it to the top of my priority for next week.  hopefully we can start running git gc to reduce rate of disk consumption16:59
*** yamamoto has joined #openstack-infra16:59
fungiapuimedo: so where is the missing pbr version problem illustrated when using install? i see it managed to figure out this was kuryr 0.1.0.dev393 when creating the egg-info16:59
*** jerryz has joined #openstack-infra16:59
*** sridhar_ram has quit IRC16:59
*** sridhar_ram has joined #openstack-infra17:00
apuimedoit happens when I try to run the software, as it likes to print its version17:00
*** thorst_ has quit IRC17:01
fungiapuimedo: got it. so starting the kuryr daemon, pbr.version throws an exception because it can't find the pbr.json file?17:01
*** dmk0202 has quit IRC17:01
*** shardy has quit IRC17:02
mordredmmm. this sounds like a fun bug17:02
*** burgerk has quit IRC17:02
*** thorst_ has joined #openstack-infra17:02
fungidoes the python system path include /lib/python2.7/site-packages?17:02
*** flepied has joined #openstack-infra17:05
*** roxanaghe has quit IRC17:06
*** yamamoto has quit IRC17:06
*** kushal has quit IRC17:07
openstackgerritNathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican
*** jistr has quit IRC17:07
*** thorst_ has quit IRC17:07
*** thorst_ has joined #openstack-infra17:08
*** apuimedo has quit IRC17:09
*** mestery has quit IRC17:10
*** mugsie has quit IRC17:10
*** yamahata has joined #openstack-infra17:12
*** roxanaghe has joined #openstack-infra17:12
*** itsuugo has quit IRC17:13
*** kushal has joined #openstack-infra17:13
*** diana_clarke has left #openstack-infra17:13
*** itsuugo has joined #openstack-infra17:15
*** jsavak has quit IRC17:16
*** bpokorny_ has quit IRC17:16
*** thorst_ has quit IRC17:17
openstackgerritMerged openstack-infra/tripleo-ci: Calculate deployment times for the Heat resources
*** uberjay_ has quit IRC17:18
*** weshay has quit IRC17:18
*** thorst_ has joined #openstack-infra17:19
*** ddieterly is now known as ddieterly[away]17:19
JayF is a project-config change, one line change, would love to get a quick review and merge if possible to fix a new job we created17:20
*** uberjay has joined #openstack-infra17:20
*** matt-borland has quit IRC17:20
*** jtomasek has joined #openstack-infra17:21
*** lezbar has quit IRC17:22
*** thorst_ has quit IRC17:23
*** flepied has quit IRC17:23
*** flepied has joined #openstack-infra17:25
*** thorst_ has joined #openstack-infra17:25
*** uberjay has quit IRC17:27
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Initial support for centos-7 jenkins slave
*** Jeffrey4l has quit IRC17:28
*** roxanaghe has quit IRC17:29
*** roxanaghe has joined #openstack-infra17:29
*** Nikolay_St has joined #openstack-infra17:29
*** inc0 has quit IRC17:29
anteayazaro: thank you17:30
*** thorst_ has quit IRC17:30
anteayafungi: yes I asked the scientific working group to go public with the question17:30
anteayafungi: so same question, better venue, they are learning17:31
*** ddieterly[away] is now known as ddieterly17:32
*** thorst_ has joined #openstack-infra17:33
*** shashank_hegde has joined #openstack-infra17:33
*** flepied has quit IRC17:34
anteayathey are a great group17:34
anteayafinding their way17:34
*** tojuvone has joined #openstack-infra17:35
*** mestery has joined #openstack-infra17:37
*** apuimedo has joined #openstack-infra17:38
*** thorst_ has quit IRC17:38
pabelangerfungi: AJaeger: project-config-core: do you mind taking a moment to review our stack to migrate away from bare-precise17:38
pabelangerhappy to report testing ubuntu-precise worked out well!17:38
*** thorst_ has joined #openstack-infra17:39
*** jsavak has joined #openstack-infra17:39
*** mugsie has joined #openstack-infra17:41
*** jang has quit IRC17:41
*** jsavak has quit IRC17:45
*** ddieterly is now known as ddieterly[away]17:45
*** yolanda has joined #openstack-infra17:46
*** jsavak has joined #openstack-infra17:46
*** thorst_ has quit IRC17:48
prometheanfireSamYaple: I'm testing your two glean patches now17:48
prometheanfirewell, building the image17:49
*** thorst_ has joined #openstack-infra17:49
*** uberjay has joined #openstack-infra17:50
openstackgerritMerged openstack-infra/system-config: Minor documentation tweaks
*** sambetts is now known as sambetts|afk17:53
*** yolanda has quit IRC17:54
*** ddieterly[away] is now known as ddieterly17:54
*** thorst_ has quit IRC17:55
*** thorst_ has joined #openstack-infra17:56
*** yarkot1 has quit IRC17:57
*** sdake_ is now known as sdake17:57
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Swift functional tests in Identity v3-only
*** thorst_ has quit IRC18:00
*** sridhar_ram has quit IRC18:00
*** thorst_ has joined #openstack-infra18:02
*** weshay has joined #openstack-infra18:02
openstackgerritAndreas Florath proposed openstack/diskimage-builder: tests/elements/fake-os: add '/tmp' as top level dir
*** ddieterly is now known as ddieterly[away]18:04
*** thorst_ has quit IRC18:06
*** kzaitsev_mb has joined #openstack-infra18:06
EmilienMis it possible for infra to delete 8.0.0 branch on openstack/puppet-aodh? I think it was created by mistake. Thanks18:07
fungiEmilienM: sure, just a sec18:07
*** yarkot1 has joined #openstack-infra18:07
*** lucasagomes is now known as lucas-afk18:07
*** tosky has quit IRC18:08
*** thorst_ has joined #openstack-infra18:08
krotscheckfungi: Did you get around to running that permissions script? If not, would you mind?18:08
fungikrotscheck: did it yesterday or the day before18:08
krotscheckfungi: Awesome, thank you.18:08
fungikrotscheck: let me know if gerrit isn't acting how you expect18:09
*** bpokorny has joined #openstack-infra18:09
*** matt-borland has joined #openstack-infra18:10
KiallHeya, any requirements-core's about for a trivial projects.txt addition?
Kialland project-config core's for ? Thanks :)18:11
*** thorst_ has quit IRC18:12
fungiEmilienM: i've deleted the 8.0.0 branch from openstack/puppet-aodh (its head was 3f0e8910b2703ed7f23ec22a8ec86e575c5f922c)18:14
*** thorst_ has joined #openstack-infra18:14
EmilienMfungi: thank you Sir. Someone (if not me) created it by mistake.18:14
*** mriedem1 has joined #openstack-infra18:15
openstackgerritMatthew Treinish proposed openstack/requirements: Change urllib3 version to urllib3>=1.15
*** mriedem has quit IRC18:17
*** thorst_ has quit IRC18:18
*** lhinds has quit IRC18:19
*** mriedem1 is now known as mriedem18:20
*** thorst_ has joined #openstack-infra18:20
*** tonytan4ever has quit IRC18:22
*** BigWillie has quit IRC18:24
*** salv-orlando has joined #openstack-infra18:24
*** inc0 has joined #openstack-infra18:25
*** BigWillie has joined #openstack-infra18:26
*** nadya has joined #openstack-infra18:27
* mordred now has driver's license18:29
* mordred would like to let everyone know that the tx driver's license stuff is AMAZINGLY well done18:30
bkeroHow so?18:30
mordredas in, you can get in line ahead of time remotely and they text you ETAs and then when you're up18:30
*** sdake_ has joined #openstack-infra18:31
fungithe fake cattle crossing on the test course are reportedly very lifelike18:31
* crinkle hopes mordred had some form of drivers license before now18:31
mordredso there is no "stand in line for 2 hours" part of the process18:31
clarkbwhat does a horseback rider with a raised arm mean?18:31
mordredcrinkle: it's a question you should not ask in too much depth ;)18:31
clarkb(real OR DL test question)18:31
crinklemordred: this is concerning18:31
bkeromordred: I'm wondering how the rental agency gave you a van without license18:31
* mordred may have still had a valid WA driver's license18:32
mordredyou konw - from having lived there years ago18:32
*** sridhar_ram has joined #openstack-infra18:32
* mordred may not have gotten his license changed in ny, la, ny, ca, or ny18:33
*** sdake has quit IRC18:33
clarkbmordred: in vancouver across the river they do first week of school stings18:34
fungiperhaps some day we'll have the option of a national driving license18:34
clarkbdrop your kid off with oregon plates, thats a paddling18:34
*** yottatsa has joined #openstack-infra18:35
*** nadya has quit IRC18:35
mordredfungi: I'm actually a fan of the state licenses - or, at least I was until they all started sharing information with each other18:35
*** yottatsa has quit IRC18:35
mordredmade it easier to just ignore silly tickets from the chesapeake bay bridge18:35
SpamapSmordred: CA has the same process for DMV btw, except they don't text you, you just make an appointment and show up and they see you within 5 minutes.18:35
mordredSpamapS: it's so much more civilized than when I was 1618:36
JayFmordred: in cali they do appointments, basically you wait ~30m if you have an appointment, all day otherwise18:36
SpamapSyeah it's not a hell hole anymore18:36
*** gordc has quit IRC18:36
SpamapSJayF: err, your DMV office sucks. ;)18:36
SpamapSI've never waited more than 5 minutes with an appointment.18:36
JayFSpamapS: All 3 I've been to (SF Fell St, Daly City, San Jose) have been that way18:36
SpamapSEven at the Hollywood DMV where there is a line of cars and people around the block every day.18:36
mordredin any case, for the first time since before the openstack project started, my driver's license actually matches my residence18:36
SpamapSmordred: be afraid18:37
fungicold toilet seats, dentist chairs and trips to dmv... call it pointless18:37
mordredSpamapS: it's weird man18:37
* JayF hopes WA DMV is not too bad, as I'll have to jump through those hoops in 2 months18:37
JayFand I think it's even more painful now that my car is paid for, b/c I'll have to move the title too18:37
bkeroOR DMV is pretty bad. It's all classic take-a-number-and-sit-down.18:37
mordredJayF: I never had problems in WA - the downtown seattle office is quite small and has not a ton of people there the times I went18:37
docaedobkero: interesting I was just going to say my OR DMV experiences have been great. But maybe less crowded in the 'burbs18:37
clarkbjayf WA is good18:37
funginc was pretty bad the last time i had to go, but that's been nearly a decade. hopefully its improved18:38
clarkbJayF title and new license was super simple18:38
bkerodocaedo: which one do you go to?18:38
clarkbthry just accept you can drive if ypu have another state dl18:38
docaedobkero: Tualatin18:38
anteayawhat is this waiting you are talking about?18:38
leifmadsenpabelanger: thanks for merging my doc changes :)18:38
*** piet has quit IRC18:38
anteayalineup? what is that?18:38
pabelangerleifmadsen: np18:38
clarkbjayf they take your photo, give you a temp and done18:39
JayFclarkb: sweet, thanks :).18:39
JayFclarkb: you in seattle metro area?18:39
clarkbJayF I was but moved back to portland a year and a half ago18:39
*** e0ne has joined #openstack-infra18:39
bkerodocaedo: sunset tc dmv was pretty terrible18:39
clarkbbkero its not bad early in the day18:39
clarkbthat is where I got the horseback rider question18:40
*** savihou has quit IRC18:40
docaedoclarkb: is the answer "the rider is saying howdy!" ?18:41
clarkbdocaedo: its something like riders ahead. I forget18:42
clarkbthe worst dmv is over by gateway18:42
clarkbthat is where I got my forst murican dl18:42
clarkbtook several days of back and forth nonsense18:42
greghaynesclarkb: docaedo its always that youre scaring the horse18:42
greghaynesguess who just got a license a few months ago18:43
clarkbat first they said my passport and micronesian dl were good then they said no I had to do all the testing and I had to go to county records office to get birth certifi ate18:43
clarkbI was not happy with them and do not intend on ever returning18:43
*** ociuhandu has quit IRC18:43
mordredgreghaynes: that's actually the real answer?18:43
mordredclarkb: no. you should not18:43
greghaynesmordred: yes18:43
*** dynarro has joined #openstack-infra18:44
greghaynesmordred: in portland we are very concerned about scaring horses18:44
docaedogreghaynes: fair enough. Have ridden horses and being on a scared one is no fun.18:44
clarkbalso in 2018 it will be useless for travel18:44
clarkbWA OR and a bunch of other states wont meet federal id requirements18:45
bkeroBecause we're not doing the RealID thing?18:45
clarkbso get a passport or pay more for real id18:45
bkerowfm, I have my GlobalEntry ID I can use anyways18:46
amrithis anyone really doing that realID stuff or will it all fall apart?18:46
amrithcan't drive across state lines18:46
bkeroclarkb: do globalentry, get one of these
bkeroEven people in the airports haven't seen them before18:46
bkerobut it's technically a federally-issued ID18:46
pabelangermy passport has RFID.... what could possible go wrong18:47
clarkbamrith an or dl is still fine for driving elsewhere just not for tsa to know who you are18:47
amriththat's interesting because I've started using my passport card for tsa as a matter of course18:47
fungiyeah, i use my usa passport card as photo id, unless i actually have to prove i'm licensed to operate a motor vehicle18:47
mordredbkero: that's an awesome idea. I should totally start doing that just to screw with the tsas18:47
amrithI've been had no trouble with that, but some building security folks don't know what a passport card is and won't accept it.18:47
bkeromordred: Haha, let me know how it works out for you :)18:47
fungiamrith: somehow i've not had any trouble there. i also use it when purchasing alcoholic beverages18:48
anteayawhat is a passport card?18:48
amrithmordred, remember that 'screwing with tsas' is on the list of famous last words like 'look honey, no hands', and 'i wonder whether this mushroom is toxic'18:49
bkero"Hold my this!"18:49
fungianteaya: the usa passport office issues them as an alternative to our passport book, though it's only good for international travel by land or water to canada, mexico and the caribbean18:49
anteayafungi: ah, interesting18:49
fungibut it's basically the size of a credit card, so easier to carry around18:50
anteayaare they less expensive? do they have a longer expiration date?18:50
* anteaya is trying to see the appeal18:50
anteayaoh okay18:50
amrithanteaya, it is easy to carry around18:50
fungiyeah they're also slightly lower cost, though that doens't matter much for me since i still need the full passport book too18:50
anteayawe have an enhanced driver's licence18:50
bkeroI figured the cards were only useful for people who lived along the border and had to cross often. Easier/safer than passport.18:51
anteayawhich allows you land access to the states18:51
anteayait appeals to drivers from canada with no desire to fly18:51
fungibkero: also convenient if you go on cruises or otherwise travel to islands in the caribbean18:52
bkerofungi: good point, I think the caribbean thing is new18:52
* bkero been meaning to go on a focus-working cruise18:52
fungior if you live in a usa protectorate in the caribbean18:52
fungisince island hopping is pretty easy there18:53
bkerosounds like fun :)18:53
fungimy wife and i have talked about relocating there at some point. st. croix is pretty cheap actually18:54
docaedobkero: you should join the Community App Catalog team, our mid-cycle is either going to be in Hawaii, or on a caribbean cruise18:54
fungicruise mid-cycles seem like they could work out well18:55
fungias long as you can line up a conference room on the ship18:55
* docaedo is kidding but hopes the idea will catch on just the same ;)18:55
fungifinding places for a group to focus can be a little challenging otherwise18:55
crinklelock everyone on a boat until the work is done, sounds productive18:56
pabelangerboattboaty mcboatface here we come18:56
fungiwe regularly go cruising with a large group of friends, and when a dozen or so of us want to find a quiet spot to have a card game or something it can be tough sometimes18:56
pabelangererr boaty*18:56
nibalizerpabelanger: cloudy mccloudface18:56
pabelangernibalizer: YES18:57
bkerodocaedo: sounds awesome :) Where do I start?18:57
funginibalizer: excellent alternative to "vanilla"18:57
nibalizeroh god18:57
nibalizerthis is why we can't trust democracy18:57
bkerofungi: do cruise ships have conference rooms?18:57
fungii have to assume so18:57
docaedoLots of rooms but little bandwidth18:57
fungithough it's a very good question18:57
fungiyou can privately book lounges for a group18:57
bkerodocaedo: sounds perfect for a meeting18:57
fungibut hard to get one all day18:58
clarkbthry didnt name the ship that18:58
fungiand yes, internet access is not great (though it's still way better than it used to be)18:58
clarkbjust the submersible18:58
anteayapabelanger: it is a sub18:58
bkeroThat's the first I'm hearing of internet access that has been more available than <1mbit for $5/min18:58
fungiclarkb: i've actually convinced christine to let me name our boat that once we get around to picking out what we want18:58
pabelangeranybody feel like +A a few ubuntu-precise changes?
pabelangeranteaya: Ya, I see that18:59
clarkbIama fan of bad pun boat names19:00
fungithe Review-MySQL trove instance in rax-dfw claims it is _still_ resizing its volume. i've opened a support ticket19:00
fungineed to accept the possibility that we might have to take a gerrit outage to mysqldump and then load into a new trove instance19:02
*** javeriak has quit IRC19:02
pabelangerfungi: ack19:02
anteayafungi: ack19:02
anteayaI'm going for a walk19:02
anteayahappy new moon in Taurus19:02
anteayaback in a while19:02
openstackgerritMerged openstack-infra/system-config: Use double quotes around bash variables
*** dynarro has quit IRC19:03
fungionce ci activity drops off a bit as the day goes on, i'm going to go ahead and start the pvmove for /home/gerrit2 onto the larger pv (i'll do it in a root screen session so other infra-root can attach and check on it in case it runs for longer than i expect)19:04
*** ddieterly[away] has quit IRC19:04
fungiit'll cause some filesystem performance degradation, so don't want to start it while people are heavily using the server19:04
*** Nikolay_St has quit IRC19:04
mordredfungi: ++19:05
*** dynarro has joined #openstack-infra19:05
nibalizerfungi: rgr19:05
fungi#status log opened support ticket 160506-iad-0001201 for Review-MySQL trove instance taking >3 hours (so far) to resize its backing volume19:06
openstackstatusfungi: finished logging19:06
*** asselin has joined #openstack-infra19:07
*** tonytan4ever has joined #openstack-infra19:07
*** ifarkas has quit IRC19:08
amrithfungi, I'm not sure how rax runs their db service but can you see the cinder volume in cinder list?19:08
amrithor are they running the nova instances and cinder volumes in some shadow tenant?19:09
nibalizerso just to confirm, looks like we're doing local mysql for storyboard?19:09
nibalizernot trove19:09
fungiamrith: i've never seen them show up in cinder list19:09
nibalizerer nvm19:10
nibalizerthe other way around19:10
mordrednibalizer: no - we should be doing19:10
*** ewilson has quit IRC19:10
mordredthat was almost a problem19:10
amrithfungi, then they're operating it with a shadow tenant. good ...19:10
funginibalizer: yes, the current storyboard production db is in a trove instance19:10
*** javeriak has joined #openstack-infra19:10
nibalizerok, so the dev server can just have a dev trove yah?19:11
mordredamrith: while we're talking to you about trove - one of the things we'd love to see is ingress and egress replication19:11
mordredamrith: at different times we've needed both ...19:11
*** e0ne has quit IRC19:11
mordredamrith: ingress as in "I have a mysql server in a nova instance but I want to migrate to using a trove instance, so I'd like to set up the trove instance as a replication slave of my current instance and then do a short failover"19:12
amrithmordred, replication on the data injest side you mean?19:12
funginibalizer: there's already a StoryBoard-Dev-MySQL trove instance in dfw for the openstackci tenant. pretty sure i created that in preparation for the server which never materialized19:12
*** ddieterly has joined #openstack-infra19:12
amriththat would be awesome19:12
*** ociuhandu has joined #openstack-infra19:12
fungialso continuous replication from trove in one provider/region to trove in another19:12
mordredamrith: egress as in "I want to run another mysql $somewhere not in this trove and replicate to it - perhaps for reporting or backups or whatever"19:12
fungiyeah, that19:12
nibalizeroh neat19:13
amrithmordred, will log those ...19:13
fungiwhich could also be trove as teh source, or just mysql in a nova instance or on a real server somewhere or...19:13
nibalizerfungi: well there is a review up to create a dev server19:13
mordredamrith: and finally - I've love to be able to do the "run my schema upgrade on a slave, then promote the slave to master" all inside of trove19:13
mordredamrith: and/or "do a schema upgrade on one node of a galera cluster at a time"19:13
nibalizerso one day we might even make the server19:13
amrithmordred, I think that works now with standard replication19:13
amrithas in mysql replication19:13
*** flwang1 has joined #openstack-infra19:14
amrithI'm not sure that this is a valid thing with galera because with galera I think it'll replicate the upgrade, no?19:14
mordredthere are two different galera settings19:14
mordredone is total order isolation which is when you want to make sure the upgrade happens everywhere19:14
amrithyou speak of RSU19:14
funginibalizer: yep! just saying it was a half-completed effort, so there are some things (like the trove instance) already done. i probably also added it to hiera, but if so then it likely got lost again when we split up the hiera file since keys not referenced in the global site manifest weren't included in the new split hiera files19:14
amrithI guess19:14
mordredthere's also RSU19:14
amrithTOI is the other19:15
amrithso trove now supports galera19:15
amrithuse the pxc guest image19:15
mordredcool. I will investigate doing this in our current account19:15
amrithiirc you can change modes with something like19:15
funginibalizer: though you may be able to git log -p in /etc/puppet/hieradata and keyword search for it19:16
amrithset wsrep_osu_meth = 'RSU'19:16
amrithor some such thing19:16
amrithmy galera knowledge is a bit rusty19:16
amrithonce you do that if you get the nodes to all go to RSU, I think it should work.19:16
amrithI wrote that down on my lsit of things to try19:16
amrithwill get back to you19:16
*** gordc has joined #openstack-infra19:17
*** ewilson has joined #openstack-infra19:18
*** roxanaghe has quit IRC19:18
fungiamrith: yeah, that's probably our biggest headache with using trove at this point. we want cross-provider backups of our data, and currently do it in a very ugly and performance-impacting fashion by running a full mysqldump on a nova instance and then performing a network copy to another nova instance somewhere else19:20
fungiso every day at 00:00 utc, gerrit and etherpad and other things become unresponsive for several minutes19:21
fungiand if we could continuously replicate to a read-only trove instance elsewhere instead, we could just take scheduled snapshots of that or something19:22
amrithfungi, where's your trove instance physically running? rax?19:22
fungi(or pause replication and take a mysqldump of the read-only replica or whatever)19:22
fungiamrith: yeah, rackspace dfw right now19:22
*** madhu_ak has quit IRC19:22
*** cody-somerville has joined #openstack-infra19:22
amrithso I know this isn't ideal, but here's a thought19:23
amrithcan you setup a second trove instance somewhere and setup replication manually?19:23
amrithafter all, everything you need to setup replication goes over the standard mysql ports.19:23
amrithand you can set it up with the SQL interface.19:24
fungiamrith: oh, really? that's a thought19:24
amrithI'd start with a painful mysqldump with a transaction id19:24
*** flwang1 has quit IRC19:24
amrithfungi, it is a cludge19:24
amrithor is that kludge?19:24
fungialso sometimes kluge19:24
amriththat thing :)19:24
*** sdake has joined #openstack-infra19:24
*** ewilson has quit IRC19:25
fungiloan words are fun19:25
*** sdake has quit IRC19:25
amrithwe're working on snapshot replication in newton19:26
*** rcernin has joined #openstack-infra19:26
*** ewilson has joined #openstack-infra19:26
amrithyou'll be able to take a cinder snapshot19:26
amrithand use that as a backup ...19:26
*** flwang1 has joined #openstack-infra19:26
*** sdake_ has quit IRC19:26
mordredamrith: so - I thought last time I looked in to doing that19:28
mordredthere was something I couldn't do19:28
mordredamrith: but I think the problem was that was before trove supported replication at all19:28
fungiamrith: when you say setup replication manually, you mean replication can be configured via the sql interface, or you mean "replicate" by taking a periodic (incremental?) mysqldump of the source and then push that into the destination as an update?19:28
mordredso my problem was getting binlogging turned on19:28
amrithyes, you'll need logging turned on19:29
mordredfungi: no - the dump is a thing you do to get the slave started19:29
amrithso here's how you get around that ...19:29
mordredfungi: if you dump with the replicatoin info included19:29
fungiaha, now i understand19:29
amrithwell, you are in RAX19:29
fungiso just bootstrap via a mysqldump with a transaction id so it knows where to continue from19:29
amriththeir trove is slightly different from upstream19:29
mordredamrith: of course it is19:29
amrithno, I mean forked different19:30
amrithand maybe not up to date with upstream19:30
amrithso assume ...19:30
mordredI can think of literally zero parts of openstack that that is not the case with there19:30
amriththat you went to (say bb)19:30
fungiamrith: wait, are you suggesting that rackspace might run non-upstream versions of openstack services?19:30
amrithyou could launch instance 119:30
amriththen launch instance 2 replica of instance 119:30
amriththat causes binlogging to get turned on19:30
amrithnow in some other cloud far away, launch a trove19:30
amrithand manually configure it to be a slave19:30
amrithwon't that work?19:30
amrithI don't know what would happen in rax, haven't used their db service in a while.19:31
amrithmordred, let me try this out and let you know19:31
amritheasy enough for me to spin up trove instances19:31
amrithoh, wait. would you know how I have to build guest images for trove? :>19:32
mordredamrith: hahah19:32
mordredamrith: it's ok - that's great info and enough for me to go poking19:32
amrithnp, got me curious now. am just launching a couple ...19:33
*** yonglihe has quit IRC19:33
*** madhu_ak has joined #openstack-infra19:33
*** Guest78215 has quit IRC19:34
*** salv-orlando has quit IRC19:39
*** apoorvad has quit IRC19:40
*** salv-orlando has joined #openstack-infra19:42
*** cody-somerville has quit IRC19:46
mtreinishclarkb: you're not in -qa, do you have any ideas on how to make: faster19:47
mtreinishright now it's taking about 4min running from my laptop19:47
*** hdaniel has joined #openstack-infra19:47
mtreinishI'm thinking we might not be able to do that dynamically19:47
*** inc0 has quit IRC19:48
*** piet has joined #openstack-infra19:49
*** teran has quit IRC19:52
*** igorbelikov has quit IRC19:52
openstackgerritMerged openstack-infra/jenkins-job-builder: builders: update cmake to match CMake plugin versions 2.x
*** igorbelikov has joined #openstack-infra19:54
openstackgerritEmilien Macchi proposed openstack-infra/project-config: add reno jobs for puppet-ceph
*** teran has joined #openstack-infra19:55
*** thorst_ has quit IRC19:55
openstackgerritMerged openstack-infra/project-config: Add ubuntu-precise to bindep-fallback
*** thorst_ has joined #openstack-infra19:57
*** dprince has quit IRC19:59
*** thorst_ has quit IRC20:00
amrithmordred, don't think it is possible to do what I suggested without access to the shell20:00
amrithand I don't think rax gives you that.20:00
openstackgerritMerged openstack-infra/project-config: Replace bare-precise with ubuntu-precise in infra-puppet-apply-jobs
*** jheroux has quit IRC20:01
openstackgerritMerged openstack-infra/project-config: Convert gate-project-config-layout to ubuntu-precise
pabelangerAJaeger: thanks!20:02
*** thorst_ has joined #openstack-infra20:02
mordredamrith: they do not20:03
mordredamrith: I have spun up an ha group at rax20:03
mordredalthough I have not played with it much yet20:03
mordredbut I'll report back when I do20:04
amriththx mordred20:04
amritham curious how to get around that now.20:04
pabelangermordred: do you have a moment to check the FIPs in osic?  I think they are leaking20:05
*** dmk0202 has joined #openstack-infra20:05
mordredpabelanger: yup!20:05
mordredwhy yes! there are many leaked fips20:06
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Add semi-colon as separator for parsing package names
*** kjorgensen has quit IRC20:06
pabelangerYa, osic has an issue this morning. I tried cleaning them up once before, but looks like they leaked out again20:06
mordredpabelanger: Cleaned 58 FIPs from osic-cloud120:07
mordredCleaned 22 FIPs from bluebox-sjc120:07
openstackgerritStephen Lowrie proposed openstack/os-testr: Add subunit-describe-calls
pabelangermordred: okay, just rechecking 308376 your FIP script20:07
pabelangerwill try that next time20:08
mordredpabelanger: and
mordredpabelanger: woot20:08
*** javeriak has quit IRC20:10
openstackgerritMerged openstack-infra/project-config: Set min-ready 0 for bare-precise
*** thorst_ has quit IRC20:11
*** salv-orlando has quit IRC20:13
*** thorst_ has joined #openstack-infra20:13
*** twm2016 has joined #openstack-infra20:13
*** apoorvad has joined #openstack-infra20:13
*** aysyd has quit IRC20:14
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: docs: add missing :jenkins-wiki: macro to CMake builder
twm2016This is kind of a reach here but I'm working a unit test refactoring from mox to mock. Does anyone know of any guides for this? I don't know too much about either of the projects but I want to learn. I found some side by side comparisons for basic commands in mox and mock but that doesn't help much when I get into the code.20:16
openstackgerritMerged openstack-infra/project-config: Add VirtualBMC to gerritbot
openstackgerritStephen Lowrie proposed openstack/os-testr: Add subunit-describe-calls
AJaegertwm2016: this is the infra channel, not a generic development one. Better ask on #openstack-dev20:17
*** thorst_ has quit IRC20:18
*** twm2016 has left #openstack-infra20:18
*** thorst_ has joined #openstack-infra20:19
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Ironic functional tests in Identity v3-only
*** cody-somerville has joined #openstack-infra20:20
*** keedya has joined #openstack-infra20:24
*** thorst_ has quit IRC20:24
*** hdaniel has quit IRC20:24
*** thorst_ has joined #openstack-infra20:25
openstackgerritamrith proposed openstack-infra/project-config: Trove client didn't have release notes in mitaka
*** thorst_ has quit IRC20:26
*** thorst_ has joined #openstack-infra20:26
openstackgerritamrith proposed openstack-infra/project-config: Trove client didn't have release notes in mitaka
*** pt_15 has joined #openstack-infra20:29
*** Sukhdev has joined #openstack-infra20:29
*** e0ne has joined #openstack-infra20:29
pabelanger#success bare-precise has been replaced by ubuntu-precise. Long live DIB20:30
openstackstatuspabelanger: Added success to Success page20:30
*** kushal has quit IRC20:30
*** BigWillie has quit IRC20:30
*** openstackgerrit has quit IRC20:32
*** openstackgerrit has joined #openstack-infra20:33
*** ddieterly is now known as ddieterly[away]20:34
*** rfolco has quit IRC20:35
openstackgerritMerged openstack-infra/jenkins-job-builder: docs: add missing :jenkins-wiki: macro to CMake builder
* dhellmann makes a note to nominate pabelanger for Most Creative Notice Author contributor award this cycle20:37
morgandhellmann: ++20:37
dhellmannmorgan : he's on a roll today20:37
morgandhellmann: so, i am officially all setup with red hat now :)20:38
morgandhellmann: and getting ready to hop a plane back home ^_^20:38
*** cody-somerville has quit IRC20:38
dhellmannmorgan : did you get a fedora or are they still out?20:38
morgandhellmann: still out20:38
openstackgerritLucas Dutra Nunes proposed openstack-infra/jenkins-job-builder: SCM repo: add ignore-projects option
dhellmannmorgan : well, welcome to the team!20:38
pabelangerdhellmann: Ha, thanks20:38
morgandhellmann: sounded like quite the trial to get those hats.20:38
*** e0ne has quit IRC20:39
pabelangermorgan: welcome!20:39
*** flwang1 has quit IRC20:39
*** jsavak has quit IRC20:39
*** e0ne has joined #openstack-infra20:40
*** e0ne has quit IRC20:40
*** Sukhdev has quit IRC20:40
*** jsavak has joined #openstack-infra20:40
*** Sukhdev has joined #openstack-infra20:41
morganpabelanger: ^_^20:41
smcginnis_So maybe you smart people can suggest what to do here.20:41
*** kgiusti has quit IRC20:41
smcginnis_We have a non-voting python-cinderclient job failing for pypy.20:41
smcginnis_Turns out the paramiko library updates recently to require cryptography 1.1.20:41
smcginnis_Which has an explicit check to not allow pypy < 2.6.20:42
smcginnis_Our pypy test I believe just installs the pypy version for the given platform.20:42
openstackgerritMerged openstack-infra/project-config: add reno jobs for puppet-ceph
smcginnis_Which for trusty is
smcginnis_So I'm wondering - is there either a way to require a later pypy? Or is there a better solution to this?20:43
fungii had one of those red fedoras years ago, but eventually gave it away. was promotional swag because my employer bought so many support licenses20:43
fungilike, somewhere back around 1999/200020:44
*** gordc has quit IRC20:44
*** aysyd has joined #openstack-infra20:44
fungismcginnis_: could pin to paramiko<2 maybe? though rsn we'll be running newton jobs on ubuntu 16.04 lts which likely has new enough pypy20:45
smcginnis_fungi, Yeah, 16.04 might be the answer. At least to get it passing in gate.20:46
smcginnis_I'd hate to cap paramiko since there's good reasons to upgrade it for security updates.20:46
fungiyep, though also as you noted it drags in cryptography, which makes pip install without prebuilt wheels more painful (needs to link against local headers for openssl, cffi, et cetera)20:48
smcginnis_I closed the tab now, but I believe Xenial includes the newer pypy by default, so I'm assuming it has all the necessary dependencies included in that.20:49
*** baoli_ has quit IRC20:50
*** thorst_ has quit IRC20:52
*** thorst_ has joined #openstack-infra20:53
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Improve pot publishing
*** grauzikas has quit IRC20:53
AJaegerproject-config cores, I missed that some repos have duplicated pot files in different paths. Could you review and merge the change above quickly, please? ^20:54
*** grauzikas has joined #openstack-infra20:54
fungiAJaeger: yep, saw the ml thread. wondered what the fix might be for that20:55
AJaegeronce the change is in, should be deleted - we update it with each change, so brute force is fine.20:56
AJaegerfungi, please check my logic. Hope it's fine now. I gave a lengthy comment...20:57
fungiAJaeger: hemanthm is also looking at making a periodic job to generate glance sample configs, so i pointed him at your potfile work as an example of how we're getting away from pushing things like that into git repos and suggested maybe publishing them on tarballs.o.o instead like that20:58
*** ddieterly[away] is now known as ddieterly20:58
*** jsavak has quit IRC20:58
*** dmk0202 has quit IRC20:58
AJaegerfungi, the suggestion to use same path is the best IMHO, see which shows the path20:58
*** amitgandhinz has quit IRC20:58
AJaegerfungi, yeah, this should be fine and we could reuse this for other repos that want sample configs20:59
*** ddieterly is now known as ddieterly[away]20:59
*** pcrews__ has joined #openstack-infra20:59
AJaegerregarding proposal jobs: is the project-config review guideline, it mentions those as we discussed in Austin.21:00
pabelangerHere is an interesting exception from nodepool:
*** thorst_ has quit IRC21:01
AJaegeranteaya: thanks for review.21:01
anteayaAJaeger: you are welcome21:01
*** Sukhdev has quit IRC21:02
*** pcrews_ has quit IRC21:02
*** ddieterly[away] is now known as ddieterly21:02
*** thorst_ has joined #openstack-infra21:03
fungiAJaeger: the potfile fix for horizon looks fine to me21:05
fungiAJaeger: and you need all of translation-source deleted from tarballs.o.o?21:06
*** baoli has joined #openstack-infra21:06
AJaegerfungi, yes, remove the whole directory...21:06
AJaegerWe'll populate it again ;)21:07
AJaegerthanks, fungi!21:07
fungidone (well, moved to my homedir temporarily in case you need it back for some reason)21:07
*** kzaitsev_mb has quit IRC21:07
fungithough i guess we might need to do that again if anything merges to trigger another update between now and when that change propagates to the worker21:07
AJaegerI shouldn't need it - it's all created data... But good to know - feel free to delete on Monday if I didn't ask for it.21:08
*** thorst_ has quit IRC21:08
AJaegeryeah, this might need redoing, let's see21:08
*** ddieterly is now known as ddieterly[away]21:08
fungialso, rackspace support just triaged our gerrit db stuck resize ticket, escalating it to their trove group looks like21:08
fungino details yet21:08
fungii'm going to pop out for an early dinner, then check back in and start the pvmove of /home/gerrit2 once i get back21:09
AJaegerthanks, fungi!21:09
dimspabelanger fungi : quick question, is bandersnatch back on its feet yet?21:09
dimsand have a good weekend :)21:09
*** thorst has joined #openstack-infra21:10
pabelangerdims: yes, we've switched to the RW mirror for the time being21:10
dimspabelanger : so the sync should be complete by now?21:10
pabelangerdims: They won't be atomic if I understand correctly, but they are up to date21:10
dimspabelanger : ack i'll kick the tires21:11
pabelangerdims: sure, let us know if there are problems21:11
dimsyou bet pabelanger !21:11
*** ddieterly[away] is now known as ddieterly21:12
openstackgerritMerged openstack-infra/project-config: Improve pot publishing
*** psilvad has quit IRC21:13
*** MarkAtwood has quit IRC21:14
zigopabelanger: Hi there!21:14
pabelangerzigo: hello21:15
zigopabelanger: It's been a week since AJaeger helped me with this:
zigo(on the Friday at the summit)21:15
zigoSo probably now you can have a look?21:15
*** ldnunes has quit IRC21:15
zigoNobody reviewed it yet...21:15
pabelangerzigo: sure, I'll check it out this weekend21:16
*** baoli has quit IRC21:16
zigomordred: Maybe you could look into it too?21:16
*** xyang1 has quit IRC21:16
zigopabelanger: How will the other side (ie: the pick-up job) should be done?21:16
pabelangerzigo: I can write the script to do that. Looks like you are using the git sha1 hash, so should be straight forward21:17
zigoI did yes.21:17
pabelangerzigo: we likely want to move the upload par out of your script, since other jobs could use it.  But should be safe place to start21:17
zigoSo, tarballs/debian/deb-openstack-pkg-tools/<sha256>/*21:17
*** roxanaghe has joined #openstack-infra21:18
pabelangerYa, I just need to see what the directory script on tarballs.o.o will look like21:18
pabelangerwe likely want tarballs.o.o/deb-openstack-pkg-toosl/<sha256>/*21:18
pabelangerminus the typo of tools21:19
zigopabelanger: Something like this:
*** thorst has quit IRC21:19
zigoMinus the .build file, which I'm deleting on purpose (we already have the build log, it'd be redondant...)21:19
*** ayoung has quit IRC21:20
*** roxanaghe has quit IRC21:20
*** MarkAtwood has joined #openstack-infra21:20
*** thorst has joined #openstack-infra21:20
zigopabelanger: What was the thing you were working on this week btw?21:20
*** regXboi has quit IRC21:21
zigoFor me, I've been sick all the way since I'm back from the summit (right after I reached home), so I didn't do anything... :(21:21
*** roxanaghe has joined #openstack-infra21:22
*** Sukhdev has joined #openstack-infra21:23
*** eharney has quit IRC21:24
*** aysyd has quit IRC21:24
*** lakshmiS has quit IRC21:24
*** ddieterly has quit IRC21:28
*** thorst has quit IRC21:28
*** vilobhmm11 has joined #openstack-infra21:28
*** ewilson has quit IRC21:29
*** vilobhmm11 has quit IRC21:30
*** vilobhmm11 has joined #openstack-infra21:30
*** matt-borland has quit IRC21:32
*** vilobhmm11 has quit IRC21:33
*** leakypipes has quit IRC21:35
*** jcoufal has quit IRC21:35
*** thorst has joined #openstack-infra21:36
EmilienMAJaeger: you never sleep!21:37
EmilienMthanks for the quick reviews21:37
*** sigmavirus24 is now known as sigmavirus24_awa21:38
*** mminesh has joined #openstack-infra21:39
*** thorst has quit IRC21:40
mmineshTrying to run git review for the first time and getting "fatal: ICLA contributor agreement requires current contact information.21:40
mminesh Please review your contact information:21:40
mminesh I have tried updating the info on the link but getting server error....21:40
mmineshany help?21:40
mmineshtest was run against the sandbox project before i perform any actual commit..but i have been unsuccessful so far21:41
*** thorst has joined #openstack-infra21:41
anteayamminesh: did you follow all the steps here:
*** thorst has quit IRC21:43
anteayamminesh: if so then read this
*** thorst has joined #openstack-infra21:43
*** mriedem has quit IRC21:44
*** vgridnev has joined #openstack-infra21:44
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Really fix translations copy
AJaegerEmilienM: I wanted to - but first needed to fix translation copying ;)21:45
AJaegerfungi, anteaya see 313707, hope that works better now...21:45
AJaegerCurrently all translation pushes fail ;(21:45
*** amrith is now known as _amrith_21:45
*** thiagop has quit IRC21:47
AJaegerproject-config cores, please review 313707 for translation push - sorry, I missed the zanata-cli behaviour of looking into all directories including .directories ;(21:47
* AJaeger calls it a day...21:48
AJaegerthanks & bye21:48
*** HeOS has joined #openstack-infra21:48
*** salv-orlando has joined #openstack-infra21:51
*** pradk has quit IRC21:52
*** davideagnello has quit IRC21:53
*** MarkAtwood has quit IRC21:57
*** tphummel has quit IRC21:58
*** piet has quit IRC21:59
openstackgerritMerged openstack/requirements: Require Babel>=2.3.4
*** thorst has quit IRC22:04
*** ewilson has joined #openstack-infra22:07
*** dynarro has quit IRC22:09
*** Sukhdev has quit IRC22:10
openstackgerritJay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
*** ayoung has joined #openstack-infra22:16
*** ewilson has quit IRC22:17
*** thorst has joined #openstack-infra22:17
*** ewilson has joined #openstack-infra22:17
*** rhallisey has quit IRC22:18
*** vilobhmm11 has joined #openstack-infra22:19
openstackgerritOpenStack Proposal Bot proposed openstack/os-testr: Updated from global requirements
*** thorst has quit IRC22:22
*** dynarro has joined #openstack-infra22:22
*** keedya has quit IRC22:24
*** tonytan4ever has quit IRC22:24
*** tphummel has joined #openstack-infra22:31
*** kzaitsev_mb has joined #openstack-infra22:32
*** roxanaghe has quit IRC22:33
*** vgridnev has quit IRC22:34
*** david-lyle has quit IRC22:35
*** ewilson has quit IRC22:37
kevinbentonis someone around familiar with devstack? i'm troubleshooting some heat gate failures and i have the issue tracked down to the recent removal of the VPNaaS code from devstack22:37
*** david-lyle has joined #openstack-infra22:37
*** vhosakot has quit IRC22:45
*** sdake has joined #openstack-infra22:45
*** markvoelker has quit IRC22:47
*** sdake_ has joined #openstack-infra22:48
*** dynarro has quit IRC22:48
*** markvoelker has joined #openstack-infra22:48
*** rbrndt has quit IRC22:49
*** sdake has quit IRC22:50
*** ewilson has joined #openstack-infra22:51
samueldmqanteaya: sorry I was afk earlier, thanks for the heads up22:52
*** r-daneel has quit IRC22:52
samueldmqanteaya: re: patch 26439822:52
openstackgerritJay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs
fungikevinbenton: you may have more luck asking in #openstack-qa since devstack is a qa project22:54
kevinbentonfungi: thx22:55
*** sdake has joined #openstack-infra22:55
anteayasamueldmq: sure22:55
*** jamesmca_ has joined #openstack-infra22:56
*** sdake_ has quit IRC22:56
fungiso... gerrit database resize update is that apparently the resize succeeded but for some reason status metadata didn't update correctly so the api is still showing it as resizing22:57
fungii'm going ahead with the pvmove for our gerrit data now22:57
*** tpsilva has quit IRC22:57
fungireminder that it's running in a root screen session on review.o.o22:57
fungialready 1% done so this may not take long22:58
*** signed8bit is now known as signed8bit_Zzz22:58
*** _amrith_ is now known as amrith23:00
fungi10% done23:00
*** jamesmca_ has quit IRC23:01
openstackgerritMerged openstack-infra/project-config: Really fix translations copy
JayFIf someone has a spare moment, is a one line change to add pyghmi to projects for Ironic jobs so a new job we added to test it will pass23:03
JayFalready has 1x+2 just needs one more.23:03
*** ams_ has quit IRC23:03
*** yamahata has quit IRC23:04
*** sdake has quit IRC23:05
*** ewilson has quit IRC23:06
*** sridhar_ram has quit IRC23:08
*** roxanaghe has joined #openstack-infra23:09
*** mixos has quit IRC23:10
openstackgerritMerged openstack-infra/project-config: Add openstack/pyghmi to the $PROJECTS variable
*** daemontool has quit IRC23:12
*** signed8bit_Zzz is now known as signed8bit23:13
fungiJayF: ^23:13
JayFthanks a bunch fungi, I know lucas will be glad that's merged when he gets back23:14
fungilucas-afk: ^23:15
*** ddieterly has joined #openstack-infra23:16
fungiwe're already at 70% on the pvmove, so should be done soon and then i can clean up the old pv and expand the lv23:17
openstackgerritDavid Lyle proposed openstack/requirements: Remove XStatic-QUnit
*** e has joined #openstack-infra23:22
*** ddieterly is now known as ddieterly[away]23:24
*** Jeffrey4l has joined #openstack-infra23:26
*** tphummel has quit IRC23:27
*** kzaitsev_mb has quit IRC23:29
openstackgerritDavid Lyle proposed openstack/requirements: Remove XStatic-Angular-FileUpload
*** sdake has joined #openstack-infra23:34
*** edmondsw has quit IRC23:34
*** xarses has quit IRC23:34
openstackgerritJeffrey Zhang proposed openstack-infra/project-config: Enable the Ubuntu binary gate on liberty branch
*** akshai has quit IRC23:35
*** signed8bit is now known as signed8bit_Zzz23:36
openstackgerritJay Faulkner proposed openstack-infra/project-config: Explicitly define if ipxe is enabled or not
openstackgerritJay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs
*** sdake_ has joined #openstack-infra23:37
*** sdake has quit IRC23:38
*** rossella_s has quit IRC23:40
openstackgerritSteven Dake proposed openstack-infra/project-config: Make oraclelinux jobs for Kolla gate always run
*** rossella_s has joined #openstack-infra23:41
pabelangerclarkb: I figure out the issue with nodepool. We incorrectly set the image-update, we inverted the minutes and hours23:44
openstackgerritPaul Belanger proposed openstack-infra/project-config: Run image-update at 10:34hrs UTC
pabelangerclarkb: ^23:45
anteayafungi: ack23:46
openstackgerritDavid Lyle proposed openstack/requirements: Remove XStatic-JQuery.Bootstrap.Wizard
JayFWhat is the difference/split between zuul/layout.yaml and jenkins/jobs/projects.yaml23:46
JayFIt seems like they contain somewhat duplicated information?23:46
fungiand it's done, so removing the old volume from the vg and then detaching it from the instance23:46
anteayapabelanger: ah it wasn't getting to the 34th hour, hence was not updating23:47
pabelangerJayF: one is zuul configuration, the other is jenkins.  We use 2 different applications to process the configuration23:47
anteayafungi: yay done23:47
fungiJayF: one configures uul (our job scheduler) and the other configures jenkins (our job runner). different pieces of software23:47
JayFBut the configs should be relatively in sync then, right?23:47
pabelangeranteaya: Ya, its running now. So, 34hrs since last update23:47
* anteaya hands fungi a z to put at the front of the uul23:47
anteayapabelanger: oh, okay now I understand23:48
fungiJayF: jenkins job configurations are the details of what runs, zuul configuration says when to run them23:48
JayFthis makes sense with why my change was failing, I think I can fix this up, cool23:48
pabelangeranteaya: ya, confusing for sure.  I just noticed it running again23:48
*** woodster_ has quit IRC23:48
anteayapabelanger: good eye23:48
fungiJayF: with zuul v3 (and to some extend 2.5) this gets less confusing as jenkins will be out of the picture23:49
fungii can't type tonight23:49
JayFyeah, I think I'm more confused now :/23:49
JayFthe "project" something is defined under in jenkins/jobs/projects.yaml don't matter, it seems?23:49
*** cody-somerville has joined #openstack-infra23:49
fungiJayF: right, it's a logical aggregation within jenkns23:49
fungiand somewhat arbitrary23:50
fungiwe have "projects" defining jobs which run under other git repos than their own in many cases23:50
JayFSo my patch I thought was done, I'm just getting to the hard part it seems23:51
JayFso going to save that for later, I think23:51
*** gyee has quit IRC23:53
*** Guest34798 has quit IRC23:54
*** |-paul-| has joined #openstack-infra23:55
*** signed8bit_Zzz is now known as signed8bit23:55
fungiokay, i've got /home/gerrit2 moved onto a 200gb ssd cinder volume and expanded so that it's now only ~47% full. should reflect that shortly23:55
*** salv-orlando has quit IRC23:56
fungii've also cleaned up and detached/deleted the old 100gb volume23:56
anteayathank you23:56
*** akshai has joined #openstack-infra23:57
pabelangerfungi: will have to check history on review.o.o to see how you did that with out stopping gerrit23:57
anteayacacti indeed reflects the change23:57
fungi#status log the Review-MySQL trove instance has now been expanded to 50gb (19% full) and /home/gerrit2 on increased to 200gb (47% full)23:57
openstackstatusfungi: finished logging23:57
fungipabelanger: create new cinder volume and have nova attach it, check dmesg to see what the device name was since you can't trust the api23:57
openstackgerritKevin Benton proposed openstack-infra/project-config: Use neutron-vpnaas grenade plugin in grenade jobs
fungiold 100gb volume was /dev/xvdb and new one was /dev/xvdc23:58
pabelangerfungi: Ah, that makes sense23:58
fungicommands i ran on review.o.o were as follows:23:58
*** akshai_ has joined #openstack-infra23:58
fungisudo parted /dev/xvdc1 mklabel msdos mkpart primary 0% 100% set 1 lvm on23:59
fungisudo pvcreate /dev/xvdc123:59
fungisudo vgextend main /dev/xvdc123:59
fungisudo pvmove /dev/xvdb1 /dev/xvdc123:59

Generated by 2.14.0 by Marius Gedminas - find it at!