Monday, 2016-01-18

*** nmagnezi has quit IRC00:01
jheskethclarkb: If you're still around, some help moving along this week would be great00:04
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: (WIP) Upload metadata for pseudo folder
openstackgerritJoshua Hesketh proposed openstack-infra/project-config: Send instructions to workers to upload metadata
*** aeng has joined #openstack-infra00:07
*** aeng has quit IRC00:25
*** mudassirlatif has joined #openstack-infra00:25
daemontoolclarkb,  that was part of the issue, thank you00:26
daemontool: )00:27
*** yamamoto has quit IRC00:27
*** Jeffrey4l has quit IRC00:28
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add a helper function to convert xml
*** mudassirlatif has quit IRC00:37
*** dims has joined #openstack-infra00:39
*** gmann has joined #openstack-infra00:43
*** thorst has joined #openstack-infra00:47
*** thorst has quit IRC00:48
*** thorst has joined #openstack-infra00:49
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for testfairy plugin
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add helper function to convert xml
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Remove unused setUp condition
openstackgerritSteve Martinelli proposed openstack-infra/project-config: bump timeout for tempest-dsvm-keystone-eventlet-full
stevemarsomething wrong with: ?00:53
*** thorst has quit IRC00:53
*** dims has quit IRC01:01
*** otter768 has joined #openstack-infra01:03
*** sabeen3 has joined #openstack-infra01:03
*** davidlenwell has quit IRC01:05
*** sabeen has quit IRC01:05
*** salv-orlando has quit IRC01:06
*** davideagnello has joined #openstack-infra01:06
*** salv-orlando has joined #openstack-infra01:07
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Omit subunit2sql/tests from coverage
*** otter768 has quit IRC01:07
*** davidlenwell has joined #openstack-infra01:07
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Added Test Selector Plugin support
*** tlian2 has joined #openstack-infra01:10
*** tlian has quit IRC01:11
*** davideagnello has quit IRC01:12
*** bardia has quit IRC01:12
*** lerty_ has joined #openstack-infra01:12
*** lerty_ has quit IRC01:12
*** dimtruck is now known as zz_dimtruck01:15
*** lerty has joined #openstack-infra01:17
*** lerty has left #openstack-infra01:17
*** rfolco has joined #openstack-infra01:18
*** rfolco has quit IRC01:18
*** baoli has quit IRC01:20
*** dingyichen has quit IRC01:23
*** fedexo has joined #openstack-infra01:25
*** aeng has joined #openstack-infra01:26
*** tjones has quit IRC01:26
*** sonali5 has joined #openstack-infra01:29
*** amotoki has joined #openstack-infra01:30
*** amotoki has quit IRC01:32
*** baoli has joined #openstack-infra01:33
*** Daisy has joined #openstack-infra01:33
*** MarkAtwood has quit IRC01:34
openstackgerritMerged openstack/requirements: Updated from global requirements
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
*** yamamoto_ has joined #openstack-infra01:42
*** MarkAtwood has joined #openstack-infra01:45
*** bpokorny has joined #openstack-infra01:46
*** miqui has quit IRC01:49
*** amotoki has joined #openstack-infra01:50
*** zhurong has joined #openstack-infra01:50
*** thorst has joined #openstack-infra01:51
*** Sukhdev has joined #openstack-infra01:51
*** MarkAtwood has quit IRC01:57
*** thorst has quit IRC01:58
*** changzhi has joined #openstack-infra02:02
*** amotoki has quit IRC02:02
*** yuanying has quit IRC02:04
*** sonali5 has quit IRC02:05
*** yuanying has joined #openstack-infra02:05
*** MarkAtwood has joined #openstack-infra02:05
*** Qiming has joined #openstack-infra02:06
*** mriedem has quit IRC02:06
*** Jeffrey4l has joined #openstack-infra02:07
*** Jeffrey4l has quit IRC02:07
*** sonali5 has joined #openstack-infra02:07
*** Jeffrey4l has joined #openstack-infra02:07
*** aeng has quit IRC02:10
*** binbincong has joined #openstack-infra02:13
*** jamielennox is now known as jamielennox|away02:14
*** yamahata has joined #openstack-infra02:15
*** zz_dimtruck is now known as dimtruck02:20
*** yamahata has quit IRC02:22
openstackgerritIWAMOTO Toshihiro proposed openstack/requirements: Bump ryu to 3.29.1
*** PsionTheory has quit IRC02:24
*** markvoelker_ has quit IRC02:26
*** fedexo has quit IRC02:27
*** bpokorny has quit IRC02:29
openstackgerritIan Wienand proposed openstack-infra/project-config: Prune old branchs in early devstack clone
ianwclarkb: ^ i think that is the right fix.  super confusing when you're grepping for logs/commands, and they show up in DIB so you think you're fixing it in the right place!02:38
*** MarkAtwood has quit IRC02:39
*** binbincong has quit IRC02:39
*** sonali5 has quit IRC02:41
*** pahuang has joined #openstack-infra02:42
*** dingyichen has joined #openstack-infra02:44
*** sonali5 has joined #openstack-infra02:44
*** hdd has joined #openstack-infra02:45
*** eliqiao_ has joined #openstack-infra02:47
*** amotoki has joined #openstack-infra02:48
*** bpokorny has joined #openstack-infra02:48
*** eliqiao has quit IRC02:50
*** sonuk has joined #openstack-infra02:51
*** baoli has quit IRC02:51
*** dims has joined #openstack-infra02:52
*** davideagnello has joined #openstack-infra02:54
*** davideagnello has quit IRC02:54
*** thorst has joined #openstack-infra02:56
*** binbincong has joined #openstack-infra02:56
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Added Test Selector Plugin support
*** rguillebert has quit IRC02:57
*** baoli has joined #openstack-infra02:59
*** yuanying has quit IRC03:00
*** yuanying has joined #openstack-infra03:01
*** baoli has quit IRC03:01
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Added Test Selector Plugin support
*** thorst has quit IRC03:03
*** otter768 has joined #openstack-infra03:04
*** nikhil has quit IRC03:04
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support
*** nikhil has joined #openstack-infra03:05
*** aeng has joined #openstack-infra03:07
*** harlowja_at_home has joined #openstack-infra03:08
*** otter768 has quit IRC03:08
*** sonuk has quit IRC03:09
*** pal has joined #openstack-infra03:10
*** pal is now known as Guest4384403:11
openstackgerritZhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project
*** mudassirlatif has joined #openstack-infra03:14
*** yuanying has quit IRC03:20
*** dimtruck is now known as zz_dimtruck03:20
*** Jeffrey4l has quit IRC03:21
*** mudassirlatif has quit IRC03:23
*** mudassirlatif has joined #openstack-infra03:25
*** links has joined #openstack-infra03:26
*** bpokorny has quit IRC03:27
*** baoli has joined #openstack-infra03:35
*** Jeffrey4l has joined #openstack-infra03:35
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support
*** Daisy has quit IRC03:37
*** ddieterly has quit IRC03:37
*** baoli has quit IRC03:39
*** mudassirlatif has quit IRC03:40
openstackgerritZhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project
*** Guest43844 has quit IRC03:42
*** sonali5 has quit IRC03:43
*** salv-orl_ has joined #openstack-infra03:45
*** salv-orlando has quit IRC03:47
stevemarso... is very different since the last time i looked at it, any tips on where the search field is? :\03:48
stevemarjust seeing a black background03:48
openstackgerritke liang proposed openstack/requirements: Bump oslo.messaging version to 4.0.0
*** Sukhdev has quit IRC03:51
*** Liuqing has joined #openstack-infra03:54
*** flwang has quit IRC03:56
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: networking-midonet: Add non-voting rally job
*** thorst has joined #openstack-infra04:01
*** bpokorny has joined #openstack-infra04:01
*** watanabe_isao has joined #openstack-infra04:02
*** treaki_ has joined #openstack-infra04:03
watanabe_isaohello all, I have meat a problem that dib cannot give me a image when I built a CI and run <nodepool image-build> command for the first time.04:03
watanabe_isaoThere is only two lines of log output, and the command waits there forever (at lease 48+ hours)04:05
watanabe_isaoHave anyone meet the same problem, or have a solution, please?04:06
*** yuanying has joined #openstack-infra04:06
*** treaki has quit IRC04:07
Thelowatanabe_isao: can you show the log s?04:07
*** thorst has quit IRC04:08
watanabe_isaoThelo, yes.04:08
*** kushal has joined #openstack-infra04:08
watanabe_isao2016-01-18 13:02:43,804 INFO gear.Connection.nodepool: Sending packet to <gear.Connection 0x7fd8cf0ca710 host: localhost port: 4730>: <gear.Packet 0x7fd8cdd48890 type: SUBMIT_JOB function: image-build:image unique: aa194908dec1428fa7e37b528b6a189f>04:08
watanabe_isao2016-01-18 13:02:43,805 INFO gear.Client.nodepool: Received packet from <gear.Connection 0x7fd8cf0ca710 host: localhost port: 4730>: <gear.Packet 0x7fd8cdd6b0d0 type: JOB_CREATED handle: H:>04:08
*** baoli has joined #openstack-infra04:10
*** baoli has quit IRC04:11
Thelowatanabe_isao: where do you seen this log?04:12
greghayneswatanabe_isao: are you running a nodepool daemon?04:12
watanabe_isaoThelo, the log is the output of <nodepool image-build> command04:12
watanabe_isaogreghaynes, no as the procedure of <> I think I should start nodepool after the image has been built.04:13
*** bpokorny has quit IRC04:14
greghayneswatanabe_isao: seems like you arent - you need to be runnign an image builder daemon in order to build images now. You can do this but running the nodepool-builder command or running a full nodepool daemon04:14
Thelowatanabe_isao: it looks like nodepool's log rather than the image-build output : in my machine the image-build about console output looks like :"2016-01-18 00:00:00,086 INFO + true"04:14
greghayneswatanabe_isao: looks like those docs are out of date. There was a recent change to how nodepool works which requires this now04:15
greghaynesThelo: We recently merged distributed builders which changed that04:15
*** Sukhdev has joined #openstack-infra04:16
Thelogreghaynes: Could that affect the CI who used the previous method? in my case, since yesterday night the image built by nodepool does not boot properly04:16
greghaynesThelo: This would not cause that - sounds like a dib issue04:17
watanabe_isaogreghaynes, thanks for telling me this. After I refreshed my browser I found the newest message. I'll have a look at this.04:18
Thelogreghaynes: yes, however I couldn't find any ERROR log in the trace ( except that : INFO sfdisk: ERROR: sector 0 does not have an msdos signature  , but I had it when it was working too )04:19
watanabe_isaogreghaynes, Thelo, thank you very much.04:19
greghaynesThelo: Ya, there were some recent changes to the dib bootloader element - what elements are you using in your images04:19
greghaynesThelo: also, when you say they dont boot properly, how far in to the boot process do they get?04:19
*** Daisy has joined #openstack-infra04:20
Thelogreghaynes: they stuck after "random: nonblocking pool is initialized"04:21
greghaynesThelo: oh, so not the bootloader then04:21
greghaynesThelo: I dont really know what would cause that04:22
Thelogreghaynes: me too04:22
Thelois there anyway to force nodepool to use yesterday's image?04:22
greghaynesdelete the newer image04:22
*** fawadkhaliq has joined #openstack-infra04:23
Thelogreghaynes: in nodepool directly?04:23
*** baoli has joined #openstack-infra04:23
greghaynesusing nodepool, yes04:23
*** otter768 has joined #openstack-infra04:24
*** maishsk has quit IRC04:25
*** hichihara has joined #openstack-infra04:26
Thelogreghaynes: should it take time?04:27
watanabe_isaoBTW, has anyone met the "slave went offline" issue before? That is as soon as a slave is done with its job and been deleted by jenkins, other slaves will went offline whatever how good progress the job runs.04:29
greghaynesThelo: new nodes should use the next older image04:29
*** vivekd has joined #openstack-infra04:29
*** oomichi has joined #openstack-infra04:31
Thelogreghaynes: The command dib-image-delete has terminated yet, so the new nodes keeps using this image.04:32
watanabe_isaoThelo, I think you should use <nodepool image-list> to get the image ids, and use <nodepool image-delete <newer-id>> to delete the today's image from provider. If I did not get greghaynes wrong.04:34
greghaynesyes, sorry, use nodepool-image-delete to delete the image uploaded to the provider, that is what your nodes boot off of04:35
*** otter768 has quit IRC04:35
*** vivekd has quit IRC04:36
Thelogrehaynes, watanabe_isao : thanks04:36
watanabe_isaogreghaynes, Thelo, I forgot to added your name. Do you have any idea about my last question of <slave went offline> issue?04:37
greghaynesI dont know much about the jenkins interaction, sorry04:38
*** ddieterly has joined #openstack-infra04:39
*** maishsk has joined #openstack-infra04:41
Thelogreghaynes: once in a while I get this error : FATAL: no longer a configured node for d-p-c-local_01-769 in my job's console04:43
Theloand the job fails04:43
Thelodo you know the reason?04:43
Theloand how to avoid that?04:43
*** ddieterly has quit IRC04:44
*** fawadkhaliq has quit IRC04:44
*** krtaylor has joined #openstack-infra04:45
*** maishsk has quit IRC04:47
*** maishsk has joined #openstack-infra04:48
watanabe_isaoThelo, seems like we are facing different problems. Mine is <ERROR: Connection was broken: Unexpected termination of the channel>, and I have not figure it out yet.04:48
*** sflanigan has joined #openstack-infra04:49
*** sflanigan has joined #openstack-infra04:49
*** oomichi has quit IRC04:50
*** ashishb has joined #openstack-infra04:52
*** sflanigan has quit IRC04:54
*** maishsk has quit IRC04:55
*** ashishb has quit IRC04:56
*** dims has quit IRC04:58
*** larainema has quit IRC04:58
*** pal has joined #openstack-infra04:58
*** pal is now known as Guest6756304:59
*** Guest67563 has quit IRC04:59
*** thorst has joined #openstack-infra05:05
*** sflanigan has joined #openstack-infra05:08
*** sflanigan has joined #openstack-infra05:08
*** coolsvap|away is now known as coolsvap05:11
openstackgerritgreghaynes proposed openstack-infra/nodepool: Handle dib build failures
*** harlowja_at_home has quit IRC05:12
*** aopincar has joined #openstack-infra05:13
*** thorst has quit IRC05:13
*** baoli has quit IRC05:17
*** salv-orl_ has quit IRC05:20
*** salv-orlando has joined #openstack-infra05:20
*** daemontool has quit IRC05:27
*** daemontool has joined #openstack-infra05:27
openstackgerritSteve Martinelli proposed openstack/requirements: bump versions for each keystone library
*** bardia has joined #openstack-infra05:30
*** daemontool has quit IRC05:33
*** fawadkhaliq has joined #openstack-infra05:34
*** sabeen3 has quit IRC05:37
*** mudassirlatif has joined #openstack-infra05:37
*** ddieterly has joined #openstack-infra05:40
*** jaosorior has joined #openstack-infra05:41
*** salv-orlando has quit IRC05:41
*** salv-orlando has joined #openstack-infra05:42
*** maishsk has joined #openstack-infra05:43
*** ddieterly has quit IRC05:45
*** isaacb has quit IRC05:46
*** tlian2 has quit IRC05:47
openstackgerritDaniel Wallace proposed openstack-infra/shade: granting and revoking privs to users and groups
*** _et_ has joined #openstack-infra05:54
*** abregman has joined #openstack-infra05:56
*** Watanabe has joined #openstack-infra05:59
*** watanabe_isao has quit IRC05:59
*** maishsk has quit IRC06:08
*** thorst has joined #openstack-infra06:10
*** aeng has quit IRC06:16
*** thorst has quit IRC06:18
*** sputnik13 has quit IRC06:20
openstackgerritZhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project
*** ashishb has joined #openstack-infra06:26
openstackgerritIan Wienand proposed openstack-infra/project-config: Cleanup the early devstack clone script
openstackgerritIan Wienand proposed openstack-infra/project-config: Prune old branches in early devstack clone
openstackgerritIan Wienand proposed openstack-infra/project-config: Cleanup the early devstack clone script
*** sonali5 has joined #openstack-infra06:29
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support
stevemarAJaeger: o/06:31
stevemarAJaeger: good morning!06:32
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
stevemarAJaeger: any reason why is not loading a dashboard for me :)06:33
*** otter768 has joined #openstack-infra06:36
*** sabeen1 has joined #openstack-infra06:36
*** sabeen2 has joined #openstack-infra06:38
*** yfried_ has quit IRC06:38
*** sabeen1 has quit IRC06:40
*** ddieterly has joined #openstack-infra06:41
*** otter768 has quit IRC06:41
*** isaacb has joined #openstack-infra06:45
*** ddieterly has quit IRC06:45
*** Ramanjaneya has joined #openstack-infra06:46
*** nikil89_ has joined #openstack-infra06:47
*** nikil89_ is now known as naggappan06:47
*** salv-orlando has quit IRC06:54
*** salv-orlando has joined #openstack-infra06:55
*** _nadya_ has joined #openstack-infra06:56
*** aeng has joined #openstack-infra06:58
*** chlong has quit IRC07:00
*** Qiming has quit IRC07:00
*** Qiming_ has joined #openstack-infra07:00
*** aeng has quit IRC07:01
openstackgerritIsaac Beckman proposed openstack/diskimage-builder: Move speedup section to image building guide
*** salv-orlando has quit IRC07:03
*** salv-orlando has joined #openstack-infra07:04
*** mudassirlatif has quit IRC07:11
*** isaacb has quit IRC07:12
*** maishsk has joined #openstack-infra07:15
*** thorst has joined #openstack-infra07:16
*** coolsvap is now known as coolsvap|away07:18
*** gildub has quit IRC07:20
*** ifarkas has joined #openstack-infra07:22
*** thorst has quit IRC07:23
*** e0ne has joined #openstack-infra07:25
*** kushal has quit IRC07:26
*** sabeen2 has quit IRC07:26
openstackgerritZhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project
*** baoli has joined #openstack-infra07:29
*** larainema has joined #openstack-infra07:29
*** kushal has joined #openstack-infra07:29
*** baoli has quit IRC07:33
*** Qiming_ has quit IRC07:36
*** nmagnezi has joined #openstack-infra07:38
*** mpavone has joined #openstack-infra07:40
*** rcernin has joined #openstack-infra07:41
*** Qiming has joined #openstack-infra07:42
*** ddieterly has joined #openstack-infra07:42
*** dguitarbite has joined #openstack-infra07:45
*** salv-orlando has quit IRC07:46
openstackgerritZhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project
*** ddieterly has quit IRC07:46
*** jtomasek has joined #openstack-infra07:46
*** maishsk has quit IRC07:46
*** salv-orlando has joined #openstack-infra07:47
*** mudassirlatif has joined #openstack-infra07:48
*** daemontool has joined #openstack-infra07:48
*** isaacb has joined #openstack-infra07:50
*** _et_ has quit IRC07:51
*** bardia has quit IRC07:53
*** zeih has joined #openstack-infra07:54
openstackgerritafazekas proposed openstack-infra/system-config: Add ebtables to the default package set
AJaegermorning, stevemar . Sorry, can't help with logstash07:55
*** esikachev has joined #openstack-infra07:55
esikachevhey guys! can you review
*** daemontool has quit IRC07:58
*** _et_ has joined #openstack-infra07:58
*** yfried_ has joined #openstack-infra07:58
changzhihi all, could someone take a look at i want check pep8 and unit test in jenkins job.08:02
changzhibut there is an error says "Job steth-pypi-both-upload not defined"08:03
AJaegerchangzhi: did you see my comments on projects.yaml?08:05
changzhiAJaeger: yes, and i uploaded a new patch set. but the same error ;-(08:06
*** salv-orlando has quit IRC08:06
AJaegerchangzhi: did you made the change I asked for?08:07
changzhiAJaeger: sure, i remove the line "gate-{name}-pep8"08:07
*** salv-orlando has joined #openstack-infra08:08
AJaegerbut you did not add pypi-jobs08:08
*** maishsk has joined #openstack-infra08:08
*** _nadya_ has quit IRC08:09
changzhiAJaeger: should like ?08:10
AJaegerchangzhi: yes08:10
*** matrohon has joined #openstack-infra08:11
AJaegerand follow - looking at pypi, we do not have permissions to publish steth yet08:11
changzhiok. what about zuul/layout.yaml? like this:
*** fawadkhaliq has quit IRC08:12
*** salv-orlando has quit IRC08:13
*** fawadkhaliq has joined #openstack-infra08:13
AJaegerremvoe the pre-release, it's part of publish-to-pypi08:13
*** markus_z has joined #openstack-infra08:14
*** hichihara has quit IRC08:17
changzhiAJaeger: ok. i have added openstackci permissions to steth just now.08:17
*** hdd has quit IRC08:18
*** wei-wang has joined #openstack-infra08:18
openstackgerritZhi Chang proposed openstack-infra/project-config: Add jenkins job into steth project
*** Jeffrey4l has quit IRC08:19
*** e0ne has quit IRC08:20
*** katyafervent is now known as katyafervent_awa08:21
*** lezbar has joined #openstack-infra08:21
*** thorst has joined #openstack-infra08:21
*** Daisy has quit IRC08:22
*** katyafervent_awa is now known as katyafervent08:22
asselinWatanabe, slave going offline might be done explicitly. most are "single-use-slaves".08:22
*** lezbar__ has joined #openstack-infra08:23
*** oomichi has joined #openstack-infra08:23
Watanabeasselin, hello, yes I'm using it.08:23
asselinWatanabe, the " Unexpected termination" is probably a network issue08:23
*** jcoufal has joined #openstack-infra08:24
asselinWatanabe, also I noticed you're using the old repo from github. I encourage you to switch to this one which is community-supported:
*** lezbar has quit IRC08:25
*** thorst has quit IRC08:27
Watanabeasselin, sure. Thank you very much. BTW, If I want the node go offline as soon as the job is over, do you know what should I use instead of single-use-slaves, please? set_node_options?08:27
*** [1]Thelo has joined #openstack-infra08:28
*** jaosorior has quit IRC08:29
*** sonali5 has quit IRC08:30
*** jaosorior has joined #openstack-infra08:30
*** Thelo has quit IRC08:30
*** [1]Thelo is now known as Thelo08:31
*** Jeffrey4l has joined #openstack-infra08:32
Watanabeasselin, sorry I was wrong. I am using single_use_node08:34
*** maishsk has quit IRC08:34
Watanabenot single-use-slaves08:34
*** fhubik has joined #openstack-infra08:35
*** yfried has joined #openstack-infra08:36
*** gildub has joined #openstack-infra08:36
*** yfried_ has quit IRC08:36
*** Jeffrey4l has quit IRC08:37
*** saggi has quit IRC08:37
*** otter768 has joined #openstack-infra08:37
*** samuelBartel has joined #openstack-infra08:37
*** jed56 has joined #openstack-infra08:38
*** dizquierdo has joined #openstack-infra08:39
*** Watanabe has quit IRC08:39
*** comstud has joined #openstack-infra08:40
*** shardy has joined #openstack-infra08:40
*** jlanoux has joined #openstack-infra08:41
*** otter768 has quit IRC08:42
*** unicell1 has joined #openstack-infra08:42
*** gibi has joined #openstack-infra08:42
*** unicell has quit IRC08:42
*** dingyichen has quit IRC08:43
*** ddieterly has joined #openstack-infra08:43
*** mudassirlatif has quit IRC08:44
*** eliqiao has joined #openstack-infra08:46
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Add DB API method to get tests from a list of failed runs
*** ddieterly has quit IRC08:48
*** Sukhdev has quit IRC08:49
*** eliqiao_ has quit IRC08:49
*** Jeffrey4l has joined #openstack-infra08:49
*** bauwser is now known as bauzas08:54
*** yfried has quit IRC08:55
*** yfried has joined #openstack-infra08:55
*** yaume has joined #openstack-infra08:55
*** aspiers has joined #openstack-infra08:59
*** fawadkhaliq has quit IRC09:01
*** fawadkhaliq has joined #openstack-infra09:01
*** daemontool has joined #openstack-infra09:02
*** daemontool has quit IRC09:03
*** sflanigan has quit IRC09:03
*** daemontool has joined #openstack-infra09:03
*** daemontool_ has joined #openstack-infra09:04
*** daemontool has quit IRC09:08
*** Daisy has joined #openstack-infra09:10
*** lezbar__ has quit IRC09:10
*** derekh has joined #openstack-infra09:11
*** daemontool_ is now known as daemontool09:11
*** lezbar__ has joined #openstack-infra09:11
*** bharathm has joined #openstack-infra09:11
*** hashar has joined #openstack-infra09:13
*** jistr has joined #openstack-infra09:14
*** HeOS has joined #openstack-infra09:17
*** jaosorior has quit IRC09:17
*** jaosorior has joined #openstack-infra09:18
*** dizquierdo has quit IRC09:21
*** sputnik13 has joined #openstack-infra09:21
*** zeih has quit IRC09:22
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Cleanup to use namedtuple
*** Daisy has quit IRC09:25
*** thorst has joined #openstack-infra09:26
*** sputnik13 has quit IRC09:26
openstackgerritJulien Danjou proposed openstack/requirements: Only install mock on Python < 3.3
*** Jeffrey4l has quit IRC09:32
*** thorst has quit IRC09:33
*** mudassirlatif has joined #openstack-infra09:35
*** fhubik is now known as fhubik_brb09:39
openstackgerritMerged openstack-infra/tripleo-ci: Revert "Add CI coverage of deleting the deployed overcloud"
*** ddieterly has joined #openstack-infra09:45
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Set DIB_NO_TMPFS=1
*** rguillebert has joined #openstack-infra09:46
*** ihrachys has joined #openstack-infra09:47
*** sshnaidm has quit IRC09:47
*** ddieterly has quit IRC09:49
*** Jeffrey4l has joined #openstack-infra09:50
*** fhubik_brb is now known as fhubik09:56
openstackgerritMerged openstack-infra/groups: Fixed some footer link & added news link
*** sshnaidm has joined #openstack-infra10:01
*** Qiming has quit IRC10:04
*** davideagnello has joined #openstack-infra10:05
*** ociuhandu has quit IRC10:06
*** ociuhandu has joined #openstack-infra10:07
*** ociuhandu has quit IRC10:07
*** pblaho has joined #openstack-infra10:07
*** ociuhandu has joined #openstack-infra10:08
*** ociuhandu has quit IRC10:08
*** davideagnello has quit IRC10:10
*** e0ne has joined #openstack-infra10:11
*** mudassirlatif has quit IRC10:11
*** mudassirlatif has joined #openstack-infra10:12
*** marcusvrn_ has joined #openstack-infra10:13
openstackgerritMiles Gould proposed openstack/requirements: Bump minimum python-ironicclient version
*** mika has quit IRC10:18
*** Jeffrey4l has quit IRC10:21
*** Hal has joined #openstack-infra10:23
*** Hal is now known as Guest6859410:23
*** cdent has joined #openstack-infra10:23
*** mika has joined #openstack-infra10:26
*** mika has joined #openstack-infra10:26
openstackgerritNicko Glayre proposed openstack-infra/jenkins-job-builder: Adding support for maven-metadata-plugin
*** ldnunes has joined #openstack-infra10:29
*** thorst has joined #openstack-infra10:31
*** kzaitsev_mb has joined #openstack-infra10:33
*** ociuhandu has joined #openstack-infra10:36
*** thorst has quit IRC10:37
*** otter768 has joined #openstack-infra10:38
*** bradjones has quit IRC10:38
*** bradjones has joined #openstack-infra10:40
*** bradjones has quit IRC10:40
*** bradjones has joined #openstack-infra10:40
*** gildub has quit IRC10:43
*** otter768 has quit IRC10:43
*** ddieterly has joined #openstack-infra10:45
*** ociuhandu has quit IRC10:47
*** ddieterly has quit IRC10:51
esikachevneed merge
*** pahuang has quit IRC10:53
*** dims has joined #openstack-infra10:54
*** mudassirlatif has quit IRC10:58
*** sfinucan has joined #openstack-infra10:59
*** lucas-dinner is now known as lucasagomes11:09
*** andrearosa has joined #openstack-infra11:09
*** _nadya_ has joined #openstack-infra11:10
andrearosawith the new gerrit, when I submit a new PS the votes get on the previous one are still there, is that correct?11:11
*** ociuhandu has joined #openstack-infra11:11
*** flaper87 has quit IRC11:11
*** flaper87 has joined #openstack-infra11:11
*** aysyd has joined #openstack-infra11:12
*** amotoki has quit IRC11:13
*** ociuhandu has quit IRC11:14
SotKIs there anyone around who could paste a copy of /etc/storyboard/storyboard.conf from (with private details removed, obviously)11:15
*** _et_ has quit IRC11:15
andreafandrearosa: I don't think that's true in general, only if you did a rebase with no conflict to resolve11:19
andreafandrearosa: but a -2 will stick I believe11:19
*** claudiub has joined #openstack-infra11:19
andrearosaandreaf: thanks for the reply, I have an example of one of my patches. I put a new PS with a code change so not only a rebase and I still get the -1 and the +1s I got before11:20
andrearosaandreaf: let me check again and I'll come back to you. thanks11:22
*** zhurong has quit IRC11:22
*** ociuhandu has joined #openstack-infra11:23
*** mase_x200 has joined #openstack-infra11:24
*** Qiming has joined #openstack-infra11:30
*** shardy_ has joined #openstack-infra11:30
*** Ramanjaneya has quit IRC11:31
*** shardy has quit IRC11:33
*** kzaitsev_mb has quit IRC11:36
AJaegerandrearosa: which change?11:36
AJaegerandrearosa: andreaf's explanation is correct, -2 is sticky and rest stays after rebase11:37
*** nmagnezi has quit IRC11:38
*** ddieterly has joined #openstack-infra11:46
*** changzhi has quit IRC11:47 is returning 503s quite a lot today. Anyone aware of this or is it just me seeing it?11:48
*** mase_x200 has quit IRC11:50
*** ddieterly has quit IRC11:50
*** oomichi is now known as oomichi_away11:51
andrearosaAJaeger: I can't show you anymore as I updated new patches and now the old votes are gone, odd, something strange happened. Probably my fault messing up with a chain of dependencies11:51
*** yamamoto_ has quit IRC11:51
andrearosaAJaeger: andreaf thank you both11:52
*** nmagnezi has joined #openstack-infra11:52
*** rfolco has joined #openstack-infra11:53
openstackgerritZara proposed openstack-infra/storyboard: Fix typo
AJaegerandrearosa: let me check quickly - can you provide me with the URL, please?11:53
andrearosaAJaeger: the problem was between PS6 and PS711:54
andrearosain the PS7 I addressed some concerns I had on PS6 but all the votes didn't go away11:54
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins in /Openstack
*** kzaitsev_mb has joined #openstack-infra11:56
AJaegerandrearosa: indeed to late to check now.11:56
*** rfolco has quit IRC11:57
AJaegerjokke_: I get the 503s also also  ;(11:57
andrearosaAJaeger: np thanks for your time. I'll bug you in case it happens again11:58
*** daemontool has quit IRC11:58
*** daemontool has joined #openstack-infra11:59
*** cdent has quit IRC12:00
*** Guest68594 has quit IRC12:03
openstackgerritMasayuki Igawa proposed openstack-infra/subunit2sql: Cleanup to use namedtuple
*** daemontool_ has joined #openstack-infra12:04
*** dizquierdo has joined #openstack-infra12:05
*** daemontool has quit IRC12:07
*** dims has quit IRC12:10
*** fawadkhaliq has quit IRC12:13
*** _nadya_ has quit IRC12:14
*** esikachev has quit IRC12:15
*** wei-wang has quit IRC12:17
*** Hal has joined #openstack-infra12:20
*** Hal is now known as Guest9404312:20
*** tdurakov is now known as Guest1662012:22
*** daemontool_ has quit IRC12:24
*** daemontool_ has joined #openstack-infra12:25
*** daemontool_ has quit IRC12:25
*** fhubik is now known as fhubik_brb12:25
*** fhubik_brb is now known as fhubik12:25
*** daemontool_ has joined #openstack-infra12:25
*** ociuhandu_ has joined #openstack-infra12:26
*** ociuhandu has quit IRC12:28
*** ociuhandu_ is now known as ociuhandu12:28
*** baoli has joined #openstack-infra12:28
*** ddieterly has joined #openstack-infra12:29
*** bradjones has quit IRC12:30
samueldmqI am getting a couple of 503 on gerrit, anyone else able to reproduce the issue?12:30
*** daemontool_ has quit IRC12:31
dulekSam-I-Am: I'm also experiencing that.12:31
openstackgerritMerged openstack-infra/project-config: Add sahara tests repository to PROJECTS
samueldmqdulek: thanks, I think that ^ was directed to me ?12:32
odyssey4mesamueldmq dulek I saw more of them last night, but they do seem to show up from time to time. I've not been getting them this morning.12:33
odyssey4meThere doesn't appear to be much consistency from what I've seen so far.12:33
*** baoli has quit IRC12:33
*** gordc has joined #openstack-infra12:33
*** tdurakov has joined #openstack-infra12:33
duleksamueldmq: That's right, sorry for pinging you Sam-I-Am!12:33
*** fhubik is now known as fhubik_brb12:34
*** cfarquhar has joined #openstack-infra12:34
*** cfarquhar has quit IRC12:34
*** cfarquhar has joined #openstack-infra12:34
*** chlong has joined #openstack-infra12:36
*** electrofelix has joined #openstack-infra12:37
samueldmqodyssey4me: perhaps CPU hitting 100% from times to times ?12:38
*** yamamoto has joined #openstack-infra12:38
*** otter768 has joined #openstack-infra12:39
*** fawadkhaliq has joined #openstack-infra12:40
*** markmcd__ has joined #openstack-infra12:42
electrofelixhashar waynr zxiiro: looks like AJaeger has responded to the request for feedback on so looks like  merged, would one of you like to complete the approval?12:43
*** ashishb has quit IRC12:44
*** otter768 has quit IRC12:44
*** markmcd1 has quit IRC12:44
*** markmcd has quit IRC12:44
*** thorst has joined #openstack-infra12:45
*** markmcd has joined #openstack-infra12:46
hasharelectrofelix: indeed :-)12:46
hasharelectrofelix: that will land "Remove linkcheck from sphinx builders list in config"12:47
hasharthough we need openstack infra to update jjjb/zuul as well grrr :D12:47
electrofelixone step at a time ;)12:48
hasharpoke me as needed, I tested those last week iirc12:48
*** _et_ has joined #openstack-infra12:48
openstackgerritMerged openstack-infra/jenkins-job-builder: Add separate linkcheck env and allow dev to select builder
electrofelixSo if someone from Infra could do the needful on, be much obliged12:49
openstackgerritMerged openstack-infra/tripleo-ci: Set DIB_NO_TMPFS=1
AJaegerelectrofelix: then let me approve the infra side as well ;)12:51
electrofelixAJaeger: thanks :)12:51
*** amotoki has joined #openstack-infra12:52
*** Ramanjaneya has joined #openstack-infra12:53
electrofelixI'll trigger a recheck on the JJB change about 30 minutes after that lands to make sure the new job works as expected12:53
AJaegerelectrofelix: good idea, thanks12:56
openstackgerritMerged openstack-infra/project-config: Add separate non-voting job to check doc links for JJB
*** bharathm has quit IRC13:00
*** mase_x200 has joined #openstack-infra13:00
*** rfolco has joined #openstack-infra13:01
*** diltram has joined #openstack-infra13:01
diltramhello everyone13:01
diltramI have a problem13:01
*** rfolco has quit IRC13:01
diltramI merged my account on launchpad13:01
*** rfolco has joined #openstack-infra13:02
diltramand I lost possibility to login to my account on review.openstack.org13:02
diltramI cannot on "new" account set my username because it's saying that it's used already13:02
diltrammay anyone help me with that?13:03
diltramis it possible to merge accounts on review.openstack?13:03
*** dprince has joined #openstack-infra13:04
*** doug-fish has joined #openstack-infra13:04
ZaraSotK asked a couple of hours ago: could anyone paste us a copy of /etc/storyboard/storyboard.conf from (with private details removed) we need it for debugging email notifications; they work on other instances but not here...13:05
*** zz_dimtruck is now known as dimtruck13:10
*** dims has joined #openstack-infra13:11
*** edmondsw has joined #openstack-infra13:12
*** _nadya_ has joined #openstack-infra13:14
*** rcernin has quit IRC13:19
*** _nadya_ has quit IRC13:19
*** dkranz has joined #openstack-infra13:20
*** daemontool has joined #openstack-infra13:21
*** mika has quit IRC13:22
*** zhurong has joined #openstack-infra13:22
*** mika has joined #openstack-infra13:23
*** shardy_ has quit IRC13:23
*** fhubik_brb is now known as fhubik13:23
*** shardy has joined #openstack-infra13:24
*** _nadya_ has joined #openstack-infra13:25
*** moravec has quit IRC13:25
*** trown|outttypeww is now known as trown13:29
*** weshay_xchat has joined #openstack-infra13:31
*** weshay_xchat is now known as weshay13:31
*** jordanP has joined #openstack-infra13:32
*** kzaitsev_mb has quit IRC13:33
*** links has quit IRC13:33
*** fawadkhaliq has quit IRC13:34
*** aojea has quit IRC13:34
*** itsuugo has joined #openstack-infra13:35
*** Ramanjaneya has quit IRC13:37
openstackgerritIsaac Beckman proposed openstack-infra/puppet-jenkins: Customize jenkins.default by template
mordredZara: one sec13:39
*** peristeri has joined #openstack-infra13:41
*** keedya has quit IRC13:42
Zaramordred: thank you! :)13:42
ZaraSotK ^13:42
SotKmordred: thanks!13:43
*** dimtruck is now known as zz_dimtruck13:44
*** binbincong has quit IRC13:45
*** dingyichen has joined #openstack-infra13:45
*** dtardivel has joined #openstack-infra13:45
*** tlian has joined #openstack-infra13:45
*** kzaitsev_mb has joined #openstack-infra13:46
*** binbincong has joined #openstack-infra13:46
*** zz_dimtruck is now known as dimtruck13:47
odyssey4meoh dear, gerrit 502's13:53
EmilienMouch, proxy error13:53
mkodereryep same for me13:53
odyssey4meI can't say whoops any more :p13:53
odyssey4me502 gone now13:53
*** Liuqing has quit IRC13:54
*** zhurong has quit IRC13:55
*** mika has quit IRC13:55
AJaegerodyssey4me: these 502s come and go ;(13:55
DuncanTgerrit is very sick it seems13:56
*** mika has joined #openstack-infra13:57
*** lennyb_ has quit IRC13:57
*** Ramanjaneya has joined #openstack-infra13:57
*** dimtruck is now known as zz_dimtruck13:58
*** bharathm has joined #openstack-infra14:00
*** edmondsw has quit IRC14:02
*** binbincong has quit IRC14:03
*** dslevin has quit IRC14:03
*** bharathm has quit IRC14:05
*** lennyb has joined #openstack-infra14:06
*** binbincong has joined #openstack-infra14:06
*** regXboi has joined #openstack-infra14:07
*** mase_x200 has quit IRC14:12
*** dkliban has joined #openstack-infra14:13
electrofelixhashar: gate-jenkins-job-builder-tox-docs-linkcheck is working so we can land
*** rlandy has joined #openstack-infra14:14
*** dtantsur is now known as dtantsur|brb14:17
openstackgerritSean McGinnis proposed openstack/requirements: Update to os-brick 0.8.0
*** keedya has joined #openstack-infra14:22
openstackgerritMerged openstack-infra/project-config: tooz: gate on etcd
openstackgerritMerged openstack-infra/storyboard-webclient: Improve usability of the "New Worklist" modal
*** jaypipes has joined #openstack-infra14:25
*** amitgandhinz has joined #openstack-infra14:25
*** yamamoto has quit IRC14:25
*** julim has joined #openstack-infra14:26
*** Ramanjaneya has quit IRC14:27
*** lucasagomes is now known as lucas-hungry14:28
*** sdake has joined #openstack-infra14:29
*** rossella_s has quit IRC14:31
*** rossella_s has joined #openstack-infra14:31
*** daemontool has quit IRC14:35
hasharelectrofelix: nice ! thank you for the link check work :-}14:36
hashardid W+114:36
*** ddieterly has quit IRC14:37
openstackgerritMerged openstack-infra/jenkins-job-builder: Remove linkcheck from sphinx builders list in config
*** fawadkhaliq has joined #openstack-infra14:38
*** tsg has joined #openstack-infra14:39
*** otter768 has joined #openstack-infra14:40
*** daemontool has joined #openstack-infra14:41
*** moravec has joined #openstack-infra14:42
*** yamamoto has joined #openstack-infra14:42
* anteaya reads Sam-I-Am's extensive research essay on mtu14:43
*** ddieterly has joined #openstack-infra14:44
*** amotoki has quit IRC14:44
*** otter768 has quit IRC14:44
fungiholiday here so i'm not paying super close attention, though i am around if something urgent needs my help14:46
fungijust give me a heads up14:46
*** Jeffrey4l has joined #openstack-infra14:47
anteayafungi: will do, happy Martin Luther King Jr day14:47
anteayamorning pabelanger14:47
anteayapabelanger: hey I just found out about this and I'm thinking of going:
*** ari has joined #openstack-infra14:48
*** bcourt has joined #openstack-infra14:48
*** naggappan has quit IRC14:48
anteayapabelanger: any chance you can make it to Kitchener on the 30th? would be great to see you14:48
*** pradk has joined #openstack-infra14:49
*** _nadya_ has quit IRC14:50
anteayaif an infra-root cares to take a peek at java melody and see where we are with gc, that would be great, thank you14:51
*** dkliban has left #openstack-infra14:52
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove a done todo list item
*** superdan is now known as dansmith14:54
*** cdent has joined #openstack-infra14:54
*** sigmavirus24_awa is now known as sigmavirus2414:55
electrofelixhashar: now we just need to fix the bad links so when sphinx releases with the fix for the broken links we can start to watch out for problems.14:56
*** cdent has quit IRC14:56
openstackgerritImre Farkas proposed openstack-infra/project-config: Add openstack/python-wsmanclient project
*** mrmartin has joined #openstack-infra14:57
pabelangeranteaya: sadly no, already have plans that day. Good news however, I'm all booked up for Austin!14:58
anteayapabelanger: yay!14:58
anteayathat is good news14:58
anteayayes the Linux thing is very short notice, hoping to meet some folks and encourage them to announce things a litte further in advance14:58
*** fhubik is now known as fhubik_brb14:59
mrmartin#success I found the reason of openid association handling bug between and askbot.14:59
openstackstatusmrmartin: Added success to Success page14:59
* mrmartin is happy now :)14:59
*** keedya_ has joined #openstack-infra15:00
anteayapabelanger: Austin is a great city, lots of live music, lots of really interesting buildings15:00
*** keedya has quit IRC15:00
anteayamrmartin: congratulations15:00
mrmartinthnx :)15:00
*** burgerk has joined #openstack-infra15:00
pabelangeranteaya: Ya, in on Saturday, out friday night ATM15:00
mrmartin90% dev and debug environment building 10% bug tracing and testing. OMG.15:01
pabelangeranteaya: never been to texas, so should be interesting15:01
samueldmqanteaya: I am looking forward to visiting Austin :)15:01
anteayapabelanger: texas is interesting, no doubt about it15:01
anteayasamueldmq: oh wonderful, it is a very nice city15:01
anteayaeasy to walk around15:01
anteayathe government buildings are impressive15:01
pabelangerThe stars at night, are big and bright!15:02
anteayapabelanger: ha ha ha15:02
pabelangeranteaya: only thing I know about Texas:
samueldmqanteaya: I am coming to keystone midcycle later this month, then I will visit it before summit :)15:03
samueldmqanteaya: and then at the summit again, woot15:03
*** _nadya_ has joined #openstack-infra15:03
hasharelectrofelix: excellent :-}15:03
anteayasamueldmq: yay twice15:03
anteayapabelanger: will watch in a minute15:04
*** zz_dimtruck is now known as dimtruck15:04
*** mriedem has joined #openstack-infra15:05
*** Jeffrey4l has quit IRC15:08
*** Qiming has quit IRC15:09
*** lezbar__ has quit IRC15:10
*** dingyichen has quit IRC15:11
*** Jeffrey4l has joined #openstack-infra15:11
*** fhubik_brb is now known as fhubik15:12
*** lezbar has joined #openstack-infra15:14
*** tdurakov has quit IRC15:14
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins
*** tdurakov has joined #openstack-infra15:17
openstackgerritAndrea Frittoli proposed openstack-infra/puppet-log_processor: Allow for python deps to be already defined
*** baoli has joined #openstack-infra15:20
andreafyolanda, clarkb: follow-up to - other libraries that may generate a conflict in puppet log_processor15:20
*** tdurakov has quit IRC15:21
*** Jeffrey4l has quit IRC15:21
openstackgerritImre Farkas proposed openstack-infra/project-config: Add openstack/python-wsmanclient project
*** baoli has quit IRC15:25
openstackgerritFausto Marzi proposed openstack-infra/project-config: Add python-jobs and python3-jobs to freezer*
*** baoli has joined #openstack-infra15:26
*** lucas-hungry is now known as lucasagomes15:28
*** med_ has quit IRC15:28
anteayapabelanger: nice15:29
anteayapabelanger: I wonder if you feel like trying that15:29
*** baoli has quit IRC15:29
*** mrmartin has quit IRC15:30
anteayaI think it is time for some breakfast and back to the mtu research mailing list post15:30
*** chlong has quit IRC15:31
*** HeOS has quit IRC15:31
openstackgerritMerged openstack-infra/irc-meetings: Adjust meeting time for US/EU doc-install-team-meeting
*** amitgandhinz has quit IRC15:34
*** baoli has joined #openstack-infra15:34
*** amitgandhinz has joined #openstack-infra15:35
*** _nadya_ has quit IRC15:35
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added two new repos for fuel-plugins
*** lezbar has quit IRC15:38
*** Sukhdev has joined #openstack-infra15:39
*** lezbar has joined #openstack-infra15:39
*** mtanino has joined #openstack-infra15:40
*** annegentle has joined #openstack-infra15:40
*** baoli has quit IRC15:42
*** bharathm has joined #openstack-infra15:42
openstackgerritDougal Matthews proposed openstack-infra/project-config: Move Tuskar to openstack-attic
*** palendae has joined #openstack-infra15:43
*** tsg has quit IRC15:44
*** sdake_ has joined #openstack-infra15:44
palendaeIs there an IP range for the infra's git servers? If so, I'd like to submit them to my work network for whitelisting15:45
anteayainfra-root ^^15:45
anteayapalendae: I don't know the answer to that question but stick around so that someone who knows can reply15:46
*** sabeen1 has joined #openstack-infra15:46
anteayaalso it is a holiday in the US today so it might be a while before you get a reply15:46
*** bharathm has quit IRC15:47
mordredpalendae: not really, no, sorry15:47
palendaeanteaya: Yep, no worries15:47
*** sdake has quit IRC15:48
mordredthey're cloud servers at rackspace - so it's essentially random IPs from the rackspace pool15:48
*** sabeen3 has joined #openstack-infra15:48
palendaemordred: For doing git-review pushes?15:48
mordredpalendae: git review pushes go to the gerrit server - and that is a single IP that we announce in advance if we need to move it15:49
mordred104.130.159.134 | 2001:4800:7818:102:be76:4eff:fe05:9b1215:50
palendaemordred: Ah, gotcha15:50
palendaemordred: Thanks!15:50
mordredsure thing!15:50
*** annegentle has quit IRC15:50
anteayapalendae: ah sorry I mis-understood15:50
anteayamordred: thank you15:50
palendaeI worded it poorly :p15:50
anteayaglad you got what you needed15:50
*** sabeen1 has quit IRC15:51
*** bcourt has quit IRC15:51
anteayapalendae: you might like this:
anteayaa graphic representation of our workflow15:51
*** mpavone has quit IRC15:52
anteayaatop the rest of our developer user manual15:52
*** hdd has joined #openstack-infra15:52
*** resker has quit IRC15:53
*** kgiusti has joined #openstack-infra15:54
*** abregman has quit IRC15:56
*** aopincar has quit IRC15:57
*** baoli has joined #openstack-infra15:57
anteayaSam-I-Am: going over your mtc research, now I have never looked at logs to evaluate for dropped packets due to the fact mtu was exceeded, so when you get a change I would be grateful for a tour of so I can learn what I should be looking for, here or -neutron or -dev, whereever is best15:58
*** rhallisey has joined #openstack-infra15:59
*** angdraug has joined #openstack-infra15:59
*** links has joined #openstack-infra16:00
*** sdake_ is now known as sdake16:01
*** isaacb has quit IRC16:02
*** dtantsur|brb is now known as dtantsur16:02
*** keedya has joined #openstack-infra16:02
*** bcourt has joined #openstack-infra16:03
*** yfried has quit IRC16:03
*** baoli has quit IRC16:04
*** flip214 has quit IRC16:09
clarkbpalendae: anteaya mordred also you can push over https and use your http proxy as ana lternative16:10
*** erlon has joined #openstack-infra16:10
*** baoli has joined #openstack-infra16:10
clarkbthis may be more desireable in some environments16:10
*** armax has joined #openstack-infra16:12
anteayabut still pushing to gerrit not to git.openstack.org16:12
gtmanfredmordred: Shrews yall get a chance to check this ?16:12
anteayaclarkb: thanks I always forget about the REST api16:12
*** vivekd has joined #openstack-infra16:12
anteayaclarkb: how was the sportsball yesterday?16:13
gtmanfredsportsball was less awesome yesterday, AMAZING ON SATURDAY16:13
palendaeclarkb: Is that just as simple as making sure that you have an https remote set for pushing in your git checkout?16:13
clarkbthe sea chickens were still asleep on the field and the large cats nommed on them16:13
clarkbpalendae: and using your gerrit http password yes16:14
gtmanfreduntil the second half when they woke up, but were missing one wing16:14
*** baoli has quit IRC16:14
anteayaclarkb: :(16:14
*** annegentle has joined #openstack-infra16:15
anteayaclarkb: enjoy your holiday16:15
*** edmondsw has joined #openstack-infra16:16
*** zeih has joined #openstack-infra16:17
openstackgerritMerged openstack-infra/release-tools: pick a different previous version if the current version exists
*** eharney has joined #openstack-infra16:20
fungion the gc activity question, here's the weekly graph (so you can compare to last monday and see where it went before we finally restarted gerrit)
mriedemis it just me or is logstash.o.o not loading?16:20
fungii have a feeling we may want to go ahead and restart gerrit now rather than wait for it to get worse later today16:21
*** Guest94043 has quit IRC16:21
fungiinfra-root: ^16:21
mordredfungi: ++16:21
fungii'm willing to do that in a few minutes16:21
*** flip214 has joined #openstack-infra16:22
*** flip214 has quit IRC16:22
*** flip214 has joined #openstack-infra16:22
openstackgerritMerged openstack-infra/release-tools: remove use of virtualenv from release scripts
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Just let shade and os-client-config handle clouds
openstackgerritMonty Taylor proposed openstack-infra/nodepool: Use shade for all OpenStack interactions
mordredShrews: ^^ I have rebased the nodepool consume-shade patches16:23
*** ddieterly has quit IRC16:23
*** links has quit IRC16:23
fungithe open files count steadily grew until it reached ~5k on saturday, then was abruptly knocked down to under 1k around midnight utc heading into sunday. i guess we have some timed cleanup happening there?16:24
anteayafungi: thank you, is there any way I can generate that kind of graph for myself?16:25
anteayamriedem: I confirm, logstash.o.o doens't give many anything beyond a black background16:26
fungianteaya: no, the javamelody gerrit plugin is restricted to administrators because it has destructive options, it's not a read-only interface. though someone was looking at a way to make these accessible via a net-snmp oid or something i think16:26
*** isaacb has joined #openstack-infra16:26
anteayafungi: okay, thanks for generating that graph16:26
fungiin which case we could add them to cacti16:26
*** woodster_ has joined #openstack-infra16:27
anteayaoh cacti would be awesome, alas I don't know how to do that work :(16:27
*** sdake has quit IRC16:27
*** tonytan4ever has joined #openstack-infra16:27
*** hashar has quit IRC16:28
stevemaranyone have a few minutes to chat about: ?16:28
fungialso http errors tracked in javamelody are way up starting around 1500 utc friday16:28
*** bnemec has quit IRC16:29
openstackgerritRussell Bryant proposed openstack-infra/project-config: Add non-voting rally job for networking-odl.
anteayastevemar: it seems the sticking point is the increase in timeout for the job16:29
anteayaand I agree with AJaeger 3 hours for a timeout is far too much for a job in the integrated gate16:30
stevemaranteaya: since around jan 15th, the gate-tempest-dsvm-keystone-eventlet-full seems to be hitting the 2 hr timeout very easily16:30
anteayastevemar: yes, that alone would be a cause for concern in my book16:30
fungistevemar: any idea what changed a few days ago to make that job so much slower?16:31
stevemaranteaya: you can check the amount of times we had to recheck here:
stevemarfungi: no idea16:31
anteayastevemar: no I believe you16:31
anteayastevemar: yeah I think finding out why the job is slower is the route I would encourage16:31
anteayaI don't think rechecking is the solution here16:31
*** sdake has joined #openstack-infra16:31
stevemarfungi: anteaya, we created the job a few weeks job, and it's been running fine16:31
stevemarfungi: anteaya sometimes it passes at around 1h 30m to 1h 40m.16:32
fungistevemar: is it spending more time in tempest than it was, or more time outside tempest? any correlation of timeouts to specific nodepool providers?16:32
openstackgerritVictor Stinner proposed openstack-infra/project-config: Remove glance python34 jobs on Liberty
anteayaI'd encourage you to move it from voting to non-voting in check and do further work investigating the job16:32
clarkbin thr past we did have to tune the eventlet procssses16:33
stevemaranteaya: i can do that16:33
stevemarfungi: i was gonna look that up last night :)16:33
clarkbiirc the default of 1 isnt enough and everything slows down when keystone is slow16:33
fungithough i guess it's hard to check that with kibana not loading16:33
anteayastevemar: yeah, we aren't fans of just increasing timeouts as it usually just masks a bigger problem that you have to investigate anyway16:33
stevemarfungi: yep! that made it extra hard last night to get proof it was timing out :)16:33
clarkbas for kibana not loading that probably means elasticsearch is unhappy16:34
stevemaranteaya: yeah, makes sense. i just wanted something to change since we have 5 or so patches that are trying to gate and days go by without merges16:34
anteayaclarkb: did you upgrade the workers over the weekend or no?16:34
*** jaosorior has quit IRC16:34
anteayastevemar: yup, I support the change in that job to non-voting in check (removing it from the gate queue) and fixing the job16:35
*** jaosorior has joined #openstack-infra16:35
clarkbanteaya I did not16:35
anteayastevemar: as clarkb says perhaps eventlet needs tuning16:35
anteayaclarkb: okey dokey16:35
openstackgerritMonty Taylor proposed openstack-infra/shade: Remove a done todo list item
Odd_BlokeSo we're specifying an email publisher as a default in our JJB configuration.  However, this doesn't get applied if there are any other publishers (which is, obviously, pretty common).  Is there any way that I can get JJB to merge the defaults with the options specified in a job(-template)?16:36
fungizaro: i'm seeing quite exceptions like these more or less continuously in our gerrit error log at the moment: (with various channel numbers)16:36
anteayathe zuul job queue graph is a very steady increase, I don't think I have seen as smooth an incline on that graph before16:37
clarkbOdd_Bloke: no16:37
*** med_ has joined #openstack-infra16:37
*** med_ is now known as Guest5121716:37
*** bnemec has joined #openstack-infra16:37
stevemarfungi: the jobs on hpcloud seem to be hitting the timeout consistently, and the passes seem to be on rax servers16:38
fungizaro: aha, looks like maybe that's just poor handling of dropped ssh connections
stevemarthere are also ovh hosts? /me shrugs16:38
anteayastevemar: yes16:39
anteayastevemar: and internap and bluebox16:39
fungistevemar: that's excellent analysis. aside from the hardware backing those, one of the biggest differences between them is that the images in hpcloud are built with dib while those in rax are built as modified snapshots of provider's base images16:39
*** _nadya_ has joined #openstack-infra16:39
anteayastevemar: as hpcloud is going away end of next week16:39
anteayawell and we are grateful to new hosts anytime, not just when one host goes away16:40
fungiovh, bluebox and internap are also using dib, so if you find timeouts on those as well (there are fewer of them so they may not be as easy to find examples for) then that would be a useful data point16:40
stevemarfungi: looks like there were two runs on ovh: one passed, one failed16:40
anteayaand as far as I know, dib everywhere is the way of the future16:40
stevemarerr, timed out, whatever16:40
*** otter768 has joined #openstack-infra16:41
fungidid the passing one complete close to the timeout threshold?16:41
fungiare some runs in hpcloud passing too, or are they consistently failing in hpcloud?16:41
fungiwow, our zuul job queue graph is pretty wacky16:43
*** fbo_ has quit IRC16:44
fungisteadily accumulating up to 25k waiting and then a couple hours ago it dropped like a stone. did someone restart something?16:44
clarkbit may be nodepool16:44
clarkbsince they share s gearman server now16:45
openstackgerritSteve Martinelli proposed openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full
*** fbo has joined #openstack-infra16:45
stevemaranteaya: fungi ^ as a bandaid, for now16:45
*** otter768 has quit IRC16:45
stevemarwhen logstash is back up, it'll be easier for me to figure out what's going on16:45
fungiclarkb: the "last reconfigured" time reported by the zuul status page seems to be ~ the same as the sheer dropoff in waiting jobs16:46
notmorganstevemar: porbably not enough workers.16:46
*** jaosorior has quit IRC16:46
notmorganstevemar: when we ran 1-worker eventlet originally in the pgsql run we ran into similar extended timeouts16:46
*** jaosorior has joined #openstack-infra16:47
notmorganeverything was hung up on keystone being slow.16:47
stevemarnotmorgan: i'm pretty sure bknudson_ just did a rename of the psql job16:47
fungi#status notice Gerrit is restarting quickly as a workaround for performance degradation16:47
openstackstatusfungi: sending notice16:47
*** zhenguo has joined #openstack-infra16:48
bknudson_might as well check the values for how many threads it's setting.16:48
*** jpr1 has joined #openstack-infra16:48
fungigerrit _should_ be back up now16:49
anteayastevemar: I'll peek when the gerrit is back16:49
-openstackstatus- NOTICE: Gerrit is restarting quickly as a workaround for performance degradation16:49
fungiseems to be working for me16:49
*** ddieterly has joined #openstack-infra16:50
openstackstatusfungi: finished sending notice16:50
fungiclarkb: doesn't look like either zuul or nodepool were restarted a couple hours ago16:50
anteayastevemar: can you remove that job from the gate queue for keystone please?16:51
fungiclarkb: so i'm starting to wonder whether we recently grew an accounting error in the stats reporting which resets on configuration reload maybe?16:51
anteayastevemar: non-voing jobs are in check only16:51
openstackgerritSteve Martinelli proposed openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full
stevemarfungi: anteaya done, i always forget that rule..16:52
* anteaya is here to remind you16:52
openstackgerritMerged openstack-infra/reviewstats: Optimize core list retrieval in reviewers
* notmorgan votes to just remove the eventlet job totally16:52
anteayastevemar: I'm happy thank you16:53
notmorganand finish the uwsgi thing instead of trying to fix "eventlet being slow"16:53
stevemarnotmorgan: looking at two failures now and the evenlet workers is set to 216:53
anteayanotmorgan: get the rest of keystone to agree and I can support that patch16:53
clarkbfungi maybe?16:53
notmorganyeah might need to push it to 4.16:53
notmorganor... just delete the job and not test eventlet :P16:53
notmorganfor mitaka16:53
openstackgerritMerged openstack-infra/reviewstats: Add python-brick-cinderclient-ext project to Cinder
stevemaranteaya: fwiw, eventlet was targeted to be removed in M, but there was some pushback16:54
fungistevemar: you're also skipping that job on stable branches but didn't explain why in the commit message. i'm okay with that, just making sure it's intentional16:54
*** julim has quit IRC16:54
notmorganeventlet is soooooo bad.16:54
anteayafungi: I took it as due to the job just being added a few weeks ago, but I agree with you, the commit message should say so16:55
fungi<imagine fist-shake emoji here>16:55
openstackgerritSonali Goyal proposed openstack-infra/jenkins-job-builder: Add Test Selector Plugin support
notmorganfungi: exactly16:55
fungianteaya: stevemar: yep, that makes sense. if it's intentional and not a copy-paste accident, i'll approve as is16:56
stevemarfungi: hmm, we should still test it with stable, *grumble16:56
openstackgerritSteve Martinelli proposed openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full
stevemarthanks for keeping me honest, i initially thought we just need it on master, but it's applicable to stable, so keep it there16:57
*** julim has joined #openstack-infra16:57
stevemarbknudson_: ^16:57
fungistevemar: if the job name is the same on stable branches as master (seems to be) then you can't make it non-voting on just some branches. you could have two versions of the job: a voting on on stable branches and a non-voting one on master. or you could switch to just running the job on stable branches and drop it from master entirely as notmorgan sort of suggested rather than making it non-voting16:58
anteayastevemar: welcome, thanks for wanting to be honest16:58
bknudson_stevemar: looks like we're running with 2 admin_workers and public_workers = None16:58
stevemarbknudson_: yeah, shouldn't it default to # of cores?16:58
stevemaror something like that16:58
bknudson_stevemar: I believe the default is the # of cores.16:58
stevemarNone cores, yay16:59
stevemarmaybe it meant Nine :P16:59
fungior nein16:59
bknudson_you can see the childs starting in the log...16:59
mordredstevemar, notmorgan: I do not understand the keep-eventlet pov even a little bit16:59
bknudson_"Starting 8 workers"16:59
stevemarmordred: something about containerizing stuff and things, you saw the ML thread17:00
bknudson_I'd also be happy to drop eventlet. They can run with uwsgi or whatever.17:00
stevemarbknudson_: is that something we had set before? or just used the default?17:01
bknudson_There's an assumption that the public interface is the one that's used in the tests, but I'm not sure that's the case.17:01
mordredor run a container management service thing that spins up and tears down single-threaded non-eventlet backend container workers17:01
mordredI mean, I hear that's all the rage in containerization land17:01
bknudson_The comments in devstack says that it's supposed to use the # of cpus. These haven't changed for some time.17:01
*** burgerk has quit IRC17:02
stevemar"oslo_service.service [-] Starting 8 workers"17:02
mordredstevemar: I may have missed that thread17:02
*** matrohon has quit IRC17:04
stevemarmordred: and later
*** dims_ has joined #openstack-infra17:05
notmorgani saw that thread17:05
notmorganand i would still rm -rf eventlet17:05
notmorganand say "make your container run apache in forground mode or uwsgi or anything else that does the same thing"17:06
bknudson_they can write their own eventlet wrapper if they want17:07
notmorganbknudson_: ++17:07
notmorganit's the "Dev" part of "DevOps" :P17:07
*** isaacb has quit IRC17:07
stevemarbknudson_: even the runs that pass set public_workers to None:
mordredk. to be fair, it doens't look like anybody had an answer for the multiple virtualenv question17:08
bknudson_stevemar: it looks like we're just seeing that some runs are slower than others... maybe because of some network issue or disk issue or whatever17:08
*** vivekd has quit IRC17:08
*** dims has quit IRC17:08
stevemarbknudson_: yeah, that's why i wanted to increase the timeout17:09
notmorganmordred: aache can handle multiple virtualenvs in mod_wsgi now, it's part of the mod_wsgi line - also mod_uwsgi [uwsgi does it fine]17:09
bknudson_stevemar: maybe admin_workers=2 is too low?17:09
stevemarbknudson_: seems fine for this rax run:
greghaynesMod proxy uwsgi will 'just work' with multiple venvs17:09
*** armax has quit IRC17:09
mordrednotmorgan: ah - awesome17:10
*** davideagnello has joined #openstack-infra17:10
mordrednotmorgan: that's the only thing I could see from the threads that was a legit 'oh, that's a good point' complaint17:10
greghaynesIts possible there is some wierd requirement for not using the proxy though...17:10
*** armax has joined #openstack-infra17:10
mordredcomplaining that apache is the default when there are other choices people _could_ use is bikeshedding17:10
notmorganmordred: WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/home/keystone/VENV/lib/python2.7/site-packages [for example]17:10
*** jlanoux has quit IRC17:10
*** hockeynut_afk has joined #openstack-infra17:10
mordrednotmorgan: and then you do a different virtualenv for horizon in that apache I guess?17:10
notmorganmordred: i didn't say it was "good" but that at least makes the DaemonProcess use a specific site-packages17:10
notmorganmordred: yeah you can do that.17:11
mordredcool. wfm17:11
notmorganmordred: in-fact i have done that17:11
notmorganmordred: but you must use daemon mod17:11
mordrednotmorgan: neat!17:11
notmorgannot the other mod_wsgi moe17:11
*** bgmccollum has left #openstack-infra17:11
notmorganideally you'd use uwsgi as the daemon because uwsgi natively handles venvs17:11
mordredyah. that sounds like the best way forward17:12
notmorganit also means you don't need to restart apache to reload <service>17:12
mordredand also, folks who prefer nginx - it works great with uwsgi17:12
notmorganeither as raw proxy or uwsgi-native17:12
greghaynesnotmorgan: you mean mod_proxy_uwsgi vs mod_uwsgi, right?17:12
bknudson_do you think it would be better for devstack to reference architecture with uwsgi ?17:12
notmorgangreghaynes: both.17:12
notmorganbknudson_: no17:13
notmorganbknudson_: well uwsgi behind apache17:13
bknudson_right, uwsgi behind apache17:13
notmorganand i'd run horizon in a separate uwsgi17:13
* notmorgan might go write some devstack today.17:13
*** fhubik is now known as fhubik_brb17:13
bknudson_devstack already has some SSL proxy stuff in it17:13
bknudson_so using proxy all the time would make devstack simpler17:13
*** mrmartin has joined #openstack-infra17:13
notmorgangreghaynes: or whatever is the right answer. i think it depends on the apache version17:14
mordredbknudson_: ++17:14
notmorgangreghaynes: could also just use mod_proxy or mod_rewrite_proxy17:14
mordredof course,I think devstack should run in notmorgan sub-url mode myself :)17:14
notmorganmordred: working on it!17:14
*** fhubik_brb is now known as fhubik17:14
notmorganmordred: ;)17:14
greghaynesYa, last I checked mod_uwsgi was not really advised17:14
mordrednotmorgan: can we call that deployment "notmorgan-style" ?17:14
bknudson_mordred: I've been trying to fix tempest-lib so that it works with keystone on sub-url.17:14
*** fhubik has quit IRC17:14
mordredbknudson_: neat!17:15
greghaynesBut if it works too then awesome17:15
notmorgangreghaynes: yeah i also want to get uwsgi to use domain sockets17:15
notmorgangreghaynes: rather than needing to run an "internal" port17:15
bknudson_the uwsgi proxy setup and subpaths are not incompatible.17:15
notmorganbknudson_: tempestlib was the blocker for suburl.17:15
bknudson_and doing always-proxy would make it easier to do in devstack17:16
bknudson_should say "would make *both* easier to do in devstack"17:16
notmorganbknudson_: but we'd need to L7 route still to do suburl so we'd have to add HAProxy or nginx in17:16
*** jswarren__ has quit IRC17:16
stevemarfungi: i made the job non-voting on all branches, that's alright i assume?17:16
bknudson_HAProxy might be the better option over apache. I don't know17:17
notmorganbknudson_: you'd still need an http service for horizon and keystone17:17
notmorganbknudson_: and apache is the best of the options since we have better federation support.17:17
notmorganthough we could just run uwsgi in http mode17:17
fungistevemar: lgtm. approved17:17
stevemarnotmorgan, bknudson_ #-keystone? :)17:17
bknudson_you can run uwsgi to serve http. why convert http->uwsgi->http? Is it just because HTTP 1.0 closed connections?17:18
stevemarfungi: ty sir17:18
notmorganbknudson_: standard testing for apache-isms since we recommend it.17:18
*** hdd has quit IRC17:18
notmorganmordred: i mean, i can try and run up a devstack using haproxy and sub-url. i think it'll all work. now.17:19
openstackgerritDoug Wiegley proposed openstack/requirements: Tweak requirements and directions for constraints scripts
*** e0ne has quit IRC17:19
*** ihrachys has quit IRC17:20
*** dizquierdo has quit IRC17:21
mgagneis there a page where I can find history and average time of gate-tempest-dsvm-neutron-full job?17:24
*** apoorvad has joined #openstack-infra17:26
*** _nadya_ has quit IRC17:26
*** vivekd has joined #openstack-infra17:27
*** _et_ has quit IRC17:27
mgagneanteaya how can I get average run time?17:27
anteayamgagne: well that functionality might not be built yet17:27
anteayait is a fairly new tool17:27
anteayait might not have all you need17:27
mgagneyea, that's what I'm looking for. I get stats for my provider but would like to know global average to get an idea about trend17:28
anteayayup makes sense17:28
anteayamtreinish is the main developer, I'm sure he would be interested in hearing your thoughts17:28
anteayahere is the repo:
*** _et_ has joined #openstack-infra17:29
anteayaand the open patches:
*** daemontool has quit IRC17:30
*** achanda has joined #openstack-infra17:30
*** daemontool has joined #openstack-infra17:31
jeblairmgagne: you may also be interested in playing with graphite a bit ^17:32
*** markus_z has quit IRC17:32
jeblairmgagne: that's an average of the runtime across all providers17:32
jeblairmgagne: open it up in the 'composer' url and you can easily edit it17:33
*** trown is now known as trown|lunch17:33
mgagnejeblair trying to play with grafana =)17:33
jeblairmgagne: it's the same data17:34
jeblairmgagne: it's a little easier to do ad-hoc things in graphite's ui though17:34
mgagnejeblair I'm a fancy guy17:34
jeblairmgagne: but if you find a graph you like, feel free to propose it to the grafana dashboards17:34
*** mrmartin has quit IRC17:34
*** mrmartin has joined #openstack-infra17:35
jeblairmgagne: for instance, we could definitely add average runtime across all providers graph to grafana17:35
*** samuelBartel has quit IRC17:35
*** jordanP has quit IRC17:35
mgagnethat's what I'm looking for but I think my grafana-fu isn't top, I'm getting 3y average17:35
*** dslevin has joined #openstack-infra17:36
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add hue-light plugin support
*** mrmartin has quit IRC17:37
*** kzaitsev_mb has quit IRC17:37
*** mrmartin has joined #openstack-infra17:38
*** _nadya_ has joined #openstack-infra17:38
openstackgerritMerged openstack-infra/project-config: disable voting for tempest-dsvm-keystone-eventlet-full
sc68calI'm having difficulty pulling down patches from gerrit17:38
*** daemontool has quit IRC17:38
*** daemontool has joined #openstack-infra17:39
fungisc68cal: did that happen just now?17:39
*** peristeri has quit IRC17:39
sc68calfungi: yeah17:39
sc68calok this time it worked17:39
fungior back before i restarted gerrit at 16:50 utc?17:39
fungithat's not a good sign17:40
sc68calfungi: like 1 minute ago. Although looks like it's happy now17:40
*** yamamoto has quit IRC17:40
*** nmagnezi has quit IRC17:40
pabelangerjeblair: did we fix the regression in zuul yet? Curious if we can tag a new release17:40
*** tonytan4ever has quit IRC17:40
fungizaro: ^ indication that this may not just be triggered by java gc busyness17:40
pabelangersc68cal: I had a hiccup too with gerrit17:40
fungijavamelody claims gerrit is spending basically no time in gc (understandable since it was rebooted barely an hour ago)17:41
*** mrmartin has quit IRC17:42
*** _nadya_ has quit IRC17:42
fungithe javamelody system load graph indicates there was a significant spike in load average a few minutes ago, which probably corresponds to this17:42
fungiload shot up to 16 for one sample17:43
AJaegercould somebody approve the infra-manual change on retiring a project, please?
*** yaume has quit IRC17:44
anteayahey AJaeger17:46
anteayafungi: what might cause it to spike?17:46
*** sabeen3 has quit IRC17:46
*** tdurakov_ has joined #openstack-infra17:47
*** Sukhdev has quit IRC17:47
fungianteaya: good q. still looking around17:48
fungianteaya: i'm betting on this, since the spikes line up
mgagnejeblair should I create a new dashboard?17:49
anteayafungi: is someone cloning from gerrit?17:50
fungianteaya: no clue what that was. could also have been a ton of api access i guess17:50
*** yfried has joined #openstack-infra17:52
AJaegerhi anteaya.17:54
AJaegeranteaya: just linked to 210638 when giving a -1  on
*** hdd has joined #openstack-infra17:55
anteayaAJaeger: nicely done17:56
anteayaI'm taking a peek at 26914917:56
anteayaI agree with fungi that the implications to changes in the governance file do affect the electoral rolls17:57
AJaegeranteaya: I'd like to remove some jobs that fail completely, could you review, please?
AJaegeranteaya: but wouldn't moving to the attic have the same problem?17:58
anteayaI wonder about creating a section somewhere in governance with names of projects that used to be official and are now attic, with a date on them so they can be removed17:58
*** jaosorior has quit IRC17:59
anteayawell as long as they were somewhere that they _were_ offical for some period of time for which their contributions could be attributed to atc status17:59
AJaegercan we merge the patch in some form and continue the discussion on this one point somehow?17:59
*** Hal has joined #openstack-infra17:59
anteayalet me look and think a bit17:59
*** Hal is now known as Guest8364318:00
*** tdurakov_ has quit IRC18:00
*** samuelBartel has joined #openstack-infra18:01
*** derekh has quit IRC18:03
*** zeih has quit IRC18:04
*** baoli has joined #openstack-infra18:04
*** tdurakov_ has joined #openstack-infra18:04
*** _nadya_ has joined #openstack-infra18:05
*** fawadkhaliq has quit IRC18:06
openstackgerritMerged openstack-dev/hacking: Avoid git clone in integration test
*** eil397 has joined #openstack-infra18:10
*** fawadkhaliq has joined #openstack-infra18:10
*** electrofelix has quit IRC18:10
*** fawadkhaliq has quit IRC18:11
AJaegerwith 268275 merged, is fine to merge now as well but needs some project-config review first if anybody wants, please18:11
openstackgerritMerged openstack-infra/project-config: Remove translation from repos that are not setup
*** yamahata has joined #openstack-infra18:15
*** tdurakov_ has quit IRC18:16
*** jistr has quit IRC18:17
*** Sukhdev has joined #openstack-infra18:17
openstackgerritMathieu Gagné proposed openstack-infra/project-config: grafana: Add job average runtimes to Nodepool dashboard
anteayaAJaeger: my lunch is ready, still reviewing and thinking18:18
mgagnejeblair ^18:18
*** angdraug has quit IRC18:19
*** tdurakov_ has joined #openstack-infra18:20
*** yolanda has joined #openstack-infra18:21
*** trown|lunch is now known as trown18:22
*** dtantsur is now known as dtantsur|afk18:22
fungianteaya: i think it's probably better to just leave it up to the tc, with some guidance, as to whether or not removal of the repo is going to have an impact on elections. most of the removals so far have gone, like, a year with no commits (aside from the occasional mass update from one of our already established contributors for whom the loss of credit on that commit will make no difference to18:22
fungiqualifying as a voter)18:22
fungii'll update my vote accordingly18:22
*** davidlenwell has quit IRC18:22
*** david-lyle has quit IRC18:23
*** davidlenwell has joined #openstack-infra18:24
*** yaume has joined #openstack-infra18:27
*** [1]Thelo has joined #openstack-infra18:28
*** jpr2 has joined #openstack-infra18:29
*** dslevin has quit IRC18:30
*** Thelo has quit IRC18:30
*** [1]Thelo is now known as Thelo18:30
*** rossella_s has quit IRC18:30
*** rajinir has joined #openstack-infra18:31
*** rossella_s has joined #openstack-infra18:31
AJaegeranteaya: sorry, I shouldn't have asked for review of that manual change with those -1s18:32
*** jpr1 has quit IRC18:32
AJaegerah, thanks, fungi18:33
openstackgerritMathieu Gagné proposed openstack-infra/project-config: grafana: Add job average runtimes to Nodepool dashboard
*** armax has quit IRC18:38
openstackgerritMorgan Fainberg proposed openstack/requirements: Add "positional" package to g-r
*** armax has joined #openstack-infra18:39
anteayafungi: is there an item in front of the tc to this effect?18:39
*** david-lyle has joined #openstack-infra18:39
fungianteaya: not directly, but every retirement which proposes removal from governance is at least acknowledged by the respective ptl (if the whole project is not completely abandoned) and put before the tc18:41
fungiso that still allows the tc to decide on a case by case basis18:41
*** otter768 has joined #openstack-infra18:42
*** dslevin has joined #openstack-infra18:43
*** doug-fish has quit IRC18:43
*** hockeynut_afk is now known as hockeynut18:43
AJaegeranteaya: we use merge-check for all active repos18:45
AJaegeranteaya: so the change to merge-check/noop is needed for removing all content from the repo18:46
*** tdurakov_ has quit IRC18:46
AJaegeranteaya: afterwards the entry can be removed completely18:46
*** david-lyle has quit IRC18:46
*** otter768 has quit IRC18:46
*** dmsimard is now known as dmsimard|afk18:47
*** bardia has joined #openstack-infra18:47
anteayaI guess I missed the part where it became obligatory18:48
anteayarather than opt-in18:48
AJaegeranteaya: it's obligatory AFAIU18:48
openstackgerritMerged openstack-infra/project-config: Remove glance python34 jobs on Liberty
*** tdurakov_ has joined #openstack-infra18:48
openstackgerritMerged openstack-infra/project-config: Put ceilometer integration test on client lib
openstackgerritAndreas Jaeger proposed openstack-infra/infra-specs: Make translation set up consistent
*** hashar has joined #openstack-infra18:51
*** annegentle has quit IRC18:51
openstackgerritMerged openstack-infra/project-config: Create packetary project
anteayathe zuul job queue graph still looks very odd to me18:51
AJaegeryes - until the restart18:52
openstackgerritMerged openstack-infra/project-config: Add jenkins job into steth project
anteayathen it just flatlined18:55
zarofungi not really around today. But what was the purpose for gerrit restart?18:55
anteayazaro: it was heavily into garbage collection again18:56
*** mrmartin has joined #openstack-infra18:56
anteayazaro: also after the restart sc68cal had a 502:
*** ashishb has joined #openstack-infra18:57
anteayazaro: so the 502's and 503's which had been attributed to the gc behaviour may not be18:57
*** pvaneck has joined #openstack-infra18:57
mordrednibalizer: I have verified - omfra cloud does support config-drive18:58
mordrednibalizer: you just have to request it when you boot an instance18:58
*** yolanda has quit IRC18:59
nibalizeroh neat18:59
sc68calmordred: is that an API extension or something we offer18:59
sc68cals/we/the openstack api/18:59
nibalizermy nodepool is humming along, which is weird to me because I don't understand it19:00
nibalizerbut investigation is proceeding :P19:00
mordredsc68cal: it's just a normal thing19:00
sc68calCAN WE SET THE MTU!?!?!?!?19:00
mordrednibalizer: we can also configure the cloud ot just always make a config drive by default19:01
* sc68cal jumps from bridge19:01
* mordred tosses sc68cal a life-vest that mostly doesn't have holes19:01
sc68calmordred: lol19:01
mordredhi mrmartin19:01
*** burgerk has joined #openstack-infra19:02
*** Sukhdev has quit IRC19:02
*** Guest83643 has quit IRC19:03
*** jcoufal has quit IRC19:03
anteayasc68cal: I got through Sam-I-Am's dissertation, thanks for prompting the conversation19:03
*** Sukhdev has joined #openstack-infra19:03
anteayaI wonder how many of our ssh bugs are related19:03
AJaegeranteaya: can you change Sam-I-Am's nic to Dr-I-Am, please ;)19:03
mrmartinmay I get a little care to this patch: This will help to bring back askbot-staging alive, and I found the bug and contributed a fix to upstream askbot, so the login would go to production.19:03
*** sabeen has joined #openstack-infra19:05
*** tdurakov_ has quit IRC19:05
*** rockyg has joined #openstack-infra19:05
sc68calanteaya: AJaeger: +1 Sam-I-Am is awesome19:06
anteayaAJaeger: :)19:06
anteayasc68cal: agreed, Sam-I-Am is awesome19:06
*** sabeen3 has joined #openstack-infra19:06
fungizaro: increased time spent in gc, heavy system load, elevated cpu utilization, frequent http 502 errors from webui and rest api19:07
openstackgerritDaniel Wallace proposed openstack-infra/shade: one function for updating users passwords
*** kzaitsev_mb has joined #openstack-infra19:08
*** _nadya_ has quit IRC19:09
*** sabeen has quit IRC19:09
*** baoli has quit IRC19:10
*** david-lyle has joined #openstack-infra19:10
*** tdurakov_ has joined #openstack-infra19:11
*** gfreeman has joined #openstack-infra19:11
*** flepied1 has joined #openstack-infra19:11
*** tonytan4ever has joined #openstack-infra19:12
*** ociuhandu has quit IRC19:12
*** flepied has quit IRC19:13
*** sfinucan has quit IRC19:14
*** achanda has quit IRC19:14
*** dslevin has quit IRC19:18
*** peristeri has joined #openstack-infra19:19
*** abregman has joined #openstack-infra19:19
*** lennyb has quit IRC19:20
*** lennyb has joined #openstack-infra19:20
*** abregman is now known as abregman|nb19:20
krotscheckDo we have statistics about job failure rate by job name?19:21
*** akshai_ has quit IRC19:22
mgagnekrotscheck I think we have
*** baoli has joined #openstack-infra19:23
mgagnekrotscheck I think you can dig into projects to get more info and list of jobs19:23
krotscheckmgagne: I see that, awesome.19:23
*** kzaitsev_mb has quit IRC19:26
openstackgerritDaniel Wallace proposed openstack-infra/shade: one function for updating users passwords
openstackgerritMerged openstack-infra/system-config: Pin ask staging branch to 0.7.x and update settings
anteayachanges to nodepool elements scripts, do they go into effect with the next image build? motivation for my question:
*** david-lyle_ has joined #openstack-infra19:27
*** dslev has joined #openstack-infra19:28
*** baoli has quit IRC19:28
*** david-lyle has quit IRC19:31
*** baoli has joined #openstack-infra19:32
*** salv-orlando has joined #openstack-infra19:32
*** akshai has joined #openstack-infra19:35
*** fawadkhaliq has joined #openstack-infra19:35
anteayagerrit just failed to take my review, it said working... and then my review disappeared19:36
anteayasecond time lucky19:36
*** david-lyle_ has quit IRC19:37
*** david-lyle has joined #openstack-infra19:39
*** baoli has quit IRC19:40
*** dmsimard|afk is now known as dmsimard19:40
*** achanda has joined #openstack-infra19:40
*** baoli has joined #openstack-infra19:41
*** mrmartin has quit IRC19:41
*** baoli has quit IRC19:42
*** yaume has quit IRC19:43
*** zeih has joined #openstack-infra19:44
*** david-lyle has quit IRC19:46
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystone starting with mitaka
*** _nadya_ has joined #openstack-infra19:49
*** ifarkas has quit IRC19:50
adam_gmordred, if you're not holidayin'.. had to rebase this to fix a conflict over the weekend
krotscheckAnybody want a wheel mirror? I know you do!
*** salv-orlando has quit IRC19:51
*** achanda has quit IRC19:53
*** kzaitsev_mb has joined #openstack-infra19:54
*** fawadkhaliq has quit IRC19:56
*** moravec has quit IRC19:56
*** e0ne has joined #openstack-infra19:56
*** bcourt has quit IRC20:03
clarkbthere is no elasticsearch process running on es0220:03
clarkbdmesg reports possible syn flooding on es ports20:04
clarkbI think the cluster must've gone crazy for some reason and tried to rebuild itself then failed and turned off20:04
clarkbbut the logs don't really say much20:04
*** baoli has joined #openstack-infra20:05
jeblairclarkb: looks like a couple days ago:
*** _nadya_ has quit IRC20:06
clarkbthe 14th?20:06
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API
clarkbI was going to look and see if rax reported anything then since that was when it last logged anything20:07
ianwclarkb: (when you're finished with elasticsearch or whatever) did you see the --prune stuff for early devstack clone in ?  i also stripped the whole thing down in
clarkbianw: I hadn't seen your changes yet. I did make a change to dib that should fix it though20:08
anteayaclarkb: :(20:08
anteayaianw: I was looking at those20:08
clarkbianw: so the prune is not sufficient20:08
anteayaianw: I haven't done much reviewing in nodepool/elements, do changes in those scripts get picked up in the next image build?20:08
clarkbianw: all the prune does is update your remote branches it doesn't delete stable/foo if origin/stable/foo is pruned20:09
AJaegeranteaya: yes20:09
anteayaAJaeger: thank you20:09
ianwanteaya: yep20:09
clarkbianw: so we still need in dib generally, thoug hI agree that in this particular instance our forked element is the larger problem20:10
*** baoli has quit IRC20:10
*** yaume has joined #openstack-infra20:10
anteayaianw: thank you20:10
*** eil397 has quit IRC20:13
*** tdurakov_ has quit IRC20:13
*** angdraug has joined #openstack-infra20:14
clarkblooks like we got to the bottom of the zuul job graph? what was going on there?20:14
anteayaclarkb: in the commit message, I can't follow the first sentence in the second paragraph:
openstackgerritMikhail S Medvedev proposed openstack-infra/system-config: Fix regex in NoRoaming workaround
anteayathe zuul job graph looks flatlined to me20:14
clarkbanteaya: I can try rewriting it20:15
*** tdurakov_ has joined #openstack-infra20:15
anteayawhich is not what I expect the zuul job graph to look like20:15
anteayaclarkb: thanks20:15
clarkbexcept new ui doesn't seem to let me edit commit message I guess I should use igt20:15
anteayaif someone knows what is happening with the zuul job graph, I haven't picked up on their understanding if they have tried to communicate it20:15
anteayaclarkb: it is supposed to20:16
*** ddieterly has quit IRC20:16
SamYapleclarkb: you can edit it20:16
anteayathat is a feature of new ui20:16
clarkbbut how20:16
*** ldnunes has quit IRC20:16
clarkbthere was a button next to the commit mesage to edit it in the past20:16
anteayapatchset 1 2 3 edit button20:16
anteayaI know20:16
anteayathis is harder to find20:16
clarkboh I have to open the thing in the diff?20:16
anteayasilly silly silly20:16
AJaegerclarkb: now you can change *each* file20:17
*** dprince has quit IRC20:17
ianwclarkb: so i don't think local branches are the issue.  have a look at and search for "14:28:17,336" ... cache-devstack-repos is picking up remotes/origin/stable/...20:17
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystone starting with mitaka
*** bcourt has joined #openstack-infra20:18
clarkbthis is confusing20:18
clarkbianw: they are an issue because there is some striaght file copying iirc rather than git cloning happening20:18
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka
mmedvededoes anyone know why there are no new logs on Most recent ones are from 3 days ago. I was checking if puppet apply was working before submitting
clarkbianw: if we used git clone everywhere we would have fewer problems likely20:19
*** tonytan4ever has quit IRC20:19
clarkbso I made edits and saved them but no new patchset20:19
clarkboh you have to go out a window and click "done editing"20:19
anteayammedvede: well possibly because image build have failed for the last few days20:19
anteayaclarkb: yes and save and really save20:19
clarkbuh still no new patchset20:20
clarkbI give up will make changes locally20:20
anteayaclarkb: must have missed a save somewhere20:20
ianwmmedvede: hmm, i bet it's related to the builders change20:20
clarkb(what good are feature slike this if they are completely unuseable20:20
SamYapleclarkb: thats the spirit!20:20
*** ddieterly has joined #openstack-infra20:20
anteayasorry to distract from the real issue here, commit message can wait20:20
SamYapleclarkb: its a pain i agree. all i can say is it can be done20:20
clarkbianw: mmedvede we did have working builds last week they just happened to fail20:20
openstackgerritDaniel Wallace proposed openstack-infra/shade: allow for updating passwords in keystone v2
openstackgerritClark Boylan proposed openstack/diskimage-builder: Actually prune stale branches in src-repos
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystone starting with mitaka
ianwclarkb: no, i feel like the logging config in system-config is not matching on the logger for dib builds now, and the logs aren't getting put into's directory20:21
clarkbanteaya: ^ hopeflly the extra punctuation makes it readable20:21
AJaegerclarkb: there's a publish edits20:21
clarkbianw: ah right I think pabelanger did mention that20:21
anteayaclarkb: thank you, yes I can read that now20:21
clarkbso we just need to update the logging config20:21
AJaegerclarkb: it's different and non-intuitive -but if you get the hang for it, it works nice20:22
ianwpabelanger: ^ did you already do a fix that i missed?20:22
clarkblooks like dfw did have planned maintenance on networking20:23
*** annegentle has joined #openstack-infra20:23
clarkbits not clear if they actually performed such things20:23
anteayaclarkb: that might have affected es02?20:23
clarkbanteaya: yes (and 06 was off too)20:23
clarkbanteaya: basically what happens when nodes can't talk to each other for a prolonged period of time is they give up20:24
anteayamriedem: logstash is back20:24
clarkbso rather than run in a known broken state they just turn off20:24
mriedemyup, just noticed20:24
anteayaclarkb: I can understand that with elastic search20:24
anteayabetter than taking out a vm I guess20:24
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka
anteayaclarkb: thanks for making logstash come back20:25
*** tonytan4ever has joined #openstack-infra20:25
ianwclarkb: hmm, no "*" ( ) should get caught and sent to public log files.  can you please put in your queue to check the private log file and see if build logs are bubbling into there?20:26
clarkbcan do20:27
ianwahhhh ... unless the builder doesn't read the log config file...20:27
*** dtardivel has quit IRC20:27
*** doug-fish has joined #openstack-infra20:28
ianwwhich it doesn't20:28
clarkbits still in procedd though20:28
anteayaso 268841 and 268241, are both patches needed or is it one or the other?20:29
pabelangerclarkb: ianw : Yup:
pabelangerlike we do for zuul20:30
*** gildub has joined #openstack-infra20:31
ianwpabelanger: great, i'll review soon20:32
pabelangerianw: great20:32
clarkbianw: grepping for image in the debug log doesn't show any of the builds logs20:32
ianwclarkb: but yeah, the builder process isn't setting up any logging?  but pabelanger's change should fix taht20:33
clarkbianw: there is no separate builder process20:34
clarkb(not yet)20:34
ianwi thought it got merged, sorry, i'm a bit out of the loop on this bit20:34
clarkbit did, but it can run in process or out of process20:35
clarkbwe are currently running in process because that was the minimal amount of change to get it working and work through the issues20:35
clarkbavoided needing like 3 changes to merge all at once too20:35
ianwoh, ok, there's a sneaky self.builder = nodepool.builder.NodePoolBuilder(self.args.config) in nodepoold20:36
*** vgridnev has joined #openstack-infra20:36
ianwthen why have logs disappeared ...20:36
*** flwang has joined #openstack-infra20:37
*** sigmavirus24 is now known as sigmavirus24_awa20:37
*** hashar has quit IRC20:37
clarkbnot sure yet but I definitely don't see them in the main debug log so I don't think its just a matter of paths changing20:37
*** sigmavirus24_awa is now known as sigmavirus2420:37
ianwhmm, no.  setup_logging has still run, so that seems all ok20:37
clarkbwe should double check the devstack job to see that they are in there20:38
clarkbas that is a simpler setup20:38
*** markvoelker has joined #openstack-infra20:38
*** lucasagomes is now known as lucas-dinner20:39
ianwi wonder if it's something to do with "import nodepool.builder" *after* setup_logging20:39
clarkbmaybe? though nodepool.nodepool is imported late too20:40
*** weshay has quit IRC20:40
ianwbut *before* the setup_logging20:40
ianwi think we can get rid of these late imports with
clarkbeasy way to check is push a change that flips the order20:41
clarkband yes there is a change to fix the statsd stuff20:41
ianwalthough i'm not 100% sure about the comments on paramiko20:41
clarkbso maybe just review that20:41
clarkbwith paramkiko we may have to whitelist its fds20:42
*** otter768 has joined #openstack-infra20:42
clarkbianw: the statsd change you wrote lgtm but we need to undo the late import for builders in it or in a follow up change (since my workaround change merged first)20:44
clarkbianw: should I go ahead and ervert my change and we merge both together?20:44
ianwclarkb: yeah, well i wasn't sure about a full cleanup because of that paramiko comment.  it seems like we could just do the imports at the top as you say, if you think it's safe?20:45
clarkbfor paramiko I think we definitely need to do something more like whitelist the fds20:45
clarkbfor the builders I think we are ok as is ? let me check that it odesn't import paramiko too20:45
clarkbya no paramiko there20:46
adam_ganyone around to +W a requirements patch that needed a rebase over the weekend? thanks!
ianwahh, ok, then yeah we should be able to do builders.  i can put a quick stack together for that20:46
clarkbianw: for builders it was just statsd which yo uaddressed20:47
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystonemiddleware starting with mitaka
*** otter768 has quit IRC20:47
annegentleanyone have hints or tips for changing the email address associated with Launchpad's login/openid? Just changing it in email settings doesn't let me log in with that email address on
*** ashishb has quit IRC20:47
*** zeih has quit IRC20:47
clarkbannegentle: you may hav to do it through the ubuntu one interface20:50
*** vivekd has quit IRC20:50
annegentleclarkb: ah ok20:50
openstackgerritIan Wienand proposed openstack-infra/nodepool: Don't use global statsd
openstackgerritIan Wienand proposed openstack-infra/nodepool: Revert "Import nodepool.builder late for statsd"
ianwclarkb / mgagne / pabelanger: ^ i suspect that might get logs routing again, of course i haven't tested it to ensure that is true :)20:51
clarkbianw: the devstack run should tell us20:52
ianwpabelanger: can we shoehorn that into your nodepool instance?20:52
*** ZZelle_ has joined #openstack-infra20:52
clarkbnibalizer: hvae you have a chance to update your puppet for the nodepool builders?20:53
*** ihrachys has joined #openstack-infra20:53
nibalizerclarkb: this got updated20:54
nibalizeri need to fix tis
*** weshay has joined #openstack-infra20:55
*** annegentle has quit IRC20:55
*** amotoki has joined #openstack-infra20:55
*** annegentle has joined #openstack-infra20:55
mordrednibalizer: HEYA!20:55
mordrednibalizer: you wanna review some ansibles?20:55
nibalizermordred: sure20:56
nibalizerare we 2.0ing20:56
*** hichihara has joined #openstack-infra20:56
mordrednibalizer: is sitting over in the corner waiting for you to buy it a drink20:56
nibalizerclarkb: its annoying but I think we probably have to pipe it all through openstackci20:57
*** baoli has joined #openstack-infra20:57
*** vgridnev has quit IRC20:58
clarkbnibalizer: couldn't we just turn it on20:58
clarkbI am wondering if we need to make it toggleable now that the nodepool code has merged20:58
clarkbmaybe tag old nodepool and old puppet if we want to designate the cutoff20:59
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Use linters job for keystoneclient starting with mitaka
nibalizerclarkb: in 240093 we created a nodepool::builder class, we need to instantiate that somewhere20:59
nibalizerand I think the polite thing is to pipe that through an openstackci class21:00
clarkbright so openstackci should do it but I see on the sytem-config side we are passing a flag through which makes it a little more complicated21:00
clarkbjust wondering if we need the flag21:00
*** baoli has quit IRC21:00
*** eil397 has joined #openstack-infra21:02
stevemarAJaeger: you have all this on the same topic branch so i do have to go searching through reviews? :)21:02
nibalizerclarkb: thats a really old patch21:03
*** ihrachys_ has joined #openstack-infra21:04
*** eil397 has quit IRC21:04
*** eil397 has joined #openstack-infra21:05
clarkbyes well I thought we could get this in in tokyo :)21:05
stevemarAJaeger: ah there they all are:
*** yaume has quit IRC21:06
openstackgerritAndre keedy proposed openstack-infra/project-config: Create a new openstack project
*** ihrachy__ has joined #openstack-infra21:07
*** ihrachy__ is now known as ihrachysh21:07
Clintjeblair: i seem to have an endless-loop-in-syncing problem in gertty.. thoughts?21:07
*** apoorvad has quit IRC21:08
*** ihrachys has quit IRC21:08
*** daemontool has quit IRC21:08
*** ihrachys_ has quit IRC21:09
openstackgerritMerged openstack-infra/project-config: grafana: Add job average runtimes to Nodepool dashboard
*** achanda has joined #openstack-infra21:12
openstackgerritSpencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers
anteayaClint: he might suggest you running it in debug mode and paste the log21:14
anteayahe often makes that suggestion21:14
Clintanteaya: sensible21:15
anteayahe's like that21:15
openstackgerritSpencer Krum proposed openstack-infra/puppet-nodepool: Nodepool image build workers
*** achanda has quit IRC21:17
*** tonytan4ever has quit IRC21:20
openstackgerritSpencer Krum proposed openstack-infra/puppet-openstackci: Consume nodepool::builder class
nibalizerclarkb: 269280 is one way of doing it21:23
clarkbI don't see any recent images in nodepool, so it may be failing to run the builds at all explaining the lack of logs21:23
clarkbI did confirm that manually triggered builds were running and that dib ran it just couldn't succeed21:23
clarkbso need to dig into thtat more21:23
clarkbgreghaynes: ^21:23
nibalizerclarkb: are you wanting to colocate a builder today or start spinning up a little pool of builders?21:23
clarkbcolocate is what we will probably start with21:24
nibalizerI'd kind of prefer to go full worker pool rather than the current stuff in 26928021:24
clarkbthe problem with jumping to a full worker pool is nothing is working right now for a variety of reasons21:24
clarkbso it will be hard to separate new fails from old fails21:24
nibalizerhaha ok21:24
greghaynesclarkb: lol21:25
nibalizeralso openstackci is built on the assumption of total colocation so thats easy21:25
clarkbwe can't build dib images21:25
greghaynesclarkb: fun21:25
*** vgridnev has joined #openstack-infra21:25
clarkbsnapshots don't appear to be updating either21:25
clarkbso I am going to try a manual trigger21:25
greghaynesSo I suspect that isn't me then unless there is some tracebacks21:25
*** kgiusti has left #openstack-infra21:26
clarkbthe dib thing isn't21:26
clarkbbut the lack of logs and lack of new snapshot images is suspicious21:26
greghaynesclarkb: ya21:26
clarkbwhich reminds me to vote on the statsd changes21:26
openstackgerritEmilien Macchi proposed openstack-infra/project-config: puppet: initiate testing for puppet-openstack-spec-helper
clarkbmanually triggered image builds are doing things21:29
*** keedya_ has quit IRC21:33
*** abregman|nb has quit IRC21:33
*** Keedya_ has joined #openstack-infra21:34
mordredclarkb: I look forward to learning things21:35
mordrednibalizer: I'd love to land so that I can verify that it's copying the right files21:37
mordredclarkb: also, I started work on getting the nodepool-shade patch up to speed21:37
mordredclarkb: the patch that's up in gerrit is not ready - but we're looking good21:37
clarkbmordred: is it the same change as before just updated?21:38
mordredclarkb: mostly21:38
mordredclarkb: there are a few differences, and I have a few more locally21:38
*** tonytan4ever has joined #openstack-infra21:38
*** Keedya_ has quit IRC21:38
mordredclarkb: because we added several things to shade since last we updated that patch based on things we needed from that work21:38
mordredso going back through and actually taking advantage of all of those things is what I'm doing currently - when I'm not trying to convince people to review puppet-apply patches :)21:39
greghaynesclarkb: I put up a nodepool image build failure fix too, btw21:39
clarkbgreghaynes: for the uncatching of fails?21:39
greghaynesclarkb: I kind of want to write a test case21:39
* clarkb goes to review that one21:39
*** yaume has joined #openstack-infra21:40
clarkbthen if we can get the statsd fix, logging fix, and fail fixes in before next restart we should be in good shape to make sure things are working again21:40
clarkbgreghaynes: looks like the dsvm jobs timed out so I have rechceked it21:40
*** apoorvad has joined #openstack-infra21:42
clarkbgreghaynes: for testing that you can setup an image build fail via the env vars for the image. We do this on at least one other image so probably that test isn't strong enough21:42
*** apoorvad has quit IRC21:42
clarkbbut maybe we need a new test too21:42
openstackgerritTristan Cacqueray proposed openstack-infra/project-config: zuul_swift_upload add support for --verbose
*** apoorvad has joined #openstack-infra21:43
greghaynesclarkb: yep, there is an image build fail test case but obviously it doesn't test enough21:43
greghaynesclarkb: I just haven't had time to look at it yet21:43
clarkbgreghaynes: test_dib_and_snap_fail is the existing test21:43
clarkbok the assertion is just that no nodes build21:44
anteayacrinkle: thanks for the etherpad21:44
clarkbso can be tightened up21:44
greghaynesclarkb: ya, easy enough21:44
ianwclarkb: so from what i can see, the issue with paramiko came down to creting a global Crypto.Random() object which was created at import time.  if you daemonize after that import it's fd for /dev/urandom got killed21:46
ianwclarkb: however, it seems like they don't do that any more ->
anteayamorning jhesketh21:46
ianwclarkb: ergo, i think with a bit of testing/version checking we can remove that bit too21:47
clarkbianw: woot21:47
crinkleanteaya: no problem21:48
*** yaume has quit IRC21:49
*** weshay has quit IRC21:49
clarkbianw: just need to figure out what the first release that fixed that is and don't allow anything older in requirements21:50
clarkbianw: then the change should be self testing in the dsvm jobs since they ssh into real VMs and do all that21:50
*** shashank_hegde has joined #openstack-infra21:50
clarkbianw: so its easy to have the computer check that it works for us if you want ot go ahead and push the change to bump the min version21:50
*** jtomasek has quit IRC21:50
*** shardy has quit IRC21:51
*** ericksonsantos has quit IRC21:51
openstackgerritMerged openstack-infra/system-config: Add script to make ansible groups from patterns
ianwyeah, 1.11.6 is the first after that commit ... good place to start21:52
*** jflynn has joined #openstack-infra21:53
*** burgerk has quit IRC21:53
*** vgridnev has quit IRC21:53
nibalizermordred: im curious where we picked up 'hieraenvironment'21:53
nibalizerthere are only puppet environments21:53
nibalizerhiera doesn't have them21:53
mordrednibalizer: it's a parameter21:55
mordrednibalizer: possibly poorly named21:55
mordrednibalizer: should we call it puppet environment instead?21:55
nibalizernot the most important21:55
nibalizerbut wiggs me every time I see it21:55
openstackgerritMerged openstack-infra/ansible-puppet: Add puppet module sync to the role
clarkbmanual snapshot build looks already so we may just need fixed logging in order to diagnose the daemonized stuff21:56
clarkbI have +2'd the two changes to do that21:56
mordrednibalizer: k. then we should update it fo sho21:56
clarkbbut should confirm in the dsvm logs that it looks correct21:57
*** tdurakov_ has quit IRC21:57
*** bcourt has quit IRC21:58
mordrednibalizer: I'll send in a follow up patch22:00
*** bknudson_ has quit IRC22:02
*** kzaitsev1mb has joined #openstack-infra22:02
*** jordanP has joined #openstack-infra22:02
*** edmondsw has quit IRC22:02
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Rename hieraenvironment to environment
mordrednibalizer: ^^ thereyago22:02
*** kzaitsev_mb has quit IRC22:02
*** weshay has joined #openstack-infra22:03
clarkbmordred: should come with a documentation update since that is something roots have to know to do22:04
*** jpr2 has quit IRC22:06
nibalizermordred: for that, is environment really not a resereved word in ansible22:06
openstackgerritgordon chung proposed openstack-infra/project-config: use ceilometerclient from git
*** moravec has joined #openstack-infra22:07
mordrednibalizer: oh! yes it is.22:08
openstackgerritIan Wienand proposed openstack-infra/nodepool: Remove workaround for early paramiko import
mordrednibalizer: gross. I guess we have to call it "puppet_environmnet" then?22:08
nibalizerwhat is the default ansible environment22:09
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Delete old queries for fixed bugs
*** jamielennox|away is now known as jamielennox22:09
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Rename hieraenvironment to puppetenvironment
*** gildub has quit IRC22:10
nibalizermordred: hrm so in i've been doing puppet_thing for config directives22:10
*** bcourt has joined #openstack-infra22:11
*** ociuhandu has joined #openstack-infra22:11
mordrednibalizer: I hate you22:11
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Rename hieraenvironment to puppet_environment
*** chlong has joined #openstack-infra22:12
mordrednibalizer: maybe this time? :)22:12
fungiwow. puppet_thing is an element in the puppet_stuff list, right?22:12
* fungi is hoping22:12
nibalizerfungi: naturally22:12
clarkboh I know the images that failed are still in "building" state in the db22:13
clarkbso nodepool says "I have nothing to do"22:13
mordredclarkb: woot22:13
clarkbso ya we should get greghaynes' fix in22:13
fungiahh, because of the failures not updating the db issue?22:13
*** regXboi has quit IRC22:13
clarkbfungi: yup22:13
fungiclarkb: greghaynes: what's the patch number on that?22:13
clarkbI rechecked it to see if the dsvm fails are consistent22:13
clarkblooks like they timed out22:13
clarkbwhcih may be just due to bluebox22:14
*** bpokorny has joined #openstack-infra22:14
fungithe source one ran in ovh22:14
fungibut both hit the timeout22:15
clarkbit looks like 1 of 2 of the VMs we boot managed to start22:15
*** peristeri has quit IRC22:16
openstackgerritMerged openstack-infra/ansible-puppet: Update to use blocks from ansible 2.0
openstackgerritMerged openstack-infra/ansible-puppet: Clean up codestyle in conditionals
*** dims has joined #openstack-infra22:17
clarkb(I am just happy I got to the bottom of why no builds were happening)22:17
clarkbI am tempted to delete them fro mthe db to confirm we get new builds tomorrow, but those will fail so \o/22:17
*** treaki_ has quit IRC22:17
*** eil397 has quit IRC22:17
clarkbI guess I should get ianw's prune fix for project-config in and maybe thats sufficient22:17
mordredclarkb, fungi: is there any way to get emacs to not write backup files in a given directory22:17
*** dims_ has quit IRC22:18
clarkbmordred: emacs?22:18
mordredyeah - that foo~ files22:18
clarkb(I honestly don't know, too much of a vim user)22:18
*** dkranz has quit IRC22:18
mordredcurrently there is a file - /etc/ansible/hosts/emergency~22:18
fungimordred: you want to change that behavior on a per-directory way?22:19
*** julim has quit IRC22:19
openstackgerritAndre keedy proposed openstack-infra/project-config: Needed-By: I9fa0ae1831baf6c17ea58d517b77cbc166fa408f
mordredfungi: maybe?22:20
fungimordred: you can tell emacs to put them all somewhere else22:20
ianwclarkb: i'm still trying to convince myself too, i don't feel like i fully understand why we're getting those branches there22:20
clarkbianw: ++ it doesn't hurt to have that change in there so I have approved it. Reviewing the script simplification too22:20
mordredfungi: for some reason, my ssh connections time out fairly constantly - _only_ to puppetmaster.o.o22:20
mordredwhich confuses me22:21
*** Keedya_ has joined #openstack-infra22:21
fungimordred: established ssh connections or ssh connection attempts?22:21
mordredfungi: both22:21
fungihuh. packet loss? are you over ipv4 or ipv6?22:21
greghaynesfungi: I don't have the patch on hand right now from phone... clarkb?22:22
greghaynes(Build fail handling)22:23
fungigreghaynes: he got it to me, thanks22:23
fungimordred: tried leaving a ping running to it and seeing if you get icmp timeouts around the same time as your ssh connection timeouts?22:23
clarkbianw: +2 on the script simplification with the caveat that we may need to update it to handle pruning based on further investigation22:23
mordredfungi: good idea22:24
*** Keedya_ has quit IRC22:24
mordredbtw - I have verified that ansible appropriately ignores the emacs autosave files22:24
mordredin the inventory dir22:24
mordredso, nevermind on the making them go away thing22:24
fungimordred: for maximum win, leave another ping going to an address at the same rackspace region as a control group22:24
clarkbI like mtr fwiw22:25
fungidid it start recently?22:25
mordredfungi: not terribly recently, no22:25
mordredpast few months22:25
fungiyeah, mtr is helpful if the loss isn't at a terminus of the connection22:26
*** aysyd has quit IRC22:26
mordredfungi: nope. timeout to puppetmaster ssh session - pings still chugging along22:26
*** eil397 has joined #openstack-infra22:26
fungimordred: interesting, i haven't been noticing puppetmaster ssh timeouts for the past several months, though i'm also connecting mostly over ipv622:26
*** ddieterly has quit IRC22:27
fungimordred: does the puppetmaster clean up the disconnected session on its end? just noticing that one of us has a couple ssh sessions connected over ipv4 for many days, so that suggests it might be something singling you out22:28
openstackgerritMerged openstack-infra/project-config: Prune old branches in early devstack clone
*** jordanP has quit IRC22:30
*** doug-fish has quit IRC22:31
SwansonIn nodepool.yaml what is  the targets: section for?22:31
clarkbSwanson: tells nodepool what jenkins matsers to attach nodes to22:31
*** doug-fish has joined #openstack-infra22:31
openstackgerritMonty Taylor proposed openstack-infra/system-config: Exclude hosts header line in expand-groups
fungispecifically, the jenkins master api endpoint urls22:31
fungiand related details22:32
mordredfungi, nibalizer: ^^ that patch - if'n you don't mind - just found that when the patch landed and I ran it on puppetmaster22:32
mordredhave tested the fix22:32
Swansonclarkb, fungi, thanks.  I must have missed it first go round and it just sez jenkins1 which has little to do with my setup.22:33
fungimordred: any risk that will greedily filter things we don't want? like a host named "thishost"? (okay, silly example)22:33
*** doug-fis_ has joined #openstack-infra22:34
nibalizermordred: what do you think about adding -m122:34
nibalizerI have similar concerns to fungi22:34
fungiyeah, limiting to the first match may make more sense22:34
mordredwe could also do ^ +hosts \([0-9]+\):22:34
fungithat seems a lot safer22:35
openstackgerritMonty Taylor proposed openstack-infra/system-config: Exclude hosts header line in expand-groups
mordredhow about that? ^^22:35
fungii feel fairly confident we'll never have a hostname matching that pattern ;)22:35
*** doug-fish has quit IRC22:36
nibalizeruh that won't work -m and -v don't combine the way I thought they did sorry22:36
*** erlon has quit IRC22:36
*** pvaneck has quit IRC22:36
*** doug-fish has joined #openstack-infra22:36
nibalizercan just drop the -m wth such a specific regex tho22:37
clarkbI am a little fuzzy on what those groups are for, is it purely convenience?22:37
fungiclarkb: conveniently applying a common role (i might not be using that word correctly) to a set of similar hosts22:37
*** doug-fis_ has quit IRC22:38
openstackgerritChristopher Aedo proposed openstack-infra/system-config: Add PyYAML to proposal slave
fungiso i guess you could call that a convenience22:38
openstackgerritMerged openstack/requirements: Update global-requirements.txt
clarkbit just seems like a lot of overhead when ansible-playbook logstash-worker* would work?22:38
ianwclarkb: so i think this explains what is going on ->
*** dizquierdo has joined #openstack-infra22:40
clarkbianw: if you run git branch -a after the lsat prune you should still have a stable/juno though22:40
clarkbianw: but you don't have an origin/stable/juno22:40
ianwclarkb: actually, the opposite?
*** moravec has quit IRC22:42
*** camunoz has quit IRC22:42
clarkbwah, hrm maybe I misread it when I was testing then22:42
anteayafungi: did you mean to vote +2 and then remove that vote on ? I ask because you usually say why either on the patch or in channel if you change your vote22:42
*** ihrachysh has quit IRC22:42
clarkbianw: in that case I think my just do a git remote update --prune may be sufficient22:43
anteayaso I wasn't sure if it was intentional or a gerrit derp22:43
pabelangerianw: sure22:43
clarkbianw: since it should update all the reachable refs and prune the invalid onces22:43
openstackgerritMonty Taylor proposed openstack-infra/system-config: Exclude hosts header line in expand-groups
*** otter768 has joined #openstack-infra22:44
mordrednibalizer, fungi: ^^22:44
*** dims has quit IRC22:44
keedyahi There22:46
keedyaHow can i delete a review?22:46
openstackgerritColleen Murphy proposed openstack-infra/puppet-gerrit: Add logic to use use available mysql module
*** eliqiao has quit IRC22:46
mordredkeedya: you cannot. but you can abandon it22:46
mordredkeedya: if you go to the review, you'll see an abandon button on it22:47
*** hichihara has quit IRC22:47
ianwclarkb: so actually, maybe the devstack-early clone wants to do "git fetch --prune --update-head-ok ../upstream +refs/heads/*:refs/heads/* +refs/remotes/*:refs/remotes/*"22:47
anteayakeedya: is this yours?
ianwclarkb: is that actually an alias for what you actually said though?22:47
*** dims has joined #openstack-infra22:47
ianwyay for a million ways to do the same thing22:47
keedyayes this is me22:48
*** mriedem is now known as mriedem_away22:48
clarkbianw: ya I think git remote update is basically that22:48
keedyai m new to the process22:48
clarkbianw: but ti does it for each remote22:48
keedyai need to update my files22:48
*** eliqiao has joined #openstack-infra22:48
anteayakeedya: great, start here:
*** otter768 has quit IRC22:48
*** achanda has joined #openstack-infra22:48
anteayakeedya: you prefer you update a change, rather than abandoning it and creating a new change22:49
keedyawill do22:49
anteayakeedya: if you need help, just ask, we can then find some documentation that helps you understand our workflow22:49
anteayakeedya: welcome22:49
*** achanda has quit IRC22:50
anteayas/you prefer you update a change/we prefer you update a change22:50
keedyaanteaya i am trying to add a project written in nodejs22:50
anteayayeah I saw that patch22:50
keedyai m having hard time finding the right documentations22:50
anteayacan you help me understand why you want it in nodejs?22:50
*** weshay has quit IRC22:50
anteayamost of our projects are in python22:50
anteayaso having a nodejs project, well we havn't seen a lot of them22:51
anteayacan you share some detail?22:51
keedyafollowing is the code and description22:52
*** achanda has joined #openstack-infra22:52
*** sabeen3 has quit IRC22:52
anteayatell me the part why this has to be in nodejs and it won't function in python22:53
*** harlowja_at_home has joined #openstack-infra22:55
mordredkeedya: neat - devananda have you seen that? ^^22:55
keedyano major reason22:56
*** amitgandhinz has quit IRC22:56
*** tonytan4ever has quit IRC22:56
*** thorst has quit IRC22:57
clarkbthe jenkins rolling restarts appear to have completed successfully last friday/saturday22:58
*** ddieterly has joined #openstack-infra22:59
*** gildub has joined #openstack-infra22:59
*** camunoz has joined #openstack-infra22:59
anteayayay success23:00
*** ari has quit IRC23:01
clarkbnibalizer: so you have nodepool running and you are able to run d-g on the built nodes?23:03
clarkbnibalizer: have you managed to test the run times on >8 cpus?23:03
*** eil397 has quit IRC23:04
*** bknudson has joined #openstack-infra23:05
*** dims has quit IRC23:05
ianwclarkb: so, why would we do "git fetch --update-head-ok ../upstream/ +refs/heads/*:refs/heads/*"  ... that says "get all refs/heads from remote server and put it in my refs/heads ... i.e. create local branches23:05
openstackgerritAndre keedy proposed openstack-infra/project-config: correct syntax error in shovel.config
ianwclarkb: that seems like it should be +refs/heads/*:refs/remotes/origin/*23:06
ianwi.e. take the remote heads, and put them under remotes/* locally23:06
*** amotoki has quit IRC23:06
*** dims has joined #openstack-infra23:07
clarkbianw: hrm that is a good question, probably to avoid needing to setup tracking branches?23:07
clarkbmake it more of a mirror instead of a clone?23:07
mordredkeedya: so - what you'll want to do is squash those two commits into a single commit23:08
ianwthis is the type of thing a comment in the code could help with :(23:08
mordredkeedya: gerrit tracks changes, not branches - when you squash them, make sure you throw away the Change-Id line of the second commit23:08
mordredkeedya: that way gerrit will know to associate the new modified patch with the original patch23:08
nibalizerclarkb: yes nodepool works well enough23:09
nibalizerbut every time i tried d-g last week it timed itself out at 115 minutes23:09
nibalizerI'll kick off another run23:09
keedyagot it!!23:10
*** camunoz has quit IRC23:10
mordredkeedya: woot!23:10
ianwclarkb: so you're right, the --prune in that latest commit will still not be sufficient, because the devstack image job is still going to find the refs/remotes branches and try to checkout icehouse23:10
keedyathanks for ur help!!!23:11
fungianteaya: i removed it after nibalizer mentioned that -m1 combined with -v didn't test out after all23:11
fungilatest patchset lgtm though23:12
*** baoli has joined #openstack-infra23:12
anteayafungi: ah thank you23:12
clarkbianw: ya so we can do something like my naive change to iterate through and delete those that are "extra" though I think I need to update it to delete the remote branche instead of the local branches23:13
*** achanda has quit IRC23:14
clarkbI kind of assumed that git would have a better way to do that but I couldn't find one hence my brute force appraoch23:14
*** sigmavirus24 is now known as sigmavirus24_awa23:15
mordredkeedya: my pleasure!23:16
*** aeng has joined #openstack-infra23:16
clarkbianw: we could look at gerrit's replication process and copy cat that. Though it does a push not pull23:16
openstackgerritfumihiko kakuma proposed openstack-infra/project-config: Run the ofagent ci job in the periodic-qa queue
*** baoli has quit IRC23:17
*** camunoz has joined #openstack-infra23:22
*** achanda has joined #openstack-infra23:23
openstackgerritMerged openstack-infra/system-config: Exclude hosts header line in expand-groups
ianwclarkb: i wonder if "git clone --mirror" didn't exist when this was all written ...23:25
*** rfolco has quit IRC23:26
clarkbmaybe? though these are not bare repos maybe they should be23:26
clarkbalso git clone --references23:28
clarkber --reference23:28
clarkboh hrm that may not cross filesystems well23:28
*** doug-fish has quit IRC23:28
clarkbya for image builds that isn't so great, however we could look into using that instead of rsync for zuul-cloner23:29
clarkbjeblair: ^23:29
ianwso i think "git fetch --prune --update-head-ok ../upstream +refs/*:refs/*" is getting closer to what we want23:29
ianwand is really what a --mirror does23:30
*** dimtruck is now known as zz_dimtruck23:30
clarkbianw: and that will delete the stale branches via the --prune?23:30
ianwclarkb: yes, everything (remotes too)23:31
clarkbbeacuse we start at refs/ that makes sense23:31
ianwi think dib's fetch of +master:master is wrong23:31
openstackgerritMonty Taylor proposed openstack-infra/system-config: You have to use egrep to get real regexes
mordredclarkb, fungi, nibalizer: ^^ sigh23:31
clarkbianw: refs/* includes master so it should still work for anyone relying on  master23:32
*** tsufiev has quit IRC23:32
*** dteselkin has quit IRC23:32
*** rakhmerov has quit IRC23:32
clarkbianw: but I agree master alone is quite restrictive23:32
*** katyafervent has quit IRC23:33
*** salv-orlando has joined #openstack-infra23:33
*** harlowja_at_home has quit IRC23:33
jheskethinfra-core: If anybody has time to help move the swift logs along the patches need reviewing (including the ones with jenkins -1 as they should work once the zuul patches are in)23:34
*** gordc has quit IRC23:34
*** dizquierdo has quit IRC23:35
*** dims has quit IRC23:35
*** gildub has quit IRC23:35
*** pahuang has joined #openstack-infra23:36
*** rakhmerov has joined #openstack-infra23:37
*** katyafervent has joined #openstack-infra23:37
*** davidlenwell has quit IRC23:38
*** dteselkin has joined #openstack-infra23:38
mordredinfra-root: I self-approved the above patch because puppetmaster is otherwise broken (and it's a single character)23:38
*** tsufiev has joined #openstack-infra23:39
*** dims has joined #openstack-infra23:39
*** dslev has quit IRC23:40
dougwigmordred: no day is complete without a rage merge.23:40
mordreddougwig: +10023:41
nibalizermordred: kk23:41
*** Qiming has joined #openstack-infra23:41
devanandakeedya: neat!23:42
clarkbjhesketh: the zuul changes lgtm23:43
*** claudiub has quit IRC23:43
jheskethclarkb: ta23:43
*** eliqiao_ has joined #openstack-infra23:45
ianwclarkb:  * [new ref]         refs/changes/99/71199/5 -> refs/changes/99/71199/5 <-- so this is a bit of a problem with "+refs/*:refs/*" ... it gets all these which we probably don't want23:47
clarkbianw: so maybe we need to do refs/heads and refs/remotes?23:48
ianwwhich is maybe why it was done like that ... and refs/remotes was just forgotten23:48
*** pahuang has quit IRC23:49
*** eliqiao has quit IRC23:49
keedyadevanda: i m trying to add shovel to openstack CI23:49
*** amotoki has joined #openstack-infra23:50
keedyaShovel is written in nodejs23:50
*** achanda has quit IRC23:50
openstackgerritMerged openstack-infra/system-config: You have to use egrep to get real regexes
*** otter768 has joined #openstack-infra23:52
*** Swanson has left #openstack-infra23:52
*** pahuang has joined #openstack-infra23:52
*** yuanying has quit IRC23:54
*** rockyg has quit IRC23:54
mordredinfra-root: ok - the ansible group generation seems to be working now23:54
mordredI will doublecheck that all of the tings are being copied where we expect all the things to be copied to tomorrow23:55
*** rajinir has quit IRC23:55
mordredand if that is happening, I plan on getting medevial on finishing up puppet-apply23:55
mordredbecause it's on my last nerve23:55
*** yuanying has joined #openstack-infra23:56
*** gildub has joined #openstack-infra23:57
*** davidlenwell has joined #openstack-infra23:58
openstackgerritMerged openstack-infra/ansible-puppet: Perform filepath checking in python
openstackgerritMerged openstack-infra/ansible-puppet: Collapse redundant hieradata tasks

Generated by 2.14.0 by Marius Gedminas - find it at!