Thursday, 2016-05-12

*** thorst__ has quit IRC00:00
*** thorst__ has joined #openstack-infra00:00
*** thors____ has quit IRC00:01
*** thorst___ has joined #openstack-infra00:01 seems to be unhappy right now00:01
*** thor_____ has quit IRC00:02
clarkbJayF: daily db backups should be happy in a minute or two00:02
JayFah, midnight utc?\00:02
*** bpokorny_ has joined #openstack-infra00:02
fungigerrit gets similarly slow at the same time (and for the same reasons)00:02
nibalizerits a nice whistle at the end of the day for west coasters00:03
*** thors____ has joined #openstack-infra00:03
fungishift change!00:03
*** thorst_ has quit IRC00:03
* fungi swaps his left shift key for the right one00:03
fungimuch better00:04
*** thorst_ has joined #openstack-infra00:04
*** thor_____ has joined #openstack-infra00:05
*** thorst__ has quit IRC00:05
*** bpokorny has quit IRC00:05
*** thorst___ has quit IRC00:05
*** thorst__ has joined #openstack-infra00:06
*** thorst___ has joined #openstack-infra00:07
*** bpokorny_ has quit IRC00:07
*** cody-somerville has quit IRC00:07
*** tho______ has joined #openstack-infra00:08
*** thors____ has quit IRC00:08
*** IlyaG has quit IRC00:08
*** thorst_ has quit IRC00:08
*** thorst_ has joined #openstack-infra00:08
*** thors____ has joined #openstack-infra00:09
*** thor_____ has quit IRC00:09
*** thorst__ has quit IRC00:10
*** jgriffith_ has joined #openstack-infra00:11
*** thorst___ has quit IRC00:11
*** signed8bit is now known as signed8bit_Zzz00:11
*** tho______ has quit IRC00:12
*** thorst__ has joined #openstack-infra00:12
*** thorst_ has quit IRC00:13
*** thorst_ has joined #openstack-infra00:13
*** thors____ has quit IRC00:14
*** ddieterly has joined #openstack-infra00:14
*** dimtruck is now known as zz_dimtruck00:14
*** thorst___ has joined #openstack-infra00:14
*** thors____ has joined #openstack-infra00:15
*** thor_____ has joined #openstack-infra00:16
*** thorst__ has quit IRC00:17
*** thorst__ has joined #openstack-infra00:17
*** thorst_ has quit IRC00:18
*** _ari_ has joined #openstack-infra00:18
*** thorst_ has joined #openstack-infra00:18
*** _ari_ has quit IRC00:18
*** thorst___ has quit IRC00:18
*** jgriffith has quit IRC00:19
*** thors____ has quit IRC00:19
*** thorst___ has joined #openstack-infra00:19
*** thors____ has joined #openstack-infra00:20
*** thor_____ has quit IRC00:20
*** thor_____ has joined #openstack-infra00:21
*** tho______ has joined #openstack-infra00:22
*** jsavak has quit IRC00:22
*** thorst__ has quit IRC00:22
*** thorst_ has quit IRC00:23
*** thorst_ has joined #openstack-infra00:23
*** thorst___ has quit IRC00:23
*** thors____ has quit IRC00:24
*** thorst__ has joined #openstack-infra00:24
*** thorst___ has joined #openstack-infra00:25
*** thor_____ has quit IRC00:25
*** tho______ has quit IRC00:26
*** thors____ has joined #openstack-infra00:26
*** kmartin has quit IRC00:26
*** thor_____ has joined #openstack-infra00:27
*** thorst_ has quit IRC00:27
*** thorst_ has joined #openstack-infra00:28
*** thorst__ has quit IRC00:29
*** thorst__ has joined #openstack-infra00:29
*** thorst___ has quit IRC00:30
*** thorst___ has joined #openstack-infra00:30
*** thors____ has quit IRC00:31
*** thors____ has joined #openstack-infra00:31
*** thor_____ has quit IRC00:31
*** thorst_ has quit IRC00:32
*** thorst_ has joined #openstack-infra00:32
*** duobei has joined #openstack-infra00:33
*** thor_____ has joined #openstack-infra00:33
*** tho______ has joined #openstack-infra00:34
*** thorst__ has quit IRC00:34
*** thorst___ has quit IRC00:35
*** thorst__ has joined #openstack-infra00:35
*** zz_dimtruck is now known as dimtruck00:35
*** thors____ has quit IRC00:36
*** hichihara has joined #openstack-infra00:36
*** gyee has quit IRC00:36
openstackgerritMerged openstack/diskimage-builder: Fix OpenSUSE support
*** thorst___ has joined #openstack-infra00:37
*** thorst_ has quit IRC00:37
*** thor_____ has quit IRC00:37
*** thorst_ has joined #openstack-infra00:38
*** tho______ has quit IRC00:38
*** thors____ has joined #openstack-infra00:39
*** thorst__ has quit IRC00:39
*** thorst__ has joined #openstack-infra00:40
*** amitgandhinz has joined #openstack-infra00:40
*** thor_____ has joined #openstack-infra00:41
*** thorst___ has quit IRC00:41
*** mtanino has quit IRC00:41
*** thorst___ has joined #openstack-infra00:42
*** thorst_ has quit IRC00:42
*** thorst_ has joined #openstack-infra00:43
*** thors____ has quit IRC00:43
*** thors____ has joined #openstack-infra00:44
*** thorst__ has quit IRC00:44
*** amitgandhinz has quit IRC00:44
*** thorst__ has joined #openstack-infra00:45
*** thor_____ has quit IRC00:45
*** baoli has joined #openstack-infra00:45
*** lifeless has quit IRC00:46
*** thorst___ has quit IRC00:46
*** lifeless has joined #openstack-infra00:47
*** thorst___ has joined #openstack-infra00:47
*** derekh has joined #openstack-infra00:47
*** thorst_ has quit IRC00:47
*** thorst_ has joined #openstack-infra00:48
*** thors____ has quit IRC00:48
derekhfungi: the tripleo cloud isn't coming back tonight, our jump host is refusing to get past the POST process and without it we got no access, we need to wait until tomorrow to find others with appropriate access to figure it out00:49
*** thors____ has joined #openstack-infra00:49
*** thorst__ has quit IRC00:49
derekhfungi: I guess you might be gone, will send a mail to the list00:49
*** thorst__ has joined #openstack-infra00:50
*** thor_____ has joined #openstack-infra00:51
*** thorst___ has quit IRC00:52
fungiderekh: yeah, i'm essentially gone, but thanks for the heads up!00:52
derekhfungi: no prob00:52
*** thorst___ has joined #openstack-infra00:52
*** dimtruck is now known as zz_dimtruck00:52
*** zz_dimtruck is now known as dimtruck00:52
*** thorst_ has quit IRC00:53
*** gildub has quit IRC00:53
*** thors____ has quit IRC00:53
*** gildub has joined #openstack-infra00:53
*** rcernin has quit IRC00:54
*** thorst_ has joined #openstack-infra00:54
*** thorst__ has quit IRC00:55
*** thorst__ has joined #openstack-infra00:55
*** thor_____ has quit IRC00:56
*** fitoduarte has quit IRC00:56
*** thors____ has joined #openstack-infra00:56
*** thorst___ has quit IRC00:56
*** gildub_ has joined #openstack-infra00:57
*** thorst___ has joined #openstack-infra00:57
*** thor_____ has joined #openstack-infra00:58
*** thorst_ has quit IRC00:58
*** thorst_ has joined #openstack-infra00:59
*** gildub has quit IRC00:59
*** thorst__ has quit IRC01:00
*** mixos has joined #openstack-infra01:00
*** zhurong has joined #openstack-infra01:00
*** thorst__ has joined #openstack-infra01:01
*** thors____ has quit IRC01:01
*** thorst___ has quit IRC01:01
*** thorst___ has joined #openstack-infra01:01
pabelangerokay, I think I deleted all the USED nodes we leaked.01:02
*** dimtruck is now known as zz_dimtruck01:02
pabelangergate looks much better now01:02
*** Qiming_ has joined #openstack-infra01:02
*** MarkAtwood has quit IRC01:02
*** thor_____ has quit IRC01:02
*** thors____ has joined #openstack-infra01:03
*** thorst_ has quit IRC01:03
*** thorst_ has joined #openstack-infra01:04
*** markvoelker_ has joined #openstack-infra01:05
*** thorst__ has quit IRC01:05
*** thor_____ has joined #openstack-infra01:05
ianwjeblair / others : i have the first part of the bandersnatch index change merged into trunk with
ianwi'll do the second part, then figure out getting it back into mercurial and proposing as a pull request01:06
*** thorst___ has quit IRC01:06
*** thorst__ has joined #openstack-infra01:06
*** tlian has quit IRC01:06
*** thorst___ has joined #openstack-infra01:07
*** thors____ has quit IRC01:08
*** signed8bit_Zzz is now known as signed8bit01:08
*** thorst_ has quit IRC01:09
*** thors____ has joined #openstack-infra01:09
*** derekh has quit IRC01:09
*** thorst_ has joined #openstack-infra01:10
*** thor_____ has quit IRC01:10
*** thorst__ has quit IRC01:10
*** thorst__ has joined #openstack-infra01:10
*** thorst___ has quit IRC01:11
*** tphummel has quit IRC01:11
*** thorst___ has joined #openstack-infra01:11
*** thor_____ has joined #openstack-infra01:12
*** tho______ has joined #openstack-infra01:13
*** thors____ has quit IRC01:13
*** thors____ has joined #openstack-infra01:14
*** thorst_ has quit IRC01:14
*** mriedem has quit IRC01:15
*** thorst__ has quit IRC01:15
*** thorst_ has joined #openstack-infra01:15
*** thorst___ has quit IRC01:16
openstackgerritMerged openstack-infra/release-tools: add extra metadata to the tag messages
*** thorst__ has joined #openstack-infra01:16
*** thorst__ has quit IRC01:16
*** thorst__ has joined #openstack-infra01:16
*** thor_____ has quit IRC01:17
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Make sure worker arguments are integers
*** tho______ has quit IRC01:18
pabelangerclarkb: fungi: Well need to land ^. nodepool-builder was confusing the args as strings01:18
*** thorst__ has quit IRC01:19
*** thors____ has quit IRC01:19
*** thorst_ has quit IRC01:20
*** pvaneck has quit IRC01:20
*** smarcet has joined #openstack-infra01:27
*** hongbin has joined #openstack-infra01:32
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Add function testing for nodepoold workers
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Add unit testing for nodepoold workers
*** smarcet has quit IRC01:36
*** mixos has quit IRC01:36
openstackgerritNate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting
*** vhosakot has joined #openstack-infra01:39
*** amitgandhinz has joined #openstack-infra01:40
*** Goneri has quit IRC01:41
*** zhurong has quit IRC01:42
*** zz_dimtruck is now known as dimtruck01:42
*** zhurong has joined #openstack-infra01:43
*** amitgandhinz has quit IRC01:45
*** amotoki has joined #openstack-infra01:46
*** gdubreui has joined #openstack-infra01:47
*** amitgandhinz has joined #openstack-infra01:48
*** tqtran has quit IRC01:48
*** gildub_ has quit IRC01:49
*** salv-orlando has joined #openstack-infra01:51
*** markvoelker_ has quit IRC01:51
*** yuanying has quit IRC01:51
*** yuanying has joined #openstack-infra01:52
*** yuanying has quit IRC01:52
*** yuanying has joined #openstack-infra01:53
*** Goneri has joined #openstack-infra01:54
*** salv-orlando has quit IRC01:56
*** markvoelker has joined #openstack-infra01:59
*** markvoelker_ has joined #openstack-infra01:59
*** kengo_sakai has joined #openstack-infra02:02
*** markvoelker has quit IRC02:03
*** zhurong has quit IRC02:04
*** zhurong has joined #openstack-infra02:05
openstackgerritMerged openstack-infra/nodepool: Cap paramiko<2.0.0
*** zqfan has joined #openstack-infra02:14
*** ccamacho has quit IRC02:14
*** ccamacho has joined #openstack-infra02:15
*** smarcet has joined #openstack-infra02:16
*** apoorvad has quit IRC02:19
*** Goneri has quit IRC02:20
*** tonytan4ever has joined #openstack-infra02:21
*** rfolco has quit IRC02:22
*** ddieterly has quit IRC02:23
*** Daisy has joined #openstack-infra02:29
hongbinyolanda: clarkb AJaeger : anyone there?02:30
*** markvoelker_ has quit IRC02:31
*** vhosakot has quit IRC02:34
*** Swami_ has joined #openstack-infra02:34
*** nadya has joined #openstack-infra02:34
*** Daisy has quit IRC02:37
*** binbincong has quit IRC02:37
*** Swami has quit IRC02:38
*** daemontool has quit IRC02:38
*** markvoelker_ has joined #openstack-infra02:38
*** gdubreui has quit IRC02:39
*** gildub has joined #openstack-infra02:39
*** nadya has quit IRC02:39
*** signed8bit has quit IRC02:40
hongbinyolanda: clarkb AJaeger Maybe I leave the message. I need your help to add me to the gerrit groups: higgins-core, higgins-release02:46
ianwi'm sure mercurial is great, but I really only have room for one dvcs in my life02:47
ianw <- i think this in bandersnatch and then we can go back to using upstream02:48
ianwjeblair / jhesketh : ^02:48
*** nelsnelson has quit IRC02:50
*** mixos has joined #openstack-infra02:50
*** baoli has quit IRC02:51
*** binbincong has joined #openstack-infra02:52
openstackgerritMerged openstack/requirements: Add smaug projects to projects.txt
openstackgerritMerged openstack/requirements: Update ovs to 2.5.0.
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements
openstackgerritlvdongbing proposed openstack-infra/project-config: New project: python-bileanclient
*** rfolco has joined #openstack-infra02:59
*** yuanying has quit IRC03:00
*** Sukhdev has quit IRC03:02
*** amitgandhinz has quit IRC03:06
*** tonytan4ever has quit IRC03:08
*** markvoelker_ has quit IRC03:13
*** hongbin has quit IRC03:13
openstackgerritMerged openstack/requirements: Block dnspython 1.13.0
*** yuanying has joined #openstack-infra03:24
*** tlian has joined #openstack-infra03:25
*** tlian has quit IRC03:25
*** dimtruck is now known as zz_dimtruck03:29
lifelessmordred: whats this 12-factor thing? did it come up again?03:29
*** links has joined #openstack-infra03:34
*** jpr has quit IRC03:37
*** gildub has quit IRC03:39
*** julim has joined #openstack-infra03:40
*** timl0101 has quit IRC03:41
*** yuanying has quit IRC03:42
*** yuanying has joined #openstack-infra03:44
*** sree has joined #openstack-infra03:49
*** anteaya has quit IRC03:51
*** rfolco has quit IRC03:52
*** yuanying has quit IRC04:01
openstackgerritMerged openstack-infra/irc-meetings: Updating Install Guide meeting details
*** yuanying has joined #openstack-infra04:07
*** julim has quit IRC04:07
*** binbincong has quit IRC04:07
*** gildub has joined #openstack-infra04:11
openstackgerritMerged openstack-infra/irc-meetings: Propose changing Zaqar meeting time
*** Liuqing has joined #openstack-infra04:21
*** binbincong has joined #openstack-infra04:23
*** pcaruana has joined #openstack-infra04:25
*** yuanying has quit IRC04:27
*** yushiro has joined #openstack-infra04:28
*** harlowja_at_home has joined #openstack-infra04:28
*** pcaruana has quit IRC04:32
*** yuanying has joined #openstack-infra04:34
*** nadya has joined #openstack-infra04:35
*** yuanying has quit IRC04:36
*** yuanying has joined #openstack-infra04:37
*** salv-orlando has joined #openstack-infra04:39
*** yuanying has quit IRC04:39
*** psachin has joined #openstack-infra04:39
yushiroHi, good evening.04:40
openstackgerritDavid Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job
*** nadya has quit IRC04:40
*** yuanying has joined #openstack-infra04:41
yushiroI'm testing some patches in our vendor repository.  Then, 'requirements'04:41
yushirotest has failed with error message.04:41
yushiroThe Error message: AttributeError: 'Requirement' object has no attribute 'extras'04:41
*** yuanying has quit IRC04:42
yushiroWould you please tell me how to fix it?04:42
*** yuanying has joined #openstack-infra04:43
*** salv-orlando has quit IRC04:46
*** yuanying has quit IRC04:48
*** yuanying has joined #openstack-infra04:51
*** yuanying has quit IRC04:53
*** cloudtrainme has quit IRC04:54
*** yuanying has joined #openstack-infra04:54
*** Sukhdev has joined #openstack-infra04:56
*** sdake has quit IRC04:57
*** salv-orlando has joined #openstack-infra04:58
*** pilgrimstack has joined #openstack-infra04:59
*** pilgrimstack1 has quit IRC04:59
*** yuanying has quit IRC05:00
ianwyushiro: ahh, that's on stable/liberty branch ...05:00
yushiroianw, Yes. Only stable/liberty branch occurs this error.05:01
ianwyushiro: i'm thinking reno should probably be in global-requirements to match liberty branch05:03
ianwbut there might be a bug, and bknudson was the last person to touch that line :)05:04
yushiroianw, oh... I see it's probably a bug.05:06
*** amitgandhinz has joined #openstack-infra05:07
ianwyushiro: yes, "extras" not defined in liberty branch of openstack-requirements05:07
*** amitgandhinz has quit IRC05:12
*** armax has quit IRC05:13
yushiroianw, I see. Hmm.. I'm not sure 'extras' attribute will be supported for stable/liberty branch. However, currently, it hasn't been supported yet.05:13
yushiroianw, Therefore, I think this commit should be reverted.05:14
ianwyushiro: <- i think this backport will work ...05:15
yushiroianw, ah, OK. Then, I hope this patch will be merged ASAP :)05:18
*** yuanying has joined #openstack-infra05:18
ianwyushiro: added a comment on there, probably best to watch that review.  i'm sure if it's wrong, someone will do the right thing quickly :)05:18
*** woodster_ has quit IRC05:18
ianwas i always say, fastest way to get something fixed is to propose the wrong way to do it :)05:18
yushiroianw, Thanks for your kindness and I understand your advice. I'll put "+1" if jenkins check passed :)05:21
*** javeriak has joined #openstack-infra05:22
*** yuanying has quit IRC05:23
*** yuanying has joined #openstack-infra05:25
*** javeriak_ has joined #openstack-infra05:26
*** Alekhya has joined #openstack-infra05:26
*** javeriak has quit IRC05:27
Alekhyafailed to create instance in jenkins slave node and getting error in debug.log as  ERROR nodepool.NodeLauncher: OpenStackCloudException launching node05:28
*** kushal has quit IRC05:31
*** yuanying has quit IRC05:31
*** harlowja_at_home has quit IRC05:31
*** yuanying has joined #openstack-infra05:32
*** yuanying has quit IRC05:33
*** yuanying has joined #openstack-infra05:35
*** amotoki_ has joined #openstack-infra05:39
Alekhyahello can anyone help me with this issue:05:42
Alekhyafailed to create instance in jenkins slave node and getting error in debug.log as  ERROR nodepool.NodeLauncher: OpenStackCloudException launching node05:42
*** amotoki has quit IRC05:43
*** binbincong has quit IRC05:44
stevemarlooks like a keystone tempest test is cause all sorts of issues05:48
*** kushal has joined #openstack-infra05:51
*** mixos has quit IRC05:54
*** rcernin has joined #openstack-infra05:54
*** yuanying has quit IRC05:55
*** yuanying has joined #openstack-infra05:57
*** binbincong has joined #openstack-infra06:00
*** asettle has joined #openstack-infra06:02
*** yuanying has quit IRC06:03
*** yuanying has joined #openstack-infra06:04
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** andreas_s has joined #openstack-infra06:06
*** asettle has quit IRC06:06
*** yushiro has quit IRC06:07
*** amitgandhinz has joined #openstack-infra06:08
*** pcaruana has joined #openstack-infra06:12
*** amitgandhinz has quit IRC06:13
AJaegerhongbin, sorry, don't have permissions to add you to new groups06:16
*** khappone has quit IRC06:17
*** SergK has quit IRC06:21
*** SergK has joined #openstack-infra06:21
*** oanson has joined #openstack-infra06:22
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** jcoufal has joined #openstack-infra06:26
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** nadya has joined #openstack-infra06:36
*** salv-orlando has quit IRC06:36
*** kzaitsev_mb has joined #openstack-infra06:39
openstackgerritJamie Lennox proposed openstack-infra/project-config: Actually use the service domain in V3 jobs
*** smarcet has quit IRC06:43
*** cloudtrainme has joined #openstack-infra06:43
*** smarcet has joined #openstack-infra06:43
*** jiggalator has joined #openstack-infra06:44
*** reed has quit IRC06:44
*** knikolla has quit IRC06:45
*** netsin has quit IRC06:46
*** Hunner has quit IRC06:46
*** Hunner has joined #openstack-infra06:46
*** Hunner has quit IRC06:46
*** Hunner has joined #openstack-infra06:46
*** reed has joined #openstack-infra06:46
*** Swami_ has quit IRC06:47
*** knikolla has joined #openstack-infra06:48
*** nelsnelson has joined #openstack-infra06:48
*** yamamoto has quit IRC06:49
*** Swami_ has joined #openstack-infra06:50
*** yamamoto has joined #openstack-infra06:50
*** thiagop has quit IRC06:50
*** khappone has joined #openstack-infra06:52
*** nelsnelson has quit IRC06:52
*** thiagop has joined #openstack-infra06:53
*** aeng_ has quit IRC06:53
*** oanson has quit IRC06:54
*** cloudtrainme has quit IRC06:54
*** cloudtrainme has joined #openstack-infra06:55
*** tbarron has quit IRC06:57
*** knikolla has quit IRC06:58
*** crinkle_ has joined #openstack-infra06:59
*** Sukhdev has quit IRC06:59
*** crinkle has quit IRC07:00
*** StevenK has quit IRC07:00
*** crinkle_ is now known as crinkle07:00
*** tbarron has joined #openstack-infra07:00
*** mmedvede has quit IRC07:00
*** knikolla has joined #openstack-infra07:01
*** bodepd_ has quit IRC07:01
*** StevenK has joined #openstack-infra07:01
akijakhi, I'd like to create a new channel: #openstack-pl (general disscussions, support in Polish). I've read and here I am :) Can you help?07:01
*** bodepd_ has joined #openstack-infra07:02
*** lennyb has quit IRC07:03
*** jaosorior has joined #openstack-infra07:03
*** sflanigan has quit IRC07:03
*** sflanigan has joined #openstack-infra07:03
*** sflanigan has joined #openstack-infra07:03
*** tesseract has joined #openstack-infra07:05
AJaegerakijak: better come back once the US wakes up07:06
*** ggnel_t has joined #openstack-infra07:07
akijakAJaeger: ok, no problem. who can help me? do you know someone? ;)07:07
AJaegerakijak: what kind of help do you need?07:08
*** amitgandhinz has joined #openstack-infra07:09
akijakAJaeger: I need someone to create a channel for me07:09
*** tesseract has quit IRC07:09
*** matrohon has joined #openstack-infra07:11
*** jlanoux has joined #openstack-infra07:11
*** rkukura has quit IRC07:12
*** arxcruz has joined #openstack-infra07:12
*** amitgandhinz has quit IRC07:13
*** shardy has joined #openstack-infra07:13
*** dalgaaf has joined #openstack-infra07:15
*** pahuang_ has quit IRC07:15
*** lennyb has joined #openstack-infra07:16
*** cloudtrainme has quit IRC07:17
*** rkukura has joined #openstack-infra07:17
AJaegerakijak: I think you can do this yourself - but if you have doubts, wait until later (16:00 your time) and ask again here, there should be people around to help.07:18
akijakAJaeger: ok, thanks07:18
*** jed56 has joined #openstack-infra07:19
*** esikachev has joined #openstack-infra07:19
*** cloudtrainme has joined #openstack-infra07:19
*** daemontool has joined #openstack-infra07:19
AJaegerteam, project-config reviewers, stevemar: This whole patch series strikes me as odd:
AJaegerIt duplicates functional jobs and adds jobs to check queues that do not test the projects at all.07:20
*** lennyb has quit IRC07:20
openstackgerritDirk Mueller proposed openstack/requirements: Add ovs with Python 3 support
AJaegerI've done some review on them but think this needs some better way - and won't have time to do it myself. If you review any of these changes, try to keep the big picture in mind07:21
*** e0ne has joined #openstack-infra07:23
*** vincentll has joined #openstack-infra07:25
*** e0ne has quit IRC07:26
*** BobH_ has quit IRC07:26
*** mmedvede has joined #openstack-infra07:26
*** mikelk has joined #openstack-infra07:28
*** |-paul-| has joined #openstack-infra07:28
*** zeih has joined #openstack-infra07:29
*** kushal has quit IRC07:30
*** kzaitsev_mb has quit IRC07:30
*** yamahata has joined #openstack-infra07:31
*** kzaitsev_mb has joined #openstack-infra07:31
*** lennyb has joined #openstack-infra07:32
*** salv-orlando has joined #openstack-infra07:37
*** dmk0202 has joined #openstack-infra07:40
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Fix apt-sources configuration for debian-minimal
*** jlanoux has quit IRC07:42
*** salv-orlando has quit IRC07:42
*** hichihara has quit IRC07:44
*** kzaitsev_mb has quit IRC07:44
*** yaume has joined #openstack-infra07:48
*** Swami_ has quit IRC07:49
*** nadya has quit IRC07:52
*** cloudtrainme has quit IRC07:52
*** asettle has joined #openstack-infra07:52
daemontoolrcarrillocruz, yolanda  at your convenience, +A is needed here new repo creation07:53
*** gsilvis has quit IRC07:55
*** gsilvis has joined #openstack-infra07:56
*** dizquierdo_afk is now known as dizquierdo07:59
*** hashar has joined #openstack-infra07:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:01
*** javeriak_ has quit IRC08:05
*** HeOS has joined #openstack-infra08:07
*** snarwade has quit IRC08:07
*** abregman has joined #openstack-infra08:07
*** rossella_s has joined #openstack-infra08:08
*** ifarkas has joined #openstack-infra08:08
*** chem has joined #openstack-infra08:09
*** ociuhandu has quit IRC08:09
*** amitgandhinz has joined #openstack-infra08:09
*** jamielennox is now known as jamielennox|away08:10
*** zhurong has quit IRC08:11
*** zhurong has joined #openstack-infra08:12
*** kushal has joined #openstack-infra08:14
*** amitgandhinz has quit IRC08:15
*** javeriak has joined #openstack-infra08:15
*** dtantsur|afk is now known as dtantsur08:17
openstackgerrityolanda.robla proposed openstack-infra/project-config: Integrate ansible-dev-magnum with real job in shade
*** lucas-dinner is now known as lucasagomes08:17
*** sflanigan has quit IRC08:18
*** mhickey has joined #openstack-infra08:20
*** zhurong_ has joined #openstack-infra08:20
*** Liuqing_ has joined #openstack-infra08:20
*** Jeffrey4l has quit IRC08:22
*** zhurong has quit IRC08:22
*** Liuqing has quit IRC08:22
*** nadya has joined #openstack-infra08:23
*** asettle has quit IRC08:25
*** pt_15 has quit IRC08:26
*** Jeffrey4l has joined #openstack-infra08:26
*** abregman has quit IRC08:26
yolandagood morning08:28
*** dingyichen has quit IRC08:28
*** heyongli has quit IRC08:30
openstackgerritMerged openstack-infra/project-config: Actually use the service domain in V3 jobs
*** heyongli has joined #openstack-infra08:31
*** asettle has joined #openstack-infra08:33
*** oanson has joined #openstack-infra08:33
*** mjblack has quit IRC08:34
openstackgerritMerged openstack-infra/project-config: New project: python-bileanclient
*** mjblack has joined #openstack-infra08:34
*** jistr has joined #openstack-infra08:35
*** kashyap has joined #openstack-infra08:36
openstackgerritMerged openstack-infra/project-config: Migrating liberasurecode and pyeclib projects
*** yamahata has quit IRC08:38
*** e0ne has joined #openstack-infra08:39
openstackgerritMerged openstack-infra/infra-manual: Document "bindep test" usage
openstackgerritMerged openstack-infra/project-config: Explicitly define if ipxe is enabled or not
ajoare we still building somewhre?08:41
ajoI was considering to use the pypi-mirror tool to build one locally08:41
ajobut it doesn't work08:41
openstackgerritMerged openstack-infra/project-config: Run identity v3 only job in keystone
daemontoolyolanda, good morning and thanks08:42
*** dtardivel has joined #openstack-infra08:42
kashyapAJaeger: Heya, got a moment?08:43
openstackgerritMerged openstack-infra/project-config: Gate tempest on - gate-tempest-dsvm-full-devstack-plugin-ceph
openstackgerritMerged openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack
kashyapAJaeger: When you do: Can I turn this into a cherry-pick without messing up?  Or just update the Change-ID similar to that of its corresponding Mitaka/Newton patch? --
*** esikachev has quit IRC08:45
openstackgerritMerged openstack-infra/project-config: Publish debian packages after build
openstackgerritMerged openstack-infra/project-config: Added openstack-server-publish-jobs to ironic-ui
openstackgerritMerged openstack-infra/project-config: Create a tripleo-ci project in JJB
openstackgerritMerged openstack-infra/project-config: api-ref gate for senlin
openstackgerritMerged openstack-infra/project-config: Disaster Recovery new repo for the Freezer Project
openstackgerritMerged openstack-infra/puppet-zuul: enable customize layout file name in zuul.conf
*** electrofelix has joined #openstack-infra08:54
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** salv-orlando has joined #openstack-infra08:56
AJaegerkashyap: what you can do: create a branch off stable/mitaka and then run "git review -X ID" to cherry-pick08:57
AJaegerThat's my preferred way to do backports08:58
*** oanson has quit IRC08:58
kashyapAJaeger: Err, I manually updated the change ID -- :-(08:58
kashyapAJaeger: I should've waited to hear you, before doing that08:58
openstackgerritDougal Matthews proposed openstack/requirements: Add Mistral to global requirements
openstackgerritMerged openstack-infra/project-config: gnocchi: Add grenade jobs
openstackgerritMerged openstack-infra/project-config: [Rally] Adding two new non-voiting jobs for `rally verify`
openstackgerritMerged openstack-infra/system-config: Update AFS docs to reference afs02.dfw
openstackgerritMerged openstack-infra/system-config: Correct new afs fileserver docs
AJaegerkashyap: note that if you change the change id, it becomes a new request, so abandon the old one. The ChangeID is a key.09:00
openstackgerritMerged openstack-infra/project-config: Add sub projects to Kuryr
kashyapAJaeger: Yeah.  I realized that09:01
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: gnocchi: use Ceph devstack plugin
*** pcaruana is now known as pcaruana|afk|09:01
kashyapAJaeger: Can I somehow turn this new change with correct change-ID into a 'cherry-picked' one?09:01
*** salv-orlando has quit IRC09:02
AJaegerJust redo it - and if you send a new patch with the same change-ID, it will override the existing one09:02
silehtAJaeger, Can you +A again this one:, gerrit was complaining about merge conflict, and rebase 'button' fix it09:02
*** esikachev has joined #openstack-infra09:03
kashyapAJaeger: Okay, let me try that.  Your "redo it" -- I take it that "Try a proper cherry-pick, resolve conflicts and use the same change ID>09:03
yolandaAJaeger, can you review ?09:03
*** jtomasek_ has joined #openstack-infra09:04
AJaegerkashyap: yes09:04
AJaegeryolanda: sure09:04
*** openstackgerrit has quit IRC09:04
*** openstackgerrit has joined #openstack-infra09:04
openstackgerritMerged openstack-infra/project-config: Move gate-grenade-dsvm-manila job to check pipeline
openstackgerritMerged openstack-infra/project-config: Add non-voting gate-swift-tox-bandit job
openstackgerritDavid TARDIVEL proposed openstack-infra/project-config: Add new puppet-watcher repository
*** hparekh has quit IRC09:11
*** amitgandhinz has joined #openstack-infra09:11
*** hparekh has joined #openstack-infra09:13
*** sonuk has joined #openstack-infra09:13
*** amitgandhinz has quit IRC09:16
*** sonuk has left #openstack-infra09:16
openstackgerritMerged openstack-infra/project-config: Integrate ansible-dev-magnum with real job in shade
openstackgerritMerged openstack-infra/project-config: Enable translation jobs for manila-ui
AJaegerWow, yolanda. What a review party ;)09:22
yolandayep, lots of things pending09:22
*** jtomasek has quit IRC09:22
yolandai'm checking puppetmaster to see if jenkins is processing correctly09:23
AJaegerthanks, yolanda09:26
openstackgerritMerged openstack-infra/project-config: gnocchi: use Ceph devstack plugin
openstackgerritPranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting.
skraynevyolanda: thank you for the merge my patch. Could you also add me to these groups as a member,members,members09:34
*** jtomasek_ is now known as jtomasek09:35
skraynevthanks yolanda09:35
*** mhickey has quit IRC09:36
yolandamm, jenkins-dev is failing09:36
yolandafatal: []: FAILED! => {"changed": false, "failed": true, "msg": "/usr/bin/timeout -s 9 30m /usr/bin/puppet apply --detailed-exitcodes --logdest syslog --no-noop /opt/system-config/production/manifests/site.pp failed with return code: 6", "rc": 6, "stderr": "Could not retrieve fact='package_provider', resolution='<anonymous>': uninitialized constant Gem\n", "stdout": "", "stdout_lines": []}09:36
yolandagoing to check other jenkins masters as well09:37
AJaegeryolanda: could you add hongbin also to higgins-core and higgins-release, please?  - he asked earlier about it.09:37
AJaegerOOps ;(09:37
openstackgerritPranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting.
yolandaok adding him09:37
*** mhickey has joined #openstack-infra09:40
*** dizquierdo has quit IRC09:40
apuimedoyolanda: thanks for the review!09:43
*** hashar has quit IRC09:44
openstackgerritMerged openstack-infra/irc-meetings: Updating Watcher team meeting time
*** dizquierdo has joined #openstack-infra09:45
*** yaume has quit IRC09:47
*** amotoki has joined #openstack-infra09:52
*** amotoki_ has quit IRC09:54
*** gildub has quit IRC09:56
*** tosky has joined #openstack-infra09:57
*** ociuhandu has joined #openstack-infra09:57
*** kmartin has joined #openstack-infra10:00
*** yamamoto has quit IRC10:05
*** jlanoux has joined #openstack-infra10:05
*** zhurong_ has quit IRC10:06
*** tosky has quit IRC10:06
*** paul-carlton2 has joined #openstack-infra10:06
*** Qiming_ has quit IRC10:06
*** tosky has joined #openstack-infra10:06
*** kmartin has quit IRC10:08
*** vkramskikh_ has joined #openstack-infra10:08
*** esikachev has quit IRC10:08
*** Liuqing_ has quit IRC10:08
*** amitgandhinz has joined #openstack-infra10:12
*** psachin has quit IRC10:12
openstackgerritIvan Kolodyazhny proposed openstack/requirements: Add global requirement update on python-brick-cinderclient-ext project
*** chem has quit IRC10:16
*** amitgandhinz has quit IRC10:17
openstackgerritNate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting
*** psachin has joined #openstack-infra10:18
*** sdague has joined #openstack-infra10:34
*** yamamoto_ has joined #openstack-infra10:34
*** duobei has quit IRC10:40
openstackgerritArtem Silenkov proposed openstack-infra/puppet-jenkins: Fix incorrect directory permissions in jenkinsuser.pp
*** asilenkov_ has joined #openstack-infra10:56
openstackgerritNikita Konovalov proposed openstack-infra/project-config: Enable api-ref jobs in Sahara
asilenkov_hello! could you please review - manifest for jenkins10:57
*** sree has quit IRC11:00
*** kushal has quit IRC11:00
*** dizquierdo is now known as dizquierdo_afk11:00
*** sree has joined #openstack-infra11:01
*** madhu_ak has quit IRC11:01
*** bastelfreak has quit IRC11:05
*** sree has quit IRC11:05
*** amotoki has quit IRC11:06
*** tellesnobrega is now known as tellesnobrega_af11:06
*** claudiub has joined #openstack-infra11:06
*** amotoki has joined #openstack-infra11:10
*** hamzy has quit IRC11:12
openstackgerritPranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting.
*** amitgandhinz has joined #openstack-infra11:13
*** tosky has quit IRC11:14
*** amotoki has quit IRC11:14
*** dansmith has quit IRC11:14
*** ccamacho is now known as ccamacho|lunch11:15
openstackgerritPranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting.
*** dansmith has joined #openstack-infra11:17
*** dansmith is now known as Guest5192611:17
*** amitgandhinz has quit IRC11:17
*** julim has joined #openstack-infra11:22
*** deadnull_ has joined #openstack-infra11:23
*** bastelfreak has joined #openstack-infra11:24
*** Liuqing has joined #openstack-infra11:25
*** jaosorior has quit IRC11:28
*** jaosorior has joined #openstack-infra11:29
*** salv-orlando has joined #openstack-infra11:30
*** gordc has joined #openstack-infra11:31
*** salv-orlando has quit IRC11:35
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job
*** ldnunes has joined #openstack-infra11:38
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** Liuqing has quit IRC11:38
*** vgridnev- has joined #openstack-infra11:39
*** vgridnev- is now known as vgridnev11:39
*** Liuqing has joined #openstack-infra11:39
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** ZZelle has joined #openstack-infra11:42
*** weshay has joined #openstack-infra11:43
*** tpsilva has joined #openstack-infra11:44
*** hashar has joined #openstack-infra11:45
*** baoli has joined #openstack-infra11:46
*** baoli has quit IRC11:46
*** baoli has joined #openstack-infra11:47
*** vincentll has quit IRC11:48
*** vincentll has joined #openstack-infra11:49
*** baoli_ has joined #openstack-infra11:49
*** baoli has quit IRC11:52
*** hamzy has joined #openstack-infra11:54
*** javeriak has quit IRC11:56
*** ttx has quit IRC11:56
*** dprince has joined #openstack-infra11:59
*** salv-orlando has joined #openstack-infra11:59
*** vincentll has quit IRC12:00
*** vincentll has joined #openstack-infra12:00
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow lists of boards and worklists to be paged
*** ttx has joined #openstack-infra12:07
*** psilvad has joined #openstack-infra12:12
*** amitgandhinz has joined #openstack-infra12:14
*** kengo_sakai has quit IRC12:15
*** rodrigods has quit IRC12:15
*** rodrigods has joined #openstack-infra12:15
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Add Higgins project to projects.txt
*** pradk has joined #openstack-infra12:17
*** amitgandhinz has quit IRC12:19
*** ddieterly has joined #openstack-infra12:21
*** javeriak has joined #openstack-infra12:23
*** aysyd has joined #openstack-infra12:27
*** julim has quit IRC12:28
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Improve the worklist filtering functionality
*** raildo has joined #openstack-infra12:28
*** kgiusti has joined #openstack-infra12:28
raildoAJaeger: hey :)12:28
AJaegerhi raildo ! Thanks12:29
*** |-paul-| has quit IRC12:29
raildoAJaeger: I liked your plan, i'll prepare an email to explain this patches and our plan12:30
AJaegerLooking at your changes, I think we should redo them. Let me point out some issues that I see, might be easier to discuss here.12:30
AJaegerArgh, trying again...12:30
AJaegerChange adds a job to magnum that does not even run magnum during the test AFAIK12:30
*** ddieterly is now known as ddieterly[away]12:30
AJaegerAnd that job is added in a few changes - so that's not testing what you expect at all.12:30
AJaegermorning, pabelanger12:31
AJaegerpabelanger: we discuss
AJaegerraildo: So, those are obviously wrong...12:31
AJaegerSecond problem: I think *all* test should only be run on master now and future stable branches (newton,...)12:31
AJaegerSo, use *one* regex - see the novaclient review that covers all these jobs12:32
raildoAJaeger: actually this patch runs the tempest tests in a devstack with keystone v3-only. So I'm supposing that the service have tests on tempest12:32
AJaegerBut it tests keystone - and is not using magnum at all AFAIK12:32
*** pblaho has quit IRC12:33
AJaegerDid you verify that it does?12:33
raildoAJaeger: second point, i totally agree with you, I'll fix this patches today :)12:33
AJaegerIf I'M wrong, please tell me - best to state that in your email and explain why you do it...12:33
*** zz_dimtruck is now known as dimtruck12:33
AJaeger3rd issue: You duplicate all the functional jobs.12:33
AJaegerI think, it might be better to use a *single* functional template and have an if in there.12:34
*** hashar has quit IRC12:34
AJaegerSo, something like - looking at
AJaegername: '{pipeline}-swift-dsvm-functional{special}{job-suffix}'12:35
*** javeriak has quit IRC12:35
*** rhallisey has joined #openstack-infra12:35
AJaegerwith if [ "{special} == '-identity-v3-only' ] ; then export DEVSTACK_LOCAL_CONFIG="ENABLE_IDENTITY_V2=False" fi12:35
AJaegerand then instantiate the template twice, once with special: '', once with special: '-identity-v3-only'12:36
raildoAJaeger: got it, this makes sense... I'll refactor all those patches with your suggestions :)12:36
AJaegerReason: It looks like you duplicate the functional jobs completely with just that single line changed...12:36
AJaegerraildo: Refactor one or two, let'S review them - and then do the rest ;)12:37
raildoAJaeger: I didn't had find a best way to make this happen...12:37
raildoAJaeger: thank you sir, I'll send a email explain my plan and make this changes asap12:37
AJaegerraildo: also, as a FYI: I'm offline starting from saturday for two weeks. I can help today with reviewing and suggestions but afterwards others have to take over - that's why I'm doing the discussio here publically, so that other project-config experts can chime in...12:37
*** rlandy has quit IRC12:38
*** pblaho has joined #openstack-infra12:38
*** ccamacho|lunch is now known as ccamacho12:38
AJaegerraildo: Thanks! Seeing the big picture will help with the reviews...12:38
raildoAJaeger: I appreciate that12:38
*** rlandy has joined #openstack-infra12:38
*** nelsnelson has joined #openstack-infra12:39
*** kushal has joined #openstack-infra12:39
*** matt-borland has joined #openstack-infra12:39
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Improve the board filtering functionality
*** esker has joined #openstack-infra12:42
*** nelsnelson has quit IRC12:43
*** esker has quit IRC12:45
*** amrith is now known as _amrith_12:46
openstackgerritNikita Konovalov proposed openstack-infra/project-config: Enable api-ref jobs in Sahara
*** sree has joined #openstack-infra12:46
*** esker has joined #openstack-infra12:47
*** andymaier has joined #openstack-infra12:47
*** rfolco has joined #openstack-infra12:48
*** andymaier has quit IRC12:49
*** kushal has quit IRC12:49
*** sree has quit IRC12:50
pabelanger#success centos AFS rpm-mirror online and being used in the gate! EG:
openstackstatuspabelanger: Added success to Success page12:52
pabelangernext stop epel12:52
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/system-config: Add openstack-requirements irc channel to bots
pabelangernot a fan of having a 3:1 building / in use ratio for OSIC.  Looks to be related to FIPs again12:54
*** ddieterly[away] is now known as ddieterly12:54
*** binbincong has quit IRC12:56
AJaegerpabelanger: great!12:56
openstackgerritClaudiu Belu proposed openstack/requirements: Bumps os-win version to 0.4.2
AJaegerdims: do you only have a change for system-config?12:56
*** julim has joined #openstack-infra12:57
dimsAJaeger : was staring at project-config now :)12:57
AJaegerdims: Add one for project-config - accessbot and gerritbot first - and make the system-config dependent on it ;)12:57
AJaegerdims: Great!12:57
dimsAJaeger : ack. appreciate it12:57
sdaguedims: wow, all the requirements pushes :)12:58
AJaegerdims: glad to see that the team is forming now...12:58
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job
dimssdague : making use of the new folks :)12:58
AJaegerdims: keep them busy ;)12:59
dimsAJaeger : yay :)12:59
sdagueman, the times when the gate was empty in the morning :)12:59
sdagueand openstack was all PST startups12:59
*** ddieterly has quit IRC12:59
dimssdague good ol' times :)13:00
*** sree has joined #openstack-infra13:00
dimssdague : i loved the apiref sprint emails, hoping seeing their names in the updates motivated folks :)13:01
*** pcaruana|afk| is now known as pcaruana13:01
skraynevAJaeger: hi. I have a short question about project-config and gerrit behavior. Do I correctly understand, that specified upstream repo will be cloned once during merging patch to project-config? the issue, that I late to realize, that my github repo was outdated, so we have 20 more commits above the head. I updated it, but looks like copy on openstack/* is not synced. How it should be fixed correctly ?13:01
*** chem has joined #openstack-infra13:01
sdaguedims: me too :)13:01
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Allow to set BUILD_TIMEOUT from the job
skraynevrepeat all history manually or anybody can sync it manually ?13:01
AJaegerskraynev: correct, "upstream" means "import once"13:01
AJaegerskraynev: 20 commits? Send them as proper reviews through
dimssdague : there's only so many carrots we have :)13:03
* AJaeger agrees with dims: Great work, sdague !13:03
*** nelsnelson has joined #openstack-infra13:04
sdaguedims: I'm actually kind of thinking about how the tooling could be generalized for any sprint. There is conceptually really only 2 things. 1) a way to count what's left and put it into a csv for the burndown chart 2) a gerrit query that returns all the related work13:04
*** mriedem has joined #openstack-infra13:04
skraynevAJaeger: yeah. ok. looks like I create more work for myself ;))) thank you.13:04
AJaegerskraynev: an infra admin might be able to help - but I'm not one of them.13:04
*** coreycb has quit IRC13:04
dimssdague : right13:04
*** edmondsw has joined #openstack-infra13:04
*** coreycb has joined #openstack-infra13:04
AJaegerskraynev: if you have an idea on how to improve so that not others hit the same problem: Changes are welcome!13:04
*** rkukura has quit IRC13:05
*** sree__ has joined #openstack-infra13:05
*** duobei has joined #openstack-infra13:06
skraynevAJaeger: :) I suppose, that it's more tricky and rare situation.13:06
*** andymaier has joined #openstack-infra13:06
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Add new channel #openstack-requirements
*** keedya has joined #openstack-infra13:07
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/system-config: Add openstack-requirements irc channel to bots
*** sree has quit IRC13:08
*** binbincong has joined #openstack-infra13:08
*** psilvad has quit IRC13:08
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Increase tempest baremetal deploy/unprovision timeouts
*** psilvad has joined #openstack-infra13:09
*** yanyanhu has joined #openstack-infra13:09
*** rossella_s has quit IRC13:09
*** rossella_s has joined #openstack-infra13:09
*** itisha has quit IRC13:09
*** rkukura has joined #openstack-infra13:10
*** amitgandhinz has joined #openstack-infra13:11
*** amitgandhinz has quit IRC13:11
*** jsavak has joined #openstack-infra13:11
*** amitgandhinz has joined #openstack-infra13:12
*** akaszuba has joined #openstack-infra13:12
*** akaszuba_ has joined #openstack-infra13:13
*** akaszuba_ has quit IRC13:13
*** links has quit IRC13:14
*** jsavak has quit IRC13:16
*** duobei has quit IRC13:16
*** jsavak has joined #openstack-infra13:16
*** javeriak has joined #openstack-infra13:17
*** cloudtrainme has joined #openstack-infra13:17
*** xyang1 has joined #openstack-infra13:18
*** chem has quit IRC13:19
*** psilvad has quit IRC13:20
*** chem has joined #openstack-infra13:21
*** jamesmca_ has joined #openstack-infra13:21
*** amotoki has joined #openstack-infra13:21
*** esikachev has joined #openstack-infra13:25
*** Guest51926 is now known as dansmith13:26
*** denisra has quit IRC13:26
*** regXboi has joined #openstack-infra13:27
*** Goneri has joined #openstack-infra13:27
*** sigmavirus24_awa is now known as sigmavirus2413:28
*** _ari_ has joined #openstack-infra13:29
*** nadya has quit IRC13:29
*** psachin has quit IRC13:30
*** psilvad has joined #openstack-infra13:30
*** zeih has quit IRC13:31
*** psachin has joined #openstack-infra13:31
*** zeih has joined #openstack-infra13:31
*** yanyanhu has quit IRC13:33
*** yamamoto_ has quit IRC13:33
*** psilvad has quit IRC13:34
*** dimtruck is now known as zz_dimtruck13:34
*** zz_dimtruck is now known as dimtruck13:34
*** ramishra has quit IRC13:35
*** psilvad has joined #openstack-infra13:35
*** ddieterly has joined #openstack-infra13:36
*** psilvad has quit IRC13:37
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: [WIP] Trying a new target just for u-c sanity check
*** psilvad has joined #openstack-infra13:37
*** ramishra has joined #openstack-infra13:38
*** jamesmca_ has quit IRC13:39
*** psilvad has quit IRC13:39
*** psilvad has joined #openstack-infra13:39
*** kien-ha has joined #openstack-infra13:40
*** kien-ha_ has joined #openstack-infra13:41
*** kien-ha_ has quit IRC13:41
*** kien-ha has quit IRC13:41
*** kien-ha has joined #openstack-infra13:41
*** dtantsur is now known as dtantsur|bbl13:42
*** dizquierdo_afk is now known as dizquierdo13:42
openstackgerritNate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: [WIP] Trying a new target just for u-c sanity check
*** jamesmca_ has joined #openstack-infra13:42
*** Alekhya has quit IRC13:42
*** cloudtrainme has quit IRC13:43
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
*** dimtruck is now known as zz_dimtruck13:44
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
fungiheads up, i've got a bunch of guests coming to stay tonight through the weekend so i'll only be around intermittently over the next couple days. disappearing now for an hour or two so i can run some errands but should be back again by 16:00 utc13:44
*** wanghua has quit IRC13:44
openstackgerritThomas Bechtold proposed openstack-infra/bindep: Add support for openSUSE and SLES
AJaegertoabctl: yeah! ^13:46
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: New tox target just for u-c sanity check
toabctlAJaeger: but still a question - I need to install diffutils for rpm-packaging but looks like it does not work. see
toabctlAJaeger: any idea why?13:46
*** ddieterly is now known as ddieterly[away]13:47
*** eharney has joined #openstack-infra13:47
*** javeriak has quit IRC13:47
*** jamesmca_ has quit IRC13:48
*** tosky has joined #openstack-infra13:49
AJaegertoabctl: strange ;(13:49
pabelangerfungi: ack13:49
pabelangertoabctl: AJaeger: looks like other-requirements.txt is empty?13:49
*** jamesmca_ has joined #openstack-infra13:50
AJaegertoabctl: see my comment on 315575, a line like "diffutils [pkg:suse linux test]" might not work13:50
AJaegertoabctl: if you add other-requirements, you replace the bindep-fallback.txt file that is normally installed. Perhaps you need some more packages13:51
toabctlAJaeger: ok. but that's independent from the jenkins failure.13:51
toabctlpabelanger: it isn't13:51
AJaegertoabctl: project-config/jenkins/data/bindep-fallback.txt is the complete list, add that to other-requirements to exclude that you're missing files13:52
pabelangertoabctl: is diffutils valid on ubuntu?13:52
toabctlpabelanger: yes. see
AJaegertoabctl: it fails for me locally with the same problem as in the gate ;)13:53
toabctlAJaeger: oh. really? it passes for me13:54
*** jamesmca_ has quit IRC13:54
toabctlAJaeger: what fails? tox -elint ?13:54
AJaegertox -e lint fails13:55
*** ddieterly[away] is now known as ddieterly13:55
AJaegeropen("/usr/share/spec-cleaner/excludes-bracketing.txt", O_RDONLY) = -1 ENOENT (No such file or directory)13:55
AJaegertoabctl: remove spec-cleaner from your install and try again...13:55
pabelangertoabctl: AJaeger: Oh, diffutils likely installed already, that's why bindep is not attempting to install it13:55
AJaegerLooks like spec-cleaner in the venv is not working properly13:56
toabctloh. so spec-cleaner is missing. not diff13:56
*** psilvad has quit IRC13:56
toabctlok. thanks a lot!13:56
AJaegertoabctl: spec-cleaner is isntalled - but spec-cleaner looks for excludes-bracketing.txt in wrong location13:56
*** andymaier has quit IRC13:56
AJaegertoabctl: packaging error in spec-cleaner - it does not work in venv!13:56
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: New CI job for requirements repo
*** anteaya has joined #openstack-infra13:58
pabelangerI am seeing: OpenStackCloudTimeout: Timeout waiting for the floating IP to be attached. in osic-cloud1 a fair bit14:03
*** mhickey has quit IRC14:03
*** jpr has joined #openstack-infra14:04
*** sdake has joined #openstack-infra14:05
*** esker has quit IRC14:06
*** esker has joined #openstack-infra14:06
pabelangerCleaned 83 FIPs from openstackjenkins-osic-cloud1 :(14:07
pabelangermordred: ^14:07
*** zz_dimtruck is now known as dimtruck14:08
AJaegerOOps ;(14:08
pabelangerYa, we appear to be timing out attaching to them14:08
*** _amrith_ is now known as amrith14:09
*** jpr has quit IRC14:09
*** roxanaghe has joined #openstack-infra14:10
*** baumann has joined #openstack-infra14:11
*** doug-fish has joined #openstack-infra14:11
cloudnullpabelanger: regarding the OSIC cloud w/ floats. I can reproduce that issue.14:12
*** sree has joined #openstack-infra14:12
cloudnullI've been testing with building 50+ nodes using heat and all of them are coming online without issue.14:12
*** roxanaghe has quit IRC14:13
cloudnullIf you have instance IDs and such I can go log diving again to see if I can find the problem14:14
*** d0ugal has quit IRC14:14
*** maishsk has joined #openstack-infra14:14
*** roxanaghe has joined #openstack-infra14:14
cloudnull -- super simple heat template to build N instances and give it a float14:14
pabelangercloudnull: sure, I can get an instance id14:15
morganpabelanger: that is no good :(14:15
*** sree__ has quit IRC14:15
*** d0ugal has joined #openstack-infra14:16
pabelangercloudnull: bebeeb5e-9125-417d-92e3-4b88e89e8dd4 is the last instances that timed out14:17
*** psilvad has joined #openstack-infra14:17
*** zeih has quit IRC14:17
*** mhickey has joined #openstack-infra14:17
* cloudnull looking now14:18
d0ugalAnyone got a moment to checkout this requirement addition? It just adds an openstack project.
*** rbrndt has joined #openstack-infra14:18
pabelangermorgan: cloudnull this is the error from shade: 2016-05-12 14:09:53,411 ERROR shade: Timeout waiting for floating IP to become active. Floating IP was created for server bebeeb5e-9125-417d-92e3-4b88e89e8dd4 but is being deleted due to activation failure.14:19
*** roxanaghe has quit IRC14:19
pabelangergoing to look at code path now14:19
*** yaume has joined #openstack-infra14:20
AJaegerd0ugal: best ask on #openstack-requirements14:20
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
*** AJaeger has quit IRC14:21
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
*** nadya has joined #openstack-infra14:21
*** deadnull_ is now known as _deadnull14:22
morganpabelanger: ok, let me know if you need extra eyes. at least we know the FIP cleaner works well.14:22
*** d0ugal has quit IRC14:23
*** cbader_ has joined #openstack-infra14:23
*** flaper87 has quit IRC14:24
*** nadya has quit IRC14:24
*** cbader has quit IRC14:24
*** Shrews has quit IRC14:24
openstackgerritJulien Danjou proposed openstack-infra/project-config: Remove check-requirements from Ceilometer and Aodh
dstufftfungi: do any of the other places support gpg? I can't think of one but I dunno!14:24
akijakhi, I'd like to create a new channel: #openstack-pl (general disscussions, support in Polish). I've read and here I am :) Who can help?14:24
*** Shrews has joined #openstack-infra14:24
*** ggillies has quit IRC14:25
*** ggillies has joined #openstack-infra14:26
clarkbakijak: I am sure several of the can help. Do you have specific questions?14:26
*** baumann1 has joined #openstack-infra14:26
*** flaper87 has joined #openstack-infra14:27
*** flaper87 has quit IRC14:27
*** flaper87 has joined #openstack-infra14:27
*** xwizard has joined #openstack-infra14:27
pabelangermorgan: sure, I am trying to trace the path to see why we are leaking the FIP now14:28
clarkbakijak: has details on registering a channel and adding our bots14:28
*** sree has quit IRC14:29
morganpabelanger: i am on a plane so limited access, will be on the ground in a couple hrs. def can jump in then, if you need eyes sooner i'll have a copy of shade handy but it'll take me a bit to be in sync.14:29
morganpabelanger: but i'll keep eyes here in case i can help14:29
*** baumann has quit IRC14:29
*** _deadnull is now known as deadnull_14:31
*** zeih has joined #openstack-infra14:32
anteayaakijak: have you read
anteayaakijak: it tells you how to register a channel yourself14:32
*** jamesmca_ has joined #openstack-infra14:32
*** yamamoto_ has joined #openstack-infra14:33
akijakclarkb: anteaya thx I'll check that. I've read which says to come here and ask :)14:34
anteayaakijak: thank you for pointing that out14:35
*** sdake has quit IRC14:35
*** pt_15 has joined #openstack-infra14:36
cloudnullpabelanger: do you have a reference to how shade running the instance create + FIP assosiate + cleanup? maybe i can try and recreate things from that persepective because I'm not able to using the API or via heat template.14:36
*** links has joined #openstack-infra14:37
*** jamesmca_ has quit IRC14:37
*** hashar has joined #openstack-infra14:38
*** esker has quit IRC14:38
akijakanteaya: so the channel (#openstack-pl) already exists but it's not mentioned on how to add it to the wiki?14:39
openstackgerritMorgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project
anteayaakijak: if you go to the wiki in the top right hand corner thaere is a link to log in14:39
nibalizergo squ14:39
anteayaakijak: what happens when you tryt to log in?14:40
nibalizerhaha ohai14:40
*** aratus has joined #openstack-infra14:40
anteayaakijak: also the channel is not registered14:40
anteayaakijak: I suggest you ask the other inhabitant to leave the channel so that you may register it14:41
anteayaor ask them to register the name14:41
anteayaakijak: it might be possible the person created the channel when they saw your earlier question about the channel14:42
akijakanteaya: I get 500 after login14:42
anteayaakijak: I highly recommend registering the channel prior to adding it to the wiki14:42
akijakanteaya: ok, thanks. i'll try to contact him.14:42
anteayaakijak: okay that means you haven't edited our wiki before and you can't edit it now, as we have disabled new account creation due to wiki spam14:43
*** yamamoto_ has quit IRC14:43
anteayaakijak: you work on getting the channel registered14:43
anteayaakijak: if you can get it registered I can add it to the wikipage14:43
akijakanteaya: ok, understood. thank you14:44
anteayaakijak: thank you14:44
anteayaakijak: if you can't contact them or get them to leave come back after about 24 hours of trying and we will see what we can do14:44
akijakanteaya: ok14:44
anteayathank you14:44
clarkbcloudnull: I believe that shade creates the fip and provides the server port to attach to at the same time. This in theory makes create and attach an atomic operation from api perspective14:45
*** psilvad has quit IRC14:45
pabelangercloudnull: It looks like we are calling create_server from shade, then wait_for_server. Obviously will all cloud bits for OSIC. Wonder if we could write a quick script to reproduce it14:45
pabelangerclarkb: ah, good to know14:46
cloudnull++ im looking at that now too. if you have something available already I can pick at i can go from there.14:46
pabelangerseems like we can always get about 11/12 FIPs to attach at once given time. The rest, fail.
*** jamesmca_ has joined #openstack-infra14:48
anteayafungi: happy company14:50
*** tonytan4ever has joined #openstack-infra14:51
morganclarkb: that is the idea, shade is intended to create/attach fip in  one call afair14:52
mordredI have a script - one sec14:52
morganpabelanger: ick14:52
*** jlanoux has quit IRC14:53
mordredcloudnull, clarkb, morgan:
mordredthat's how we create servers14:54
* morgan nods.14:54
pabelangermordred: nice, thanks14:54
cloudnullIm monitoring the logs in vivo right now and doing the a similar op w/ heat using a node count of 25 and its complementing the op in <2 min
cloudnullmordred:  testing that now14:55
mordredcloudnull: lemme go look at heat source and see how they're doing server_floating_ip14:56
mordredlooks to be about the same14:58
*** zeih has quit IRC15:00
mwhahahaany chance on getting the ci box back on gerrit today? i noticed from the logs yesterday15:01
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** esker has joined #openstack-infra15:01
*** mhickey has quit IRC15:01
mordredmwhahaha: I have now run that command15:02
*** nadya has joined #openstack-infra15:02
*** thorst_ has joined #openstack-infra15:02
*** esker has quit IRC15:02
openstackgerritPaul Belanger proposed openstack-infra/system-config: Add EPEL mirror
*** Liuqing has quit IRC15:03
*** esker has joined #openstack-infra15:04
*** ddieterly is now known as ddieterly[away]15:05
*** ddieterly[away] is now known as ddieterly15:05
*** ddieterly is now known as ddieterly[away]15:05
*** ddieterly[away] is now known as ddieterly15:06
openstackgerritThomas Bechtold proposed openstack-infra/bindep: Add support for openSUSE and SLES
mordredpabelanger: I'm going to start calling you mr. mirror creator man person15:06
pabelangermordred: love me some AFS15:07
*** mixos has joined #openstack-infra15:09
anteayamordred: that's a lot to type15:11
anteayamr. silver lining15:11
mordredanteaya: tab complete also doesn't work very well for it15:14
*** r-mibu has quit IRC15:14
anteayamordred: true15:14
*** r-mibu has joined #openstack-infra15:14
*** mixos has quit IRC15:14
*** mhickey has joined #openstack-infra15:15
*** yamahata has joined #openstack-infra15:16
*** kmartin has joined #openstack-infra15:17
*** sdake has joined #openstack-infra15:17
*** d0ugal has joined #openstack-infra15:17
*** aratus has quit IRC15:20
*** aratus has joined #openstack-infra15:20
openstackgerritSean Dague proposed openstack-infra/project-config: Revert "Actually use the service domain in V3 jobs"
sdaguecan I get a quick review of ^^^15:22
*** vhosakot has joined #openstack-infra15:22
*** esikachev has quit IRC15:22
sdaguethere is a devstack fix out there, but everything is so backed up it would be better to just revert, let the fix land, then reapply15:22
*** harlowja_at_home has joined #openstack-infra15:22
openstackgerritMonty Taylor proposed openstack-infra/project-config: Clean floating IPs on OSIC and BlueBox
mordredsdague: looking15:23
mordredsdague: lgtm15:23
sdaguemordred: thanks15:24
*** jlanoux has joined #openstack-infra15:24
*** |-paul-| has joined #openstack-infra15:24
mordredpabelanger: when we restart nodepool today to pick up the fip cleaning patch, we should also land
mordredpabelanger: so that we use it :)15:24
sdaguethere are some devstack neutron fixes in gate (outside the sliding window) that would be good to let land. Hopefully the project config gets applied before they get out of the window15:25
*** aratus has left #openstack-infra15:25
openstackgerritMerged openstack-infra/nodepool: Cleanup leaked floating ips periodically
clarkbmordred: I would really like to have owrking fips in bluebox and osic before we do that15:25
clarkbmordred: reduce the total number of variables at play15:25
mordredclarkb: k15:27
nibalizerso many leaked ips in osic15:27
nibalizerand a ton of instances without ip addresses15:27
mordredclarkb: is there something else we think is breaking now other than the delete code not deleting them?15:27
* mordred feels he may be missing something15:27
clarkbmordred: yes, the attach isn't attaching which leads to the need for a delete which then fails15:27
*** asilenkov_ has quit IRC15:27
clarkbmordred: ideally we would figure out hwy it takes an hour for that fip attach to eventually time out15:28
nibalizerclarkb: should i go ahead and clean up the leaks?15:28
mordredoh! I was not aware it was taking an hour15:28
clarkbmordred: if you look at the server age they basically hang around for an hour then go away15:28
mordredclarkb: is that taking an hour on our side?15:28
clarkbat least that is what it looked like on the small sample I grepped logs on15:28
pabelangermordred: sure15:29
mordredclarkb: sorry for being dense - but could you say that again with less pronouns?15:29
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Remove voting gate v3 only job
pabelangerjust read clarkb response15:29
mtreinishanteaya, yolanda, fungi: ^^^ if you get a sec15:30
mordredpabelanger, clarkb: also, I think we can still restart nodepool to pick up the patch and just leave it unenabled in the project-config15:30
clarkbmordred: nodepool boots an instance in osic, creates/attaches fip to it, this create/attach times out after an hour, this timeout triggers the delete fip code which fails, at this point we have leaked the fip15:30
mordredclarkb: awesome. thank you15:30
*** ningy__ has joined #openstack-infra15:31
clarkbmordred: my suggestion is that we don't worry about the auto leaked fip cleanup yet because it won't address the root issue which is fip create/attach time outs15:31
*** nadya has quit IRC15:31
clarkbmordred: that would also be fine15:32
clarkbmordred: where that == leave it disabled for now15:32
pabelangerya, it takes exactly 60mins for the FIP timeout to happen. What are the downsides to exposing that to nodepool and lowering it?15:32
*** links has quit IRC15:32
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Revert "Changing gate on devstack identity v3 only voting"
clarkbpabelanger: I don't think lowering it is the right behavior either, we need to figure out why a floating ip attach owuld take that long and not succeed15:32
mordredclarkb: well, we're passing in a timeout of 360015:32
mordredto wait for server15:33
clarkbnodepool passes that in? I thought defeault boot timeout was something around 300 seconds15:33
mordred    def waitForServer(self, server, timeout=3600):15:33
mordredit's possible we should change that15:33
clarkbin any case the actual timeout value isn't super important, the why is there such a high rate of timeouts is what we need to determine15:33
pabelangerclarkb: agreed, was just suggesting maybe a more conservative moving forward15:33
mordredyah. agree15:33
anteayamtreinish: I want to see how a gate only job got added as well as how a job got added to devstack and tempest without qa knowing about it15:34
*** armax has joined #openstack-infra15:34
mordredclarkb: so - I don't think there is an actual mystery15:34
mordredthe floating ip attach is just not working15:35
mordredit's broken15:35
mordredbut we've told nodepool to wait for an hour15:35
openstackgerritYaroslav Lobankov proposed openstack-infra/project-config: [Rally] Fixing sahara installation for `rally verify` jobs
mordredto see if it'll work15:35
openstackgerritMatthew Treinish proposed openstack-infra/project-config: Revert "Changing gate on devstack identity v3 only voting"
clarkbmordred: but it only just recently started to break15:35
mordredso I'm pretty sure the underlying problem is "sometimes floating ips don't attach"15:35
openstackgerritMerged openstack-infra/jenkins-job-builder: Add deadline option for build-timeout plugin
clarkbmordred: like this worked fine for a couple months15:35
mtreinishanteaya: I repushed it as a revert:
mtreinishso the history is there15:35
anteayamtreinish: thanks that helps15:35
*** dmk0202 has quit IRC15:35
mtreinishanteaya: it looks like sdague -1'd it and then it was approved on the next rev (without sdague's vote)15:36
openstackgerritYaroslav Lobankov proposed openstack-infra/project-config: [Rally] Fixing sahara installation for `rally verify` jobs
*** dmk0202 has joined #openstack-infra15:36
mordredclarkb: we only recently added the code to do a wait loop looking for the active state on the floating ips15:36
*** e0ne has quit IRC15:36
mordredso the hour-wait is a result of that change15:37
clarkbmordred: so previously we would fail on the ssh timeout?15:37
anteayamtreinish: yeah I see that15:37
clarkbwhich is shorter?15:37
mordredclarkb: yah15:37
Shrewsmordred: i think we weren't passing the timeout down to the attach before, iirc15:37
mordredor we'd fail on the server not having a FIP in its addresses record actually15:37
*** arxcruz has quit IRC15:37
*** mikelk has quit IRC15:37
mordredShrews: yah. that's right15:37
anteayayolanda jhesketh can you share your thoughts on why 315638 was +2'd and +A'd without further consultation with sdague who was -1 on the patch?15:37
clarkbmordred: ok, I think the correct value to pass into wait for server would be the boot timeout in the config or some derivative of that number15:37
mordredclarkb: I agree15:38
mordredclarkb: oh - we do pass in self.launch_timeout15:38
anteayamtreinish: I'm +2 on your revert15:39
* mordred needs to go board plane - will look again once he's on board15:39
anteayamordred: happy flying15:39
*** psachin has quit IRC15:39
mtreinishanteaya: thanks15:40
anteayamtreinish: thanks for posting it as a revert15:40
openstackgerritLucas Alvares Gomes proposed openstack-infra/project-config: Add virtualbmc from source non-voting test
pabelangerclarkb: a follow up patch to nodepool-builder from yesterday: Making sure our worker args are integers15:40
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update Plot plugin
*** deadnull_ is now known as _deadnull15:41
pabelangerclarkb: since puppet is still disabled, I manually applied the patch and restarted nodepool-builder last night15:41
clarkbok will review15:41
sdakepabelanger there may be a problem with the centos bindep images - possibly related to selinux15:41
*** mtanino has joined #openstack-infra15:41
sdakepabelanger we are capturing the audit log now, and i'll let you know15:41
sdakeif there is something wrong with them, could you fix it?15:41
sdakeif we identify somethign labeled wrong?15:41
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Update Plot plugin
sdakerabbitmq consistently crashess in the gate15:42
clarkbpabelanger: todays uploads will run with 12 threads then?15:42
pabelangersdake: send a log once you have it. We landed a patch to DIB to fix selinux security context a while back15:43
pabelangerclarkb: yup, it should15:43
pabelangerclarkb: I tested it last night with centos-715:43
*** madhu_ak has joined #openstack-infra15:43
clarkbpabelanger: sdake and rabbitmq is installed by the jobs which should confoigure selinux at job run time not at image build15:43
sdakeclarkb we install rabbitmq in a container15:44
sdakeclarkb i dont think this affects other gate jobs outside of kolla15:44
ningy__dhellmann: could you take a look at this patch
*** hashar has quit IRC15:44
ningy__or, anyone could take a look at it's blocking rally jenkins ci job15:45
pabelangerclarkb: I would think that too15:45
*** andreas_s has quit IRC15:46
ningy__it blocks rally monasca gate job15:46
ningy__I can't get rally monasca gate job pass without this one merged15:46
pabelangerclarkb: uploads not as fast as last night, since we are iterating over providers.  Currently uploading to osic-cloud1, ovh-bhs1 and ovh-gra115:46
clarkbningy__: there is a push to treat the requirements as its own project team and they have a new irc channel now #openstack-requirements15:47
clarkbningy__: probably best to start there as you should get attention from the people workong on that stuff15:47
ningy__ok. thanks clarkb15:47
*** tphummel has joined #openstack-infra15:47
*** matrohon has quit IRC15:49
*** yaume has quit IRC15:50
*** vincentll has quit IRC15:51
*** harlowja_at_home has quit IRC15:52
*** timl0101 has joined #openstack-infra15:53
*** claudiub has quit IRC15:53
*** Swami has joined #openstack-infra15:53
*** madhu_ak has quit IRC15:54
*** |-paul-| has quit IRC15:56
*** kmartin has quit IRC15:57
*** MarkAtwood has joined #openstack-infra15:57
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-datera-cinder
*** pfallenop has quit IRC15:58
*** ffrank has joined #openstack-infra15:59
*** jcoufal has quit IRC16:00
*** ddieterly is now known as ddieterly[away]16:00
fungidstufft: yeah, i doubt maven, npm, goget, et cetera pay any attention to gpg either. the only place i know we would end up being able to leverage it is for deb repos we publish (and then it would be signatures on the indices not the packages since that's what secure apt pays attention to anyway)16:01
*** doug-fis_ has joined #openstack-infra16:01
jeblairdstufft: ianw updated my bandersnatch PR as requested by christian here:
*** ddieterly[away] is now known as ddieterly16:03
jeblair(i abandoned (declined?) my own pr in favor of that)16:03
*** jtomasek has quit IRC16:03
*** doug-fi__ has joined #openstack-infra16:03
*** rossella_s has quit IRC16:04
*** pfallenop has joined #openstack-infra16:04
*** doug-fish has quit IRC16:05
*** AJaeger has joined #openstack-infra16:06
*** doug-fis_ has quit IRC16:06
*** jaosorior has quit IRC16:06
*** doug-fi__ has quit IRC16:08
*** ri0 has joined #openstack-infra16:13
*** sdake has quit IRC16:13
* fungi suddenly remenbers that there is no rebase in hg16:14
*** mkoderer__ has quit IRC16:14
openstackgerritDavid Lyle proposed openstack/requirements: Remove XStatic-QUnit
*** dmk0202 has quit IRC16:15
anteayafungi: :/16:15
openstackgerritMerged openstack-infra/project-config: Revert "Changing gate on devstack identity v3 only voting"
anteayathanks AJaeger16:16
anteayamtreinish: ^^16:16
*** kien-ha has quit IRC16:17
fungioh, heh, now i see 315636 was abandoned16:17
clarkbfungi: there is sort of but I never get it to work right (hg rebase is a plugin)16:17
*** apoorvad has joined #openstack-infra16:19
*** akaszuba has quit IRC16:19
fungiweird that got approved when the only qa reviewer to provide feedback left a -1 on it16:19
pabelangerclarkb: jeblair: we appear to be pushing 400Mbps uploads from nodepool.o.o ATM. Do you know what the limit of our pipe is?16:20
fungiespecially as it was reverted for entirely the same reason sdague pointed out in his review16:20
fungiraildo: ^ what was the longer term plan there?16:20
*** d0ugal has quit IRC16:20
clarkbpabelanger: I want to say its 800mbps total, but would have to check the rax flavor details16:21
*** d0ugal has joined #openstack-infra16:21
raildofungi: we had this patch nv for a while on tempest and devstack ( almost one year) and looking the job history, we thought it could start be voting, after that, I start to add this same job in every service16:22
*** andymaier has joined #openstack-infra16:22
pabelangerclarkb: that gives me something to work with16:23
raildofungi: for now, I'm adding it as non-voting16:23
cloudnullclarkb mordred pabelanger: so i can recreate the FIP timeout issue using shade but not w/ other methods and it seems it only happens when I multi-thread the jobs.16:23
morganAJaeger: i expect to have the last bits needed for ldappool import ready today. :)16:23
morganAJaeger: thanks for the quick review yesterday16:23
fungiraildo: yeah, it needs to be running on other services changes too, and passing, then switched to voting on all of them at the same time if that's the desire16:23
cloudnulldoing it in serial doesn't seem to cause the issue16:23
fungiraildo: but having it voting only on qa repo changes is just going to create deadlocks16:24
zigofungi: The build now always fails, and I have no explanation:
zigoWhat am I missing?16:24
raildofungi: got it... sorry about that, I'll take a note on it16:24
raildofungi: thanks for the help16:24
clarkbcloudnull: huh, we serialize the api calls to handle rate limiting sanely but we do run a different shade cloud object for each cloud region in different threads16:24
cloudnullive been trying this
zigoIt's looking like it never can fetch packages...16:24
cloudnullwhen i do it in a serial loop it seems to work fine16:25
fungiraildo: also, our project-config reviewers shouldn't have approved the change without positive feedback from qa representatives (qa ptl or qa-infra liaison)16:25
cloudnullthen again im only testing w/ 25 nodes16:25
*** xiangxinyong has quit IRC16:25
morganAJaeger: ok ldappool should now pass pep8 and py27 tests and we should good for importing.16:25
fungidid we recently update the hideci.js again to try and solve the requirements failure message handling? looks like the results table is broken again16:26
cloudnullI also noticed that in some cases I've ended up with duplicate instances.16:26
clarkbfungi: yes mordred indicated he touched the file earlier today as per your directions yesterday16:26
mordredfungi: I did the touch earlier16:26
*** x00350071 has joined #openstack-infra16:26
cloudnullwell instance names that is .16:26
fungiclarkb: mordred: oh, so the change we merged yesterday hadn't taken effect? anyone have a revert going yet?16:26
cloudnulllike in my last round i had two nodes w/ the name "ADMIN_TEST_NODE-bea07735-63f2-4b26-9052-d2682733f9c6"16:26
*** tellesnobrega_af is now known as tellesnobrega16:27
raildofungi: right now, AJaeger is help me with those new patches, do you recommend someone else that I can ask for reviews?16:27
*** d0ugal has quit IRC16:27
mordredcloudnull: lovely16:27
* zigo tried to add an apt-get update16:27
clarkbfungi: no revert that I have seen16:27
mordredcloudnull: that's against a devstack?16:27
fungiclarkb: mordred: in the past we saw hideci.js cease getting bundled entirely ifg it updated without updating the timestamp on the site header, so this (continuing working but not updating) is new behavior in 2.11 i guess16:27
*** baumann1 has quit IRC16:27
cloudnulland idk if thats my cloud problem or if shade is retrying an op and creating a new node16:27
*** nadya has joined #openstack-infra16:27
cloudnullmordred: no thats aginst the osic16:28
mriedemis it just me or are all CI comments toggled by default now?16:28
fungiraildo: sdague reviewed (and left a -1 on) the original patch which was subsequently reverted by mtreinish, so either of them would be good candidates16:28
*** mkoderer__ has joined #openstack-infra16:28
mordredcloudnull: kk. admin/SuperSecrete threw me :)16:28
fungimriedem: it's broken, we're in the process of reverting16:28
mriedemfungi: sweet16:28
anteayafungi raildo also voting jobs are in both check and gate, not just on gate16:28
*** bpokorny has joined #openstack-infra16:28
mordredShrews: ^^ cloudnull has a script that reproduces the problem16:28
openstackgerritMerged openstack-infra/project-config: Revert "Actually use the service domain in V3 jobs"
raildoanteaya: yeap, I learned this in the wrong way :(16:29
anteayafungi raildo the patch had voting jobs only on the gate16:29
anteayaraildo fungi well as long as you learned it16:29
anteayaraildo fungi the hard way can also be effective16:29
cloudnullthis is the stack i see
raildoanteaya: ++16:29
pabelangerzigo: Ya, refreshing apt caches is a good start. never know how older they'll be when we launch a new node16:29
morgananteaya: if you have a moment your eyes (to make sure I didn't do something silly) would be welcome16:30
mordredcloudnull: yah. that's the error we're getting too16:30
zigopabelanger: Thanks for your review btw! :)16:30
anteayaraildo: great thank you16:30
anteayamorgan: I will look16:31
morgananteaya: much appreciated16:31
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Revert "Allow 3rd party CI's to have same pipeline names"
*** asettle has quit IRC16:31
zigomordred: I'm trying to get the publish job to run with a new patch (since the publish macro got merged), then we'll have to work on the pick-up job. Will you be able to contribute it?16:31
raildoanteaya: fungi thank you guys :)16:31
fungiclarkb: mordred: sdague: ^16:31
mordredzigo: yup!16:31
anteayaraildo: you are welcome and I'm not a guy16:31
raildoanteaya: ops... sorry16:32
anteayamorgan: is this the lib that has the question up to legal discuss16:32
*** bpokorny has quit IRC16:32
anteayaraildo: if you care, I like folks people team group infra and occasionally cats16:32
anteayaif you use cats do so sparingly16:32
morgananteaya: the only legal concern was that someone was moving it in-tree of keystone and causing licensing issues. we're adopting it so no conflict now16:32
* mordred hands anteaya a cat16:32
*** bpokorny has joined #openstack-infra16:32
clarkbmordred: cloudnull setting auto_ip to False results in floating ip creation with shade?16:33
morgananteaya: the previous maintainer is transferring to me/openstack-keystone team to forward, as it is a dead project otherwise16:33
anteayamordred: thank you, my cat is hunting again at night, warning to all mice in the neighbourhood16:33
morgananteaya: we will need to keep MPL license though (meh, could be worse)16:33
raildoanteaya: I'll start to use cats :)16:33
mordredanteaya: mine has been bringing us half-eaten lizards16:33
anteayamorgan: ah wonderful16:33
clarkboh setting the pool is like setting auto ip16:33
clarkbbut more explicit16:33
anteayaraildo: ha ha ha16:33
pabelangerclarkb: is the reason we upload raw images to ovh because of a performance issue with qcow2 in the cloud?16:33
clarkbpabelanger: I think it is because they don't accept qcow216:34
anteayamordred: nice16:34
morgananteaya: yeah he was super happy his code was helping someone.16:34
anteayamordred: can you keep the lizard skin?16:34
*** jlanoux has quit IRC16:34
* anteaya has several tails from red squirrels in the freezer16:34
anteayamorgan: oh awesome16:34
pabelangerclarkb: okay, that make sense16:35
clarkbpabelanger: it is probably worth double checking that assumption though16:35
*** slaweq has joined #openstack-infra16:35
*** sridhar_ram has joined #openstack-infra16:36
sdaguefungi: gah, ok.16:37
pabelangerclarkb: sure, I can ask16:37
clarkbpabelanger: or just try an upload of qcow2 and boot it :)16:38
*** slaweq has quit IRC16:38
mordredcloudnull: looking at your code, since you're using multiprocess it's not an internal shade needs-a-mutex issue or anything ... this makes me wonder if the issue is that neutron gets screwed up somewhere with a certain level of parallel actions16:38
anteayamorgan: did you mean to +1 your own patch?16:38
mordreds/neutron/some combination of nova and neutron/16:38
morgananteaya: eh no, it just happened when i commented the upstream was ready for import16:38
mordredthe instance name thing is weird too16:38
clarkbmordred: cloudnull or if a particular sequence of actions is problematic16:38
*** slaweq has joined #openstack-infra16:39
morgananteaya: it doesn't really matter, unfortunately on plane wifi so hard to load gerrit.16:39
mordredclarkb: yah - like a sequence that happens in some combinatoin due to the parallel actions16:39
openstackgerritCarl Baldwin proposed openstack-infra/irc-meetings: Moving the Neutron L3 meeting agenda to etherpad
clarkbmordred: ya jus thinking that nodepool won't do parallel actions against a single cloud region16:39
mordredthread one does one step then thread two does another one or something16:39
sdaguefungi: do you have an instance where things were broken by this?16:39
sdagueto see if I can find a quick hotfix?16:39
anteayamorgan: yup, figured it was something like that, just looked unusual to me for you16:39
*** baumann has joined #openstack-infra16:39
mordredclarkb: if it's like "thread one create, thread two get, thread one get, thread three delete"16:40
mordredwe won't do more than one API call at a time16:40
mordredbut we can have multiple actions in flight at the cloud that we're polling on status for16:40
openstackgerritAlistair Coles proposed openstack-infra/project-config: Make gate-swift-tox-func-in-process-fast-post voting
fungisdague: sure, for example16:40
mordredthis is gonna be FUN16:40
clarkbwe can interleave the api calls for multiple "tasks"16:40
fungisdague: i'm seeing the results table entirely missing under firefox at least16:41
AJaegermordred: so, for your ldappool repo: That is a complete independent repo, correct? Not part of Big Tent via keystone?16:41
sdaguefungi: actually, it looks like it might be an easy fix16:41
stevemarmorgan: ^16:41
sdagueit's just a var rename miss16:41
mordredwhen I get airborne I'm going to try the cloudnull script with a ton of logging16:41
fungisdague: oh, cool. i'm happy to iterate on that real quick if you can push it up16:41
morganAJaeger: correct16:41
sdagueyeh, give me 10 minutes to locally test this fix16:42
*** d0ugal has joined #openstack-infra16:42
fungisdague: i'll hand-patch it in and confirm so we can get stuff working again16:42
jeblairclarkb, mordred: would it be helpful to add more logging of tasks (so we know exactly which tasks (ie, which ips/servers/etc) we are executing?16:42
AJaegermorgan: ok. Please make openstackci  Package Index Owner...16:42
*** esker has quit IRC16:42
morganAJaeger: the coreteam will be made up of -keystone (and likely will inherit openstack-keystone core) but it is not intended to be bigtent16:42
clarkbjeblair: maybe? its also possible that using cloudnull's reproduction case is a better place for that16:42
pabelangerclarkb: that will work too16:42
AJaegermorgan: as soon as pypi is fine, I can +2...16:43
jeblairclarkb: k, just throwing out ideas16:43
morganAJaeger: maintainer is not enough? i'll up it to owner :)16:43
clarkbjeblair: just because it is so much simpler than nodepools mass of logging16:43
AJaegermorgan: As documented: Owner, please ;)16:43
*** doug-fish has joined #openstack-infra16:43
openstackgerritSpencer Krum proposed openstack-infra/system-config: Puppetboard renamed the module
morganAJaeger: done16:44
*** andymaier has quit IRC16:44
*** signed8bit has joined #openstack-infra16:45
AJaegermorgan: +2ed16:45
*** jamesmcarthur has quit IRC16:45
*** Jeffrey4l has quit IRC16:45
morganAJaeger: woot.16:45
morganAJaeger: are new projects processed on fridays still or just as folks get around to it?16:45
AJaegermorgan: as time permits...16:45
anteayamorgan: so didn't get any traffic on it16:45
morgani only ask since i'm corrdinating with the former maintainer :)16:45
openstackgerritSean Dague proposed openstack-infra/system-config: fix typo in hideci.js
sdaguefungi: that should be it16:46
anteayadoes anyone have any opinions on importing a project with an MPL licence?16:46
anteayait is about to become an openstack repo?16:46
morgananteaya: MPL is apache2 compat.16:46
anteayamorgan: awesome16:46
sdaguethere is also a regression about jenkins no longer always being in the top of the list, let me see if I can sort out that one.16:46
morgananteaya: the only real issue was moving itnto tree with keystone and causing odd licencing issues with keystone having MPL code in it's tree16:47
anteayamorgan: I know very little about licensing and since the thread on legal discuss got no traffic asking once more for objections16:47
AJaegerfungi, pabelanger: for the bindep changes: As I'm going now on vacation, I'm not continuing on the bindep changes. Feel free to move it forward as you see fit and change the plan as needed ;)16:47
morgananteaya: this is not big tent, so it just uses the ci-infra and publishes to pypi16:47
*** gyee has joined #openstack-infra16:47
anteayamorgan: ah okay, so since this is it's own project, no licensing clashes with keystone now16:47
morganstevemar: ^ can you update that leagal thread to indicate this is resolved ?16:47
pabelangerAJaeger: sure, great work driving this16:47
anteayamorgan: perfect thank you16:48
fungisdague: giving it a shot now16:48
*** doug-fish has quit IRC16:48
morgananteaya: exactly16:48
*** jyuso has joined #openstack-infra16:48
fungisdague: i hand-patched it in and a force reload in my browser shows a previously broken display is working again16:49
*** sdake has joined #openstack-infra16:49
fungisdague: i'll approve this and abandon my revert--thanks!16:49
sdaguefungi: cool16:49
*** alex_xu has quit IRC16:49
*** thorst_ has quit IRC16:50
*** heyongli has quit IRC16:50
*** eliqiao has quit IRC16:50
*** jyuso1 has quit IRC16:51
*** eliqiao has joined #openstack-infra16:51
openstackgerritMerged openstack/requirements: Updated from generate-constraints
*** heyongli has joined #openstack-infra16:52
*** woodster_ has joined #openstack-infra16:52
openstackgerritMerged openstack/requirements: Add global requirement update on python-brick-cinderclient-ext project
*** alex_xu has joined #openstack-infra16:52
cloudnullsorry in osa meeting.16:54
*** javeriak has joined #openstack-infra16:54
cloudnullmordred:  it may be a parallel action problem and neutron becoming un-happy.16:54
*** lucasagomes is now known as lucas-dinner16:54
*** sambetts is now known as sambetts|afk16:55
cloudnullhowever I cant recreate it in heat.16:55
*** matt-borland has quit IRC16:55
cloudnullwhich may be due to some retry shinanigans they have to make that go16:55
nibalizernow i just touch 15 things a day and get no real progress done16:55
*** thorst_ has joined #openstack-infra16:56
stevemarmorgan: anteaya i'll reply to my own post on legal :P16:56
*** mkoderer__ has quit IRC16:57
cloudnullalso i running and rerunning tests i reliably get nodes with the same names and i cant see where thats coming from.16:57
anteayastevemar: thank you, you seem to be the only one concerned with the topic, and I thank you for that!16:58
funginibalizer: sounds like my day!16:58
stevemaranteaya: :P16:58
anteayastevemar: :)16:58
anteayanibalizer: the infra piano16:59
nibalizerfungi: i think you get more done than me :)16:59
fungitickle those ivories16:59
nibalizerthat was acutally a ww for a differen conversation, but it applies here ofc16:59
funginibalizer: i hardly see how. so... much... interruption16:59
* nibalizer looks at, weeps16:59
mordredcloudnull: so I wonder ... in shade we're not considering the fip actually attached until it shows up in the nova server dict16:59
mordredcloudnull: but heat is tracking the fip directly, so probably does not care whether nova "knows" about the fip at all17:00
mordredclarkb, Shrews ^^17:00
* anteaya hands nibalizer some tissue and a cat17:01
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support ftp publisher
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: run shell builders defined in JJB
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support zuul stop commands
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: register jobs from JJB
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: send ZMQ start/complete events
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add zuul_runner module
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: run ansible
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support job timeouts
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support scp publisher
*** thorst_ has quit IRC17:01
openstackgerritCarl Baldwin proposed openstack-infra/irc-meetings: Move the agenda for routed networks meeting to etherpad
*** mhickey has quit IRC17:01
cloudnullmordred: that may be17:02
mordredalso - heat is not checking for the fip to go into an active state - it's just issuing the create call17:02
*** abregman has joined #openstack-infra17:02
morganmordred: i think this is a case where shade needs to learn to track the fip =/17:02
openstackgerritSean Dague proposed openstack-infra/system-config: Create stable order for CI results stable
morganmordred: or pointing towards that17:02
mordredmorgan: it's starting to sound like that17:02
morganmordred: unfortunate :(17:03
sdaguefungi: one more niceity there
*** e0ne has joined #openstack-infra17:03
*** pvaneck has joined #openstack-infra17:03
*** thorst_ has joined #openstack-infra17:03
mordredmorgan: I will try a patch that does that inefficiently and see if the script still fails17:03
morganmordred: ok. i will happily look over said code when you have it17:04
*** mkoderer__ has joined #openstack-infra17:04
*** jerryz has joined #openstack-infra17:05
cloudnullmorgan mordred so maybe needs to be adjusted to track the FIP instead of looking for the update in server info17:05
fungisdague: is sort_by_name only sorting by the first character in each string, or is there some magic iteration hidden in the sort() method?17:05
*** lakshmiS has joined #openstack-infra17:06
sdaguefungi: this is javascript17:06
mordredcloudnull: yah. thats' exactly that I'm thinking17:06
morgancloudnull: that is the thought17:06
sdaguethat's not a string index17:06
sdaguethere is a tuple17:06
sdaguewell, array17:06
fungisdague: yeah, i should remember with js to check all assumptions at the door17:06
sdaguewe've built up 2 arrays, names + results, which then a bigger datastructure is created from17:07
fungiso nonfirstparty is an array of arrays. now it makes more sense17:07
sdague[0] is the ci+pipeline name17:07
sdague[1] is the result list17:07
sdagueas an array17:07
*** thorst_ has quit IRC17:07
fungiyep, lgtm17:07
* fungi still gets cross-eyed looking at javascript... at least it's not java!17:08
fungisoon i can say "at least it's not go!" (except, of course, when it _is_ go i guess)17:08
sdagueat least it's not haskell17:09
* Clint squints.17:09
* fungi still codes with a lisp17:09
* clarkb imagines fungi would enjoy haskell, that math background would be helpful17:09
fungi"haskell: at least it's not fortran!"17:10
AJaegercobol ;)17:10
fungioh yeah, there are definitely some unfortunate cobol experiences i'd like to forget17:10
fungion sco osr, no less17:11
AJaegerat least it gave me some money at that time...17:11
*** savihou has joined #openstack-infra17:11
fungiat least i got to work with some awesome (for the day) 4-way smp i486 servers17:12
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add private_key_file configuration for launcher
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add usernname configuration option for launcher
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add streaming console support
fungicrammed full of 32-line serial terminal multiplexer cards17:12
AJaegerthe good old days ;)17:13
fungier, concentrator cards. the multiplexers were in separate boxes17:13
*** davideagnello has joined #openstack-infra17:14
*** signed8bit is now known as signed8bit_Zzz17:16
pabelangerclarkb: ssh that is our qcow2 image in ovh-bhs117:16
clarkbpabelanger: neat17:17
openstackgerritSpencer Krum proposed openstack-infra/system-config: Send ansible-puppet timing information to graphite
openstackgerritPaul Belanger proposed openstack-infra/project-config: Switch to qcow2 for OVH clouds
pabelangerclarkb: ^think just that17:18
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add ssl upgrade playbook
clarkbpabelanger: can you share an image show on that image?17:21
greghaynesooo more qcow2 clouds17:21
nibalizerpabelanger: so i can't log in to that17:21
openstackgerritClark Boylan proposed openstack-infra/system-config: Add BHS1 region to openstackjenkins-ovh all-clouds
clarkbI just ran into ^ when trying to find the image myself17:22
pabelangernibalizer: I just deleted it17:22
pabelangerlet me launch another17:22
openstackgerritMerged openstack-infra/nodepool: Make sure worker arguments are integers
nibalizerpabelanger: no need17:22
*** doug-fish has joined #openstack-infra17:22
clarkbpabelanger: what user and region is the image? I want to look at the details17:22
nibalizerjust wondering if that was expected17:22
clarkbthat works too :)17:22
clarkbok disk format is still qcow2 so they aren't doing a conversion or anything, cool17:23
pabelangerclarkb: it was ovh-bhs117:23
*** doug-fish has quit IRC17:23
clarkbbut it has an rbd path17:23
pabelangerI've deleted it already, but can upload again17:23
clarkbI thought that qcow2 and ceph didn't play nice but it seems to work17:23
*** doug-fish has joined #openstack-infra17:24
*** dtantsur|bbl is now known as dtantsur17:24
*** jistr has quit IRC17:24
*** abregman has quit IRC17:24
pabelangerI didn't know that, will have to google to learn more17:24
*** vkramskikh_ has quit IRC17:24
*** e0ne has quit IRC17:24
*** salv-orlando has quit IRC17:24
*** DevBox has joined #openstack-infra17:24
*** jamesmca_ has quit IRC17:25
clarkbpabelanger: it has to do with copy on write in ceph with a shared image. Basically that image is shared by all 100 of our current test instances and only the bits unique to each get written17:25
*** ddieterly is now known as ddieterly[away]17:25
clarkbpabelanger: it is possible they just don't do that when we qcow217:25
openstackgerritMerged openstack-infra/system-config: fix typo in hideci.js
pabelangerclarkb: okay, good to know17:26
*** signed8bit_Zzz is now known as signed8bit17:26
*** tosky has quit IRC17:26
*** shardy has quit IRC17:27
clarkbpabelanger: maybe we can get jean daniel to ack that change?17:28
*** doug-fish has quit IRC17:28
pabelangerclarkb: WFM17:28
AJaegeranteaya, sdague, mtreinish please see for raildo 's v3 identity plan...17:32
*** IlyaG has joined #openstack-infra17:33
*** IlyaG has quit IRC17:34
*** d0ugal has quit IRC17:34
*** IlyaG has joined #openstack-infra17:34
*** julim has quit IRC17:35
*** rfolco has quit IRC17:35
raildothanks AJaeger, I already got some feedback from anteaya and fungi, but I appreciate any other feedback related to this :)17:35
*** rfolco has joined #openstack-infra17:35
anteayanibalizer: what is the line in that realizes Zara on the dev server? I'd like to learn so I can spot it next time, I looked at that patch and obviously missed it17:35
*** gyee has quit IRC17:35
*** pt_15 has quit IRC17:36
anteayastevemar: thank you for replying to yourself on legal-discuss17:37
*** ldnunes has quit IRC17:37
*** _deadnull is now known as deadnull_17:37
clarkbjesusaur: ok, when is a good time today to work on getting the logstash indexer rule stuff merged and applied to the workers?17:38
*** ldnunes has joined #openstack-infra17:38
*** alex_xu has quit IRC17:38
clarkbhrm /me notices we need to upgrade a puppet module
clarkbpleia2: ^ the zanata manifests also use the archive module, any concern doing that upgrade from zanata's perspective?17:41
AJaegerraildo: I suggest to mark all your changes as WIP for now, and go forward with one or two and iterate on them before finalizing the others...17:41
*** markvoelker has joined #openstack-infra17:41
*** Sukhdev has joined #openstack-infra17:42
*** SumitNaiksatam has joined #openstack-infra17:42
raildoAJaeger: I'm applying the suggestions on the nova patch, when I finish it, I'll mark all the other patches as WIP17:42
notmynameclarkb: mordred: fungi: FYI the current 3rd party CI for the golang code in swift is using a Makefile ( and simply runs `make test` in jenkins17:43
notmynameI added that info to the etherpad17:43
mordrednotmyname: awesome. thakns17:43
sdagueraildo: my primary concern is the stuff we think might cause breaks, like that project-config patch, we verify manually17:43
funginotmyname: excellent!17:43
sdaguea part of me longs for 'make test' in the python projects17:43
mordredsdague: clarkb has been advocating for that for ages17:44
nibalizeranteaya: there is no line17:44
fungisdague: you and clarkb could collaborate on getting his make-based tox replacement going again17:44
notmynamesdague: `./configure && make && sudo make install` FTW ;-)17:44
sdaguenotmyname: ++ exactly17:44
nibalizeri gad hoped for the patch i -1'd to be parented on my patch17:44
AJaegersdague: is correct? I doubt that  gate-tempest-dsvm-neutron-identity-v3-only-full-nv tests swift - can you enlighten me, please?17:44
mordrednotmyname: ++17:44
*** e0ne has joined #openstack-infra17:44
nibalizerim realizing now this was a secret plan i did not communicate, sorry17:45
mordrednotmyname: you know I was doing a ton of autoconf/automake stuff in drizzle before we spun up this openstack thing17:45
notmynamemordred: cool. or brain-warping. I've heard all that is rather crazy ;-)17:45
notmynamemordred: m4 or something like that?17:45
mordredit is17:45
mordrednotmyname: if you think pbr is bad, you should see the m4 version of it17:45
fungim4 is what makes it _less_ brain-warping actually17:45
notmynamemordred: lol17:45
*** ffrank has quit IRC17:46
fungiyou don't even want to look at the non-macro versions of that stuff17:46
mordrednotmyname, fungi
raildosdague: do you have any suggestion to improve the plan that I explained on the email? because this is my concern too, I don't want to send patches to cause breaks17:47
notmynamemordred: of course it's called "pandora". you don't want to open it17:47
anteayanibalizer: that would be why I can't find the line, there also is no spoon17:47
clarkbmordred: you are bringing back memories of swig and python and sadness17:48
anteayanibalizer: well you are communicating it now17:48
mordred is especially fun17:48
anteayanibalizer: so that you for that17:48
mordrednotmyname: :)17:48
*** doug-fis_ has joined #openstack-infra17:48
jeblaireven supports git17:49
*** mjblack has quit IRC17:49
sdagueraildo: mostly, we just need to make sure the patches get some manual verification, or test patches against projects that might break17:49
mordredjeblair: although I think we've learned more git now than at the time we wrote that - so I think those git things could likely be done better17:50
*** agentleweb has joined #openstack-infra17:50
openstackgerritMerged openstack-infra/project-config: Import ldappool into gerrit and setup project
openstackgerritMonty Taylor proposed openstack-infra/shade: Batch calls to list_floating_ips
anteayanibalizer: is there anything I need to do to help your secret plan to realize Zara to be realized?17:50
mordredanteaya: nicely phrased17:51
anteayamordred: thanks17:51
*** tosky has joined #openstack-infra17:51
anteayanibalizer: I did the best I could by submitted a patch I knew folks wouldn't like17:51
clarkbjesusaur: see comment on re making sure our rules are up to date17:51
openstackgerritMerged openstack/requirements: Add openstack/designate-tempest-plugin to projects list
anteayaI can't language anymore17:52
anteayaI try to language17:52
anteayahave tenses match17:52
*** d0ugal has joined #openstack-infra17:52
anteayaI fail consistently17:52
JayFWhere are the mirror configs at for you all? I see a puppet-bandersnatch module in openstack-infra ord but can't find the repo that calls it17:52
anteayaI have no language anymore17:52
JayFhelping mat128 do some research on adding Tiny Core Linux (tinyipa) mirrors in17:53
anteayaJayF: pabelanger is our resident many mirrors man person17:53
*** amrith has quit IRC17:53
anteayaJayF: he is a wealth of information17:53
openstackgerritPaul Belanger proposed openstack-infra/system-config: Also mirror centos 7 cloud repo
JayFpabelanger: is a good human being, and helpful and kind too :)17:53
pabelangerEmilienM: ^17:53
anteayamordred: wasn't that your name for him earlier today?17:53
anteayaJayF: indeed he is17:53
raildosdague: ++ I'll be aware of this manual verification on the next patches17:54
mordredmr. many mirrors guy person?17:54
EmilienMpabelanger: I love it17:54
JayFpabelanger: Where are mirrors configured? i.e. where would I go to find out what I'd have to do to add an additional mirror (tinycorelinux) to it?17:54
anteayamordred: that might have been it17:54
*** doug-fis_ has quit IRC17:54
JayFuh is that patch the asnwer to my question? lol17:54
anteayamordred: I mis-typed, see my earlier mini rant on no longer being able to language17:54
mat128JayF: looks close to it!17:54
pabelangerJayF: look at mirror_update.pp in openstack-infra/system-config that will get you started17:54
*** wajdi has joined #openstack-infra17:54
pabelangerJayF: mat128: you might be able to just rsync the mirror, but it depends on if said mirror purges files.17:55
pabelangerJayF: mat128: other wise, we need to do more creative things17:55
mat128pabelanger: I'll check with the TinyCore guys how they expect us to clone/mirror their repo, because there are no docs on it17:55
nibalizeranteaya: realize( Plan) yes17:56
mat128pabelanger: but otherwise, simply adding it to mirror_update.pp should get us started, right?17:56
mat128pabelanger: then it's all plumbing17:56
anteayanibalizer: awesome17:56
nibalizeranteaya: yes basically what you do is add that realize statement to the node definition i added in my patch17:56
nibalizeror, even better17:56
*** agentleweb has quit IRC17:56
pabelangermat128: Yup, mirror_update.pp is the entry point.  We'd likely need to discussion what the AFS structure looks like with jeblair too17:57
nibalizercreate a class called openstack_project::storyboard::dev next to the existing storyboard class and all it does is has htat realize statemen then calls the real storyboard class17:57
anteayacreate a class as a child to your patch?17:57
clarkbjesusaur: I think if we can get pleia2 and maybe nibalizer to ack the puppet-archive update we are good to approve these chages17:57
clarkbjesusaur: then double check the rules in the reconsumable repo match what we have in production then switch to them17:58
*** tqtran has joined #openstack-infra17:58
anteayaclarkb: isn't crinkle +2 on that repo yet?17:58
crinkleanteaya: not system-config17:58
*** pcaruana has quit IRC17:58
*** asselin has quit IRC17:58
anteayaah system-config, sorry17:59
anteayahad modules in my head17:59
*** dtardivel has quit IRC17:59
*** asettle has joined #openstack-infra17:59
*** esikache1 has joined #openstack-infra18:00
anteayaokey dokey, thank you18:01
nibalizerclarkb: jeblair link?18:01
clarkbnibalizer: that one18:01
nibalizerer jesusaur sorry jeblair18:01
stevemarmorgan: AJaeger is there some manual intervention needed for creating the ldappool repo in
morganstevemar: i'm guessing puppet/tasks have not run yet.18:02
morganstevemar: i don't see the stuff in gerrit yet.18:02
anteayastevemar: shouldn't be needed18:02
anteayastevemar morgan I'm going with what morgan is saying18:02
anteayastevemar: might I suggest a cup of tea?18:02
morganstevemar: i'm keeping watch on it18:02
clarkbnibalizer: yay ty18:02
stevemarmorgan: anteaya ah yeah, it's still going through the post job18:03
anteayastevemar: ah yeah that would have an effect18:03
stevemaranteaya: i will take your suggestion, but go with coffee and girl guide cookies instead18:03
*** mriedem has quit IRC18:03
anteayammmm, girl guide cookies18:03
*** mriedem has joined #openstack-infra18:03
anteayastevemar: do you have the chocolate covered ones?18:03
tdasilvahello, can some help me adding me to liberasurecode-core and liberasurecode-release groups in gerrit18:03
anteayastevemar: and hey did you see how much the red cross raised?18:04
anteayastevemar: record numbers18:04
stevemaranteaya: yep, not the mint kind though18:04
anteayastevemar: cool, they sound good18:04
*** andymaier has joined #openstack-infra18:04
stevemaranteaya: yah, gov matched all donations right18:05
anteayawill match, yes, not sure when that will happen though18:05
*** doug-fish has joined #openstack-infra18:05
anteayamoney is still pouring in as far as I know18:05
AJaegerstevemar: wait an hour after the change merges...18:05
*** hongbin has joined #openstack-infra18:05
stevemarAJaeger: but i want it *now* :P18:05
AJaegerstevemar: then timetravel in the future ;)18:06
pleia2clarkb, jesusaur: ugh, quite the delta between what we're using and where we are, I'll have a look at the changelog to see if anything pops out at me as a potential issue18:06
anteayastevemar: plus alberta gov is matching all alberta donations, so any alberta donations is * 318:06
stevemarAJaeger: :)18:06
stevemaranteaya: nice18:06
clarkbpleia2: should we remove nibalizer's +A?18:06
anteayastevemar: yeah, they were handing out money and cash cards yesterday18:07
pleia2clarkb: ah, no, he's more familiar with the module than I am18:07
*** esikache1 has quit IRC18:07
*** dtantsur is now known as dtantsur|afk18:07
pleia2clarkb: and I just did look at our use of it, it's pretty simple, should be fine18:08
*** d0ugal has quit IRC18:08
*** doug-fis_ has joined #openstack-infra18:08
*** _amrith_ has joined #openstack-infra18:08
*** _amrith_ is now known as amrith18:08
openstackgerritMerged openstack/requirements: Add magnum-ui to projects.txt
*** electrofelix has quit IRC18:09
*** rcernin has quit IRC18:09
jesusaurclarkb: im currently working on a bandersnatch upgrade, but that should be done by lunch18:09
openstackgerritMerged openstack/requirements: Add horizon-cisco-ui to projects.txt
clarkbjesusaur: cool, I will try to get the prep work of merging the cleanup changes done18:09
*** asilenkov_ has joined #openstack-infra18:10
*** doug-fish has quit IRC18:10
stevemaranteaya: costco had it set up so you could donate as your paid for groceries18:11
tdasilvaAJaeger, yolanda: can you help configuring liberasurecode-core and liberasurecode-release groups in gerrit?18:11
*** andymaier has quit IRC18:11
*** jpr has joined #openstack-infra18:11
AJaegerbetter ask soomebody else...18:11
clarkbtdasilva: you need the initial group member set?18:12
*** deadnull_ has quit IRC18:12
tdasilvaclarkb: correct18:12
*** tphummel has quit IRC18:12
clarkbtdasilva: what gerrit user should be added?18:12
anteayastevemar: awesome!18:13
*** doug-fis_ has quit IRC18:13
tdasilvaclarkb: thiago18:13
nibalizeryah archive didn't break the basics18:13
pleia2nibalizer: thanks18:13
nibalizerand if it did we can fix archive :P18:13
* nibalizer would love so much to have infra voting on voxpupuli18:13
clarkbtdasilva: done18:14
* persia finally understands why so many folk were dressed in red yesterday18:14
tdasilvaclarkb: thanks!!!18:14
openstackgerritAnita Kuno proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server
pleia2we shouldn't actually notice any problems immediately anyway, not until we update something (the xml file or a source), and we'd likely do that on dev first anyway18:14
*** javeriak has quit IRC18:14
anteayapersia: why were so many folks dressed in red yesterday?18:14
* anteaya was not dressed in red yesterday18:14
*** jistr has joined #openstack-infra18:15
*** jistr is now known as jistr|afk18:15
persiaanteaya: Red Cross drive18:15
anteayapersia: where you are?18:15
*** amitgandhinz has quit IRC18:15
anteayapersia: ah, yeah that would be it18:16
anteayanibalizer: 31570618:16
*** jpr has quit IRC18:16
*** javeriak has joined #openstack-infra18:16
*** amitgandhinz has joined #openstack-infra18:16
*** julim has joined #openstack-infra18:17
*** openstackgerrit has quit IRC18:18
*** openstackgerrit has joined #openstack-infra18:18
*** markvoelker_ has joined #openstack-infra18:19
*** andymaier has joined #openstack-infra18:20
*** hashar has joined #openstack-infra18:20
*** javeriak has quit IRC18:21
*** markvoelker has quit IRC18:22
anteayanibalizer: off line for a while, will check back later, thanks for your help on this18:22
*** jtomasek has joined #openstack-infra18:26
*** ddieterly[away] has quit IRC18:26
*** ri0 has quit IRC18:27
*** onovy has quit IRC18:28
*** onovy has joined #openstack-infra18:28
openstackgerritMerged openstack-infra/system-config: Upgrade the archive puppet module version
*** ri0 has joined #openstack-infra18:29
*** njohnston__ has left #openstack-infra18:29
openstackgerritNate Johnston proposed openstack-infra/irc-meetings: Change time for FWaaS meeting
*** njohnston__ has joined #openstack-infra18:29
*** johnny_ has joined #openstack-infra18:29
openstackgerritMerged openstack-infra/puppet-logstash: Fix beaker tests
*** chem has quit IRC18:30
openstackgerritMonty Taylor proposed openstack-infra/shade: Add floating IPs to server dict ourselves
mordredShrews: ^^18:31
*** claudiub has joined #openstack-infra18:31
*** jpr has joined #openstack-infra18:31
mordredcloudnull: ^^ that is not tested at all - but is a first-sketch at pulling the info from neutron ourselves18:31
mordredit needs a config flag18:31
*** ri0 has quit IRC18:33
openstackgerritJames E. Blair proposed openstack-infra/zuul: Properly shutdown apscheduler on reconfigure
jeblairclarkb, Shrews, morgan, pabelanger, jhesketh: ^ that should address the flaky test_timer that has been sniping all of the v25 changes18:35
cloudnulli love the commit message.18:35
cloudnullmuch sadness18:35
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support ftp publisher
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: run shell builders defined in JJB
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support zuul stop commands
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: register jobs from JJB
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: send ZMQ start/complete events
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add zuul_runner module
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: add streaming console support
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: run ansible
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support job timeouts
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: support scp publisher
pabelangerjeblair: ack18:35
*** mjblack has joined #openstack-infra18:36
*** ddieterly has joined #openstack-infra18:36
*** kmartin has joined #openstack-infra18:36
mordredcloudnull: :)18:37
*** esker has joined #openstack-infra18:38
*** esker has quit IRC18:39
*** esker has joined #openstack-infra18:39
*** tphummel has joined #openstack-infra18:39
*** tphummel has quit IRC18:39
jeblairmordred: why continue on ipv6?18:41
clarkbjeblair: neutron doens't support ipv6 floating IPs18:41
jeblairclarkb: good, i hoped that was the answer :)18:41
*** e0ne has quit IRC18:41
mordredclarkb, jeblair: details notwithstanding - does that general idea seem somewhat un vomit inducing?18:42
openstackgerritMerged openstack-infra/system-config: Add BHS1 region to openstackjenkins-ovh all-clouds
*** sdake has quit IRC18:44
clarkbmordred: it is unfortunate that we have to list all floating IPs to check18:45
clarkbbut that is a result of not having a shared key here other than the ip addr18:45
clarkb*reliable shared key18:45
jeblairmordred: i think it could probably use a code comment explaining why we have to do that... i honestly think most of that explanation is in irc atm -- so that's my feedback as someone who isn't deeply involved in this issue.18:45
mordredclarkb: yah. that is, btw, why the previous patch adds batched listing18:45
jeblairmordred: and i left a nit on the change.18:45
mordredjeblair: cool. I can do that18:45
*** nadya has quit IRC18:46
*** dmk0202 has joined #openstack-infra18:47
mordredcloudnull: any chance I could get an account on osic with enough quota to do your 25-server test? (and to test shade things on it from time to time?) I will most of the time not use it18:47
*** oanson has joined #openstack-infra18:48
*** IlyaG has quit IRC18:49
*** jamesmcarthur has joined #openstack-infra18:49
openstackgerritMerged openstack/requirements: Add pointer for SUSE python packaging devel project
cloudnullmordred: i want to say yes but so let me ask the right people. :)18:52
openstackgerritMerged openstack/requirements: Update global-requirements.txt for python-kafka version
* cloudnull is a follower of production controls these days.18:52
clarkbmordred: maybe we can suggest that neutron api grow a reverse lookup?18:53
*** tellesnobrega has left #openstack-infra18:53
clarkbmordred: seems more efficient ot have eg mysql do this join than python doing multiple api queries18:53
*** ddieterly is now known as ddieterly[away]18:54
*** dizquierdo is now known as dizquierdo_afk18:55
cloudnullalso mordred on the FIP topic when i do a create of my 25 nodes I see 25 returns for a FIP however not all are unique18:55
cloudnullthe number differs from run to run18:55
pabelangerclarkb: jeblair: we're making good progress on the DIB uploads today, 23 left. That is with 12 upload workers.  I haven't see our TX peak 400Mb, so we could likely bump up a few more upload-workers, assuming 800Mb TX18:56
cloudnullbut in general there is some number of non-uniqe ids and that number is the amount of servers missing a fip18:56
*** awaugama has joined #openstack-infra18:56
jeblairpabelanger: yeah, i'm still thinking 16 :)18:56
clarkbpabelanger: we should double check on that, rax does some weird thing where you get half your bandwidth on public interface iirc18:56
jeblairclarkb: i thought that was old?18:56
openstackgerritMerged openstack/requirements: Add pifpaf to requirements
clarkbjeblair: if they changed that behavior then I didn't realize they did18:57
pabelangerbluebox we are pushing 227Mb currently, definitely the fastest one18:57
pabelangerclarkb: suggests who to confirm with?18:57
clarkbpabelanger: osic and bluebox are quick and use qcow2 so its double win18:57
pabelangerclarkb: yar18:58
clarkbpabelanger: I think you can flavor list to get the rx/tx factor the ndouble check rax docs for what those numbers mean18:58
pabelangerwe're only getting 20Mb from rax18:58
pabelangerclarkb: ack18:58
fungipabelanger: clarkb: yep, it does look like we're hitting the 400m cap probably
pabelangerfungi: Ah, good idea checking cacti18:58
*** sambetts|afk has quit IRC18:59
awaugamamay be in the wrong place, I'm new to openstack and trying to sigh the ICLA on and I'm getting a "Code Review - Error, server error, cannot store information" every time I try to submit the agreement.  any ideas?18:59
mordredcloudnull: WOW18:59
fungiso i wouldn't expect to be able to push it any faster than we already are (and we may already be overdriving that to the point where we're hampering throughput slightly, depending on how graceful their rate limiting implementation is)19:00
mordredcloudnull: that seems like a large bug19:00
fungiawaugama: have you been following the instructions at ?19:00
fungiawaugama: the most likely causes are covered in the second paragraph (about setting up your foundation membership correctly first)19:01
mordredarmax: cloudnull may have found a quite spectacular bug in neutron19:01
armaxmordred: only one? :)19:01
openstackgerritMerged openstack/requirements: Upgrade PyKMIP to 0.5.0
openstackgerritMerged openstack/requirements: Drop influxdb
mordredarmax: :)19:02
fungiarmax: well, it has to do with floating ips, so i suppose you could class that as an entire family of bugs19:02
*** d0ugal has joined #openstack-infra19:02
cloudnullto get that data I added a print here took the stdout and did my gistthiname's19:02
mordredarmax: we're trying to track down why sometimes creating a floating ip on a server's port does not result in a floating ip on that server19:02
*** sambetts has joined #openstack-infra19:03
armaxmordred: wasn’t that we talked about last time?19:03
*** jerryz has quit IRC19:03
*** HeOS has quit IRC19:03
cloudnullmordred: this may be our clouds fault too. we're still running kilo19:04
mordredarmax: yes - but we've got a friendly cloud operator helping us find more issues19:04
cloudnullso maybe fixed in !kilo19:04
cloudnullwe're going to upgrade tomorrow though19:04
*** mjblack has quit IRC19:04
fungiosic has deployed openstack's shiny new kilo release19:04
mordredarmax: f1169479-7d88-4526-a60a-82f18b617077 is an id for 2 different fip entries19:04
fungicloudnull: ooh! mitaka? or liberty?19:04
cloudnullmitaka fast follow19:05
mordredarmax: in
fungibaby steps. better than not upgrading!19:05
AJaegerfungi, speaking about kilo: openstack-manuals has frozen stable/kilo - could you put this on your list of branches to retire, when you go through them, please?19:05
cloudnulli wanted to do the upgrade ages ago , it was hard to convince folks19:05
awaugamafungi: yeah, i followed the instructions, just tried to recreate the account to verify the email was in use.19:05
cloudnullbut i think we're getting on the right track19:06
awaugamafungi: actually, just saw, I think I did community.  Let me fix that19:06
fungiawaugama: yep, we tried to call that out explicitly in the directions since it's a common mistake some people make19:06
dansmithdo we know that zuul queue is at 4600 items?19:06
fungilet me know if that doesn't help, and we can take troubleshooting further19:06
fungidansmith: we do now i guess? ;)19:07
AJaegerdansmith: continue watching - it should go down again eventually ;)19:07
dansmithit's been climbing for like 30m19:07
awaugamafungi: thanks.  Guess I was going too fast.  Do I need to register again or is there a way to upgrade accounts?19:07
mordredcloudnull: I find it curious that it was hard to convince people ... seems like running mitaka would be more useful for osic's mission19:07
dansmithwhich is usually a bad sign I thought19:07
*** armax has quit IRC19:07
mordredcloudnull: but I'm glad you have had success!19:07
cloudnullanyway. armax, this may be a kilo specific bug and maybe infra is seeing that problem in OSIC and BlueBox because we're still on the Kilo release?19:07
fungiarmax just lost his floating ip19:07
awaugamafungi: found it, nevermind19:08
cloudnullmordred: yes i would agree. but I have to deal with people and people dont make sense. :)19:09
fungidansmith: most common cause for the events/results queues spiking is reconfiguration taking a while. i'm looking now to see if the debug logs indicate any signs of actual distress19:09
*** matt-borland has joined #openstack-infra19:09
*** asettle has quit IRC19:09
mordredcloudnull: people are often the bug19:09
mordredcloudnull: I think our bluebox cloud is actually juno19:10
mordredjlk: yah?19:10
dansmithfungi: okay, I just have it in red right above my head all the time, so I notice when it starts to grow.. usually gives me a feeling of what is normal and not19:10
AJaegerzuul queue just went from 5500 to 0...19:10
dansmithah just flushed19:10
* fungi ceases investigating19:10
mordredjlk: btw - see the scrollback about what cloudnull found wrt duplicate FIP entries19:10
dtroyerAnyone know what builds the ICS files from the irc-meetings repo?19:11
fungidansmith: looks like maybe a bunch of changes merged for multiple high-volume projects and zuul was generating the merge-check work requests for all their remaining open changes19:11
dtroyerthe real question is what decides the odd/even week start...19:11
clarkbdtroyer: yaml2ical iirc19:11
mordredinfra-root: I'm inclined to wait until osic upgrades to see if that fixes it before we land the "build the network info ourselves" patch19:12
dansmithfungi: yeah, and fake-merge-conflict'd several of mine :)19:12
clarkbdtroyer: its some iso standard and date prints them out with the right flag19:12
jlkoh good times19:12
dansmithfungi: anyway, just seemed really high and long, so.. sorry :)19:12
mordredjlk: right?19:12
jlkcloudnull: if you have a bug filed on that, can you link me? I want to make sure we get any fix into our Mitaka update19:12
fungidansmith: which ones got flagged as merge-conflicting?19:12
jeblairfungi: saw another ord connection error19:12
dansmithfungi: for example :
*** asettle has joined #openstack-infra19:13
clarkbdtroyer: date +%U19:13
fungidansmith: oh, that's marked as merge conflicting because it has a non-current parent. gerrit won't actually let that merge19:13
dtroyerclarkb: nice.  Too bad $EMPLOYER, and Mac Calendar both tell me this is w2019:14
dansmithfungi: it does? it's the bottom of the stack19:14
fungidansmith: orange dot next to the "parents"19:14
dtroyerclarkb: thanks19:14
dansmithfungi: and I just rechecked and it started running19:14
pabelangerclarkb: apparently we have 1600.00/mbps:
fungidansmith: oh, nevermind, that's just saying that the parent is no longer the branch tip i guess, not that the change depended on a change which has a newer patchset19:15
pabelangerclarkb: but according to Outbound public Internet bandwidth can be up to 40% of the aggregate limit19:15
pabelangerso, 640mbps?19:16
AJaegerfungi, shall I send an email to openstack-infra for kilo branch retirement of openstack-manual, or do you have a list somewhere?19:16
fungijeblair: what sort of ord network interruption?19:16
*** oanson has quit IRC19:16
*** salv-orlando has joined #openstack-infra19:16
fungiAJaeger: the stable branch team has been assembling a list of repos for me to delete branches on, i haven't started yet (2015.1.4 was only tagged a day or two ago)19:17
*** ddieterly[away] is now known as ddieterly19:17
AJaegerit's not critical, just wanted to know what's the best way is to get added to that list - since the stable team does not take care of openstack-manuals in general. Shall I ask them to add it?19:18
*** bpokorny has quit IRC19:18
*** timl0101 has quit IRC19:18
fungiAJaeger: not sure, i've been letting them handle more of that work this time around for the projects under release management, but an e-mail to the infra ml is probably fine as a placeholder19:18
*** timl0101 has joined #openstack-infra19:18
openstackgerritPaul Belanger proposed openstack-infra/system-config: Bump nodepool-builder to 16 upload-workers
AJaegerwill do, thanks19:19
pabelangerjeblair: clarkb: fungi: ^ lets see if we can break 400mbps19:19
fungipabelanger: i'm not immediately opposed, but judging from the graph i have a feeling we're already seeing some minor packet loss slowing down the current transfers and will just end up making that worse if we try to add more19:20
*** jang has joined #openstack-infra19:21
AJaegerteam, I'll sign off in a few minutes - and then it's vacation time. Might do some reviews tomorrow but don't count on me for the next two weeks... Have fun!19:21
mordredAJaeger: enjoy!19:21
fungiAJaeger: enjoy your time off!19:22
AJaegerthanks, will try ;)19:22
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only
*** kfarr has joined #openstack-infra19:23
clarkbmordred: ok finally left comments on the fip change, its just questions no -119:24
mordredclarkb: cool - did you catch my thing above about wanting to see if the osic upgrade fixes it before landing that?19:25
clarkbmordred: I didn't but that seems reaosnably, note bluebox also suffers this behavior19:25
cloudnulljlk: i've not filled a bug yet. however I will when I can rule out its not just a problem on my end19:26
mordredclarkb: yah. they're also going to upgrade our bluebox cloud19:27
mordredclarkb: I think we'll likely wind up landing the patch anyway, so that shade works well with older clouds - but if we've got info on whether it's fixed or not it'll help us track when/how we need to enable the setting19:27
*** AJaeger has quit IRC19:28
mordredclarkb: since it does have a cost to enable19:28
*** r-daneel has joined #openstack-infra19:28
cloudnullI'm hoping (figers crossed) that this is some neutron thing that was resolved in liberty.19:30
cloudnullim going to keep beating on it, it may be something else19:30
jeblairfungi: oh the usual couldn't contact git.o.o from a job running in ord19:30
jeblairfungi: just mentioning it so we know it's still occasionally happening19:31
fungijeblair: oh yeah, it's definitely still happening
*** Sukhdev has quit IRC19:32
fungiif you go to the logstash link there and add the node_provider column, you'll see it's (almost) all rax-ord19:32
*** Sukhdev has joined #openstack-infra19:32
fungiocassionally bluebox-sjc119:32
*** slaweq_ has joined #openstack-infra19:32
*** esikache1 has joined #openstack-infra19:32
*** slaweq has quit IRC19:32
fungigoing back a couple pages, i can find one hit for rax-iad too19:32
*** esikache1 has quit IRC19:33
*** nadya has joined #openstack-infra19:35
*** asettle has quit IRC19:35
*** apoorvad has quit IRC19:36
*** slaweq_ has quit IRC19:37
cloudnullmordred: i've slammed this together in shade to track a FIP and return server data when attached -- its quite a bit faster but the duplicate FIP ID issues are still present.19:38
* cloudnull goes off to eat19:38
*** bswartz has quit IRC19:39
*** slaweq_ has joined #openstack-infra19:41
pabelangerfungi: ya, if people want to try, I am game.19:42
*** ociuhandu has quit IRC19:43
pabelangerwe're also down to 6 images, only rax-iad19:43
pabelangerand 00:04:28:52 into uploads, so we've made progress19:43
*** bpokorny has joined #openstack-infra19:45
pabelangerfungi: clarkb: any good documentation for creating a kilo-eol tag?19:47
pabelangerlooking in the release team docs now19:47
fungipabelanger: it's something i've done manually in the past, though i'm trying to entice te stable branch managers and/or release team to take on more of that this time around19:48
fungirather, they've expressed interest in doing more of it19:48
fungithey're still in the process of putting together the list of repos that need it done19:49
*** tphummel has joined #openstack-infra19:49
*** signed8bit has quit IRC19:49
pabelangerfungi: okay, so maybe I can just add openstack-manuals to that list19:49
mtreinishclarkb: I'm working on adding elastic-recheck support to openstack-health: and right now I'm trying to get around how slow it can be19:51
*** matrohon has joined #openstack-infra19:51
fungipabelanger: yep, i just asked about it over in #openstack-stable19:51
*** jistr|afk has quit IRC19:52
clarkbmtreinish: the es queries being slow? write better queries :)19:52
clarkbmtreinish: last I saw many of them are search every file for this one string that only shows up in ome place ever19:52
mtreinishclarkb: right now the fastest I've managed to get it is about 20sec but it often takes several minutes to run all the queries all the 10 most recent changes (it's doing it in parallel with a thread per change)19:52
*** yamahata has quit IRC19:52
pabelangerfungi: sure, connecting in19:53
clarkbmtreinish: the trick is to filter as much as possible so build_name specifics and filename specifics and so on19:53
mtreinishclarkb: heh, I'm treating the e-r queries as a black box right now19:53
mtreinishclarkb: it's using whatever the e-r classifier.classify() filters too19:53
mtreinishwhich I think is zuul build uuid, change num, and rev19:54
mtreinishand I also have it filter to within the last hour19:54
fungipabelanger: conversation there is a lot more intermittent than here, so probably safe to wait until refreshes and that way i won't need to reiterate19:54
*** Swami has quit IRC19:54
mtreinishbut it's still dependent on whatever the actually query in e-r is19:54
clarkbmtreinish: oh you are doing it per change? maybe I dont understand what you are trying to do19:54
pabelangerfungi: Yup, I'm good with that19:55
clarkber per build19:55
*** apoorvad has joined #openstack-infra19:55
mtreinishclarkb: on o-h's home page there is a list of 10 recent failures19:55
openstackgerritDavanum Srinivas (dims) proposed openstack-infra/project-config: Adding library Delimiter to the CI system
mtreinishI'm trying to add a list of identified bugs to that list19:55
pabelangerWouldn't mind a quick review on adds the cloud repo to our centos mirror.  About 1.7G of new data19:56
clarkbgotcha that should be reasonably quick, definitely double chwck the queries though bad queries will be slow19:56
asselin_dstufft, does anything else need to be done after upgrading to 1.10 bandersnatch? or should it just 'work'?19:56
jeblairfungi, pabelanger:
mtreinishso I'm using the classifier from e-r bot which, loops over all the queries for a single build19:56
dstufftasselin_: should just work19:56
dstufftI mean19:57
clarkbmtreinish: we also have a ton of queries that can be deleted19:57
dstufftupgrading will trigger a full sync19:57
mtreinishclarkb: yeah, I guess I"ll need to instrument e-r a bit better to see where it's spending the most time19:57
dstufftso you'll have to wait for that19:57
mtreinishclarkb: yeah, there is definitely some pruning we can do on the e-r19:57
asselin_dstufft, oh...when? how?19:57
*** Swami has joined #openstack-infra19:57
fungijeblair: alternate url for the same page... what's the relevance?19:58
dstufftasselin_: I'm not sure the question, the next time you run `bandersnatch mirror` it should automatically do a full sync19:58
dstufftbut ti'll take longer than normal19:58
jeblairfungi: it updates immediately19:58
*** yamahata has joined #openstack-infra19:58
fungijeblair: neither of them are updating immediately for me19:58
pabelangerjeblair: danke19:58
fungijeblair: still has 19:39 as the most recent content19:59
*** asettle has joined #openstack-infra19:59
asselin_dstufft, ok....seems only one package updated. will check again in next 5 minutes....19:59
*** asettle has quit IRC19:59
jeblairfungi: 2016-05-12T19:53:23  <pabelanger> ohai!19:59
pabelangeryup, I see that19:59
fungijeblair: weird. forced refreshes in my browser are still at 19:39 on both those pages19:59
*** asettle has joined #openstack-infra20:00
* fungi wonders if there are some weird cache headers set on there20:00
jeblairfungi: that is weird.  i see 19:39 only on the latest.log.html; date.log shows 19:5320:00
*** asettle has quit IRC20:00
fungijeblair: refreshed again just now and they both updated for me20:00
jeblairfungi: they are bothe at 19:53 now for me as well20:01
*** jamesmcarthur has quit IRC20:03
*** slaweq_ is now known as slaweq20:05
openstackgerritDavid Lyle proposed openstack/requirements: Remove django-bootstrap-form
*** ddieterly is now known as ddieterly[away]20:07
fungijeblair: having trouble tracking down why dansmith's 315681,2 change consistently failed to merge according to zuul. even with merge -s resolve it didn't conflict with the master branch tip for me, and the times zuul reported merge failures for it didn't seem to coincide with events on its depends-on change either. the zuul debug log just reports that it failed to merge20:07
*** awaugama has quit IRC20:07
dansmithfungi: it failed after recheck too.. I just pushed up another20:08
fungidansmith: yeah, i saw that. seems strange, but it's likely i'm missing something20:08
dansmithI don't get it either.. it failed immediately when I first pushed it up, hardly had time to be in conflict with anything :)20:09
fungiwell, actually the last merge failure for it happened when zuul reported job failures on 312755,1320:09
fungi(its depends-on)20:09
*** yamahata has quit IRC20:09
fungibut previous merge failures didn't seem to coorelate to anything there20:09
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Consul has now merged (and it works)
jeblairfungi: ack, will look20:10
fungidansmith: also on 312755,13 what did you mean by "jobs didn't start"?20:11
fungilooks like you got check results reported on that change 15 minutes before you added that recheck comment20:11
*** ddieterly[away] is now known as ddieterly20:11
*** amrith is now known as _amrith_20:12
dansmithfungi: I meant one of the jobs did this:
*** njohnston__ is now known as njohnston_afk20:13
fungidansmith: oh, joy. that's yet another network failure in rackspace ord20:14
dansmithfungi: \o/20:14
* fungi checks up on the long-standing ticket about this20:14
openstackgerritDoug Hellmann proposed openstack-infra/project-config: add check-requirements to networking projects
fungisigh. they're confused _still_20:15
*** maishsk has quit IRC20:17
openstackgerritDoug Hellmann proposed openstack/requirements: add a validation command to ensure projects use check-requirements
*** maishsk has joined #openstack-infra20:18
*** ayoung has quit IRC20:20
*** wajdi has quit IRC20:20
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Consul has now merged (and it works)
*** dizquierdo_afk has quit IRC20:22
*** keedya has quit IRC20:22
jeblairzaro: what's the status on the git gc investigation?20:23
*** Sukhdev has quit IRC20:23
asselin_dstufft, seems to be running now, though for some reason we don't see the outpu in syslog. Might be an issue on our system. Thanks for the confirmation.20:23
*** Sukhdev has joined #openstack-infra20:24
*** kgiusti has left #openstack-infra20:24
fungirackspace's network engineering team wants us to avoid deleting instances which exhibit this failure mode. anybody have ideas on how we might make that dream a reality? throwaway d-g change to bysy-wait the setup script if a ping check to git.o.o fails but short-circuits and ends immediately if not, then recheck it a bunch and hold one that takes too long?20:24
mtreinishclarkb: it looks like each individual es query is taking like 1-2 secs., but that adds up when we have to run 115 of them20:24
*** rcernin has joined #openstack-infra20:25
jeblairfungi: ooh, in zuulv2.5 or zuulv3, we could have a special exit code or something that we use to tell nodepool to hold the node20:26
jesusaurclarkb: so where are we with the logstash-filter repo?20:26
mordredjeblair: oooh20:26
openstackgerritNikhil Komawar proposed openstack-infra/project-config: Add #openstack-quota to the IRC channels
openstackgerritDavid Lyle proposed openstack/requirements: Remove XStatic-JQuery.Bootstrap.Wizard
fungijeblair: speaking of gerrit git gc, not sure if you saw while you were away last week but i ended up doubling the ~gerrit2 volume because we've seen constant growth in the git repos since the 2.11 upgrade20:27
*** abregman has joined #openstack-infra20:27
jeblairfungi: i missed that, thanks20:28
fungijeblair: the yearly graph is pretty obvious
fungimid-december it starts growing linearly20:28
fungijeblair: unrelated to git but still in a similar vein, we also nearly ran out of space in the gerrit trove instance, so i grew its backing volume from 10gb to 50gb20:29
*** wznoinsk has quit IRC20:32
*** yamahata has joined #openstack-infra20:33
openstackgerritDoug Hellmann proposed openstack-infra/project-config: ensure all projects syncing global requirements use check-requirements
openstackgerritJeremy Stanley proposed openstack-infra/devstack-gate: DO NOT MERGE - attempt to find a network failure
clarkbjesusaur: the test fixes have merged, need to make sure those rules are up to date then merge the rule changes and merge switch over to them20:35
clarkbjesusaur: I am finishing up lunch give me aboht 20 minutes20:35
*** ericksonsantos has quit IRC20:35
mordredjeblair, fungi: related to that, I have started (eversoslightly) poking at trove clustering and replication and how that might or might not help yus20:35
jesusaurclarkb: ok20:35
*** abregman has quit IRC20:36
*** hamzy has quit IRC20:36
mordredI have not accomplished as much investigation as I'd like20:36
clarkbjesusaur: maybe you can diff new rules and production?20:36
jesusaurclarkb: yeah, was just about to make sure our filters are up-to-date20:36
openstackgerritNikhil Komawar proposed openstack-infra/project-config: Adding library Delimiter to the CI system
fungimordred: that would be an awesome outcome if it accomplishes what we need20:37
fungii'm going to go find some grub, back in a while20:37
*** aeng has joined #openstack-infra20:39
openstackgerritMatthew Treinish proposed openstack-infra/elastic-recheck: Remove queries with no hits
*** raildo is now known as raildo-afk20:41
*** eharney has quit IRC20:42
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] make grenade job temporary non-voting
timothyb89fungi, pabelanger, clarkb, anteaya, mtreinish: if you have a moment, could you take a look at this revised stackviz patch?
*** thorst_ has joined #openstack-infra20:42
openstackgerritVitaly Gridnev proposed openstack-infra/project-config: [sahara] return acls for sahara-tests
jeblairdansmith, fungi: 315681 Depends-On 312755 has parent 312751;  If i check out master and cherry-pick 312751, 312755, and 315681, i get a file-level merge conflict20:43
pabelangertimothyb89: Ya, I meant to leave a comment on that.  It looks good to be, but I'll likely drop a -1 workflow on it, so I can help monitor it once we land it20:43
pabelangertimothyb89: if we get enough votes, we can likely build images this weekend20:43
timothyb89pabelanger: good to hear, thank you!20:43
jeblairdansmith, fungi: which zuul did as well:
*** gyee has joined #openstack-infra20:44
jesusaurclarkb: I've confirmed that the filter section is identical, we can either change the example output to use elasticsaerch instead of elasticsearch_http in a new patchset or in a follow-up change20:44
*** _ari_ has quit IRC20:44
*** wznoinsk has joined #openstack-infra20:45
pabelangertimothyb89: Ya, I'm happy now that we don't sudo npm install npm20:45
openstackgerritMerged openstack-infra/devstack-gate: Increase default OS_TEST_TIMEOUT to 1200
dansmithjeblair: ah20:45
*** Sukhdev has quit IRC20:46
*** ddieterly is now known as ddieterly[away]20:46
openstackgerritMerged openstack/requirements: Add review guideline about projects.txt updates
openstackgerritMerged openstack/requirements: Update to recent release of tosca-parser 0.5.0
*** amitgandhinz has quit IRC20:49
*** amitgandhinz has joined #openstack-infra20:49
*** armax has joined #openstack-infra20:50
*** Goneri has quit IRC20:50
armaxmordred, fungi: sorry for the abrupt disconnection20:50
clarkbjesusaur: lets do a followup change20:50
openstackgerritSpencer Krum proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server
clarkbjesusaur: now to find the change that uses that new list of rules20:50
nibalizeranteaya: ^^ is what I'm thinking20:51
*** nadya has quit IRC20:51
*** andymaier has quit IRC20:51
jesusaurclarkb: 31005220:51
*** nadya has joined #openstack-infra20:51
*** DevBox has quit IRC20:52
anteayanibalizer: thanks, yeah I'm glad you did the patch yourself I didn't get that vision from the words I read and re-read earlier20:52
openstackgerritEdgar Magana proposed openstack-infra/irc-meetings: Moving meetings from odd weeks to even
anteayanibalizer: thank you20:52
sdaguejhesketh: when you get up - would be awesome to put Jenkins back on top20:52
jesusaurclarkb: which depends on 31005120:52
clarkbty ty20:53
armaxmordred: so it looks two concurrent requests end up getting the same FIP20:53
clarkbjesusaur: I have approved the filters project change20:53
armaxat least from what I gather reading
openstackgerritMerged openstack-infra/logstash-filters: Move the input and output sections to example files
mordredarmax: yah. that's our take20:54
mordredarmax: they're currently running kilo but are upgrading soon to liberty and then to mitaka soon after20:54
mordredarmax: so I think we'd like to see if it's fixed in newer20:54
*** kzaitsev_mb has joined #openstack-infra20:54
armaxknowing how well we deal with concurrency, it’s entirely possible that unless this was found out before20:54
openstackgerritAnita Kuno proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server
*** dprince has quit IRC20:54
armaxit’s still in master20:54
clarkbjesusaur: I am going to disable puppet on all but one of the logstash workers then we can get 31005[12] in20:55
armaxmordred: let me see what my fishing expedition is gonna yield20:55
anteayanibalizer: did I get your email address correct for the Co-Authored By: bit ?20:55
jesusaurclarkb: sounds good20:55
nibalizeranteaya: i prefer but I don't actually care :)20:55
mordredarmax: cool - thanks - and let me know if I can be helpful20:56
anteayanibalizer: I care, I'll change it, thanks20:56
cloudnullarmax: i might be able to get a newton env stood up today and see if its still bothersome their using the same shade bits i'm attacking the OSIC with.20:56
armaxmordred: will do, I’ll file a bug nonethless just to err on the side of caution20:56
openstackgerritAnita Kuno proposed openstack-infra/system-config: Realize Zara on the newly created storyboard-dev server
armaxcloudnull: how often does that occur?20:56
*** nadya has quit IRC20:56
cloudnulldepends on how lazy im feeling that day20:57
clarkbjesusaur: puppet disabling is done, now to finish up code reviews20:57
armaxI don’t recall a race like this ever being tackled but I am getting old fast so my memory is starting to fail me20:57
cloudnull++ ^ I know how you feel . :)20:58
* cloudnull goes to yell at some kids20:58
anteayaclarkb: 31005 looks like it is missing a digit if it is a patch id20:58
armaxcloudnull: do you happen to have the sequence of steps you take to trigger the issue?20:58
anteayacloudnull: make sure they get off your lawn20:58
cloudnull^ that20:58
clarkbanteaya: 310051 and 31005220:58
clarkbanteaya: my regex would match both20:59
*** rbrndt has quit IRC20:59
cloudnullanteaya: indeed. :)20:59
clarkbanyone else want to review 310051 and 310052 before jesusaur and I switch the logstash workers over to use the logstash filters project?20:59
*** kzaitsev_mb has quit IRC20:59
armaxcloudnull: so the create_server does the whole lot: boot and assign fip?20:59
*** maishsk has quit IRC21:00
anteayaclarkb: thanks21:00
clarkbarmax: correct, and it assigns the fip using the server's port id21:00
anteayaclarkb: ah I see what you are saying, sorry21:00
cloudnullarmax: in shade yes.21:00
cloudnullthough it may be able to be broken down.21:00
* cloudnull today is my first day looking at the shade internals21:00
armaxcloudnull: that’s ok21:01
*** puiterwijk is now known as puiterwijk|cld21:01
*** kzaitsev_mb has joined #openstack-infra21:01
mordredthe steps in create server are essentially "nova create server ; port list (find server port id) ; neutron floating ip create (port_id)"21:01
mordredthen poll waiting for the fip to go active, then poll the nova server until the fip shows up in the server addresses list21:01
armaxmordred: yup got it21:01
mordredcloudnull: looking at shade internals is like looking straight into a pit of evil21:02
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add private_key_file configuration for launcher
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add usernname configuration option for launcher
* clarkb doesn't hear any screaming and will move forward on approving these changes21:03
cloudnullmordred: its not _SO_ bad21:03
mordredclarkb: I support your decisions wrt ES21:03
armaxI am pulling kevinbenton in, as he’s younger and his memory is better than mine21:03
*** sdake has joined #openstack-infra21:03
mordredah youth21:03
*** yolanda has quit IRC21:04
armaxmordred: I know, right?21:04
armaxmordred: I think this race sounds mildy familiar and relatively low hanging fruit21:04
*** yolanda has joined #openstack-infra21:04
clarkbjesusaur: logstash-worker20 will be the chosen host, if you want ot watch puppet board once things merge21:04
kevinbentonso what are the API requests being made that race?21:04
cloudnull***he's going to change the world***21:04
kevinbentonthe association call right?21:05
mordredkevinbenton: create-floating-ip(port_id)21:05
anteayacloudnull: he's like that21:05
jesusaurclarkb: thanks21:05
kevinbentonmordred: ah, so a create with a specified port id up front21:05
armaxkevinbenton: so it’s association21:06
*** maishsk has joined #openstack-infra21:06
kevinbentonmordred, armax: perhaps my memory isn't as good. i thought i saw a patch about this floating around somewhere21:08
kevinbenton(pun not intended)21:08
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Add support for additional launcher config options
armaxkevinbenton: it wouldn’t be difficult to find out whether this has been fixed by looking at the code and logical model21:08
openstackgerritMatthew Thode proposed openstack/requirements: Add Gentoo to distro status
armaxkevinbenton: I am filing a bug so that we don’t forget21:09
kevinbentonarmax: yeah, it doens't look fixed21:09
openstackgerritJason Dunsmore proposed openstack/requirements: Update oslo.service to 1.9.0
*** baoli_ has quit IRC21:10
kevinbentonmordred: do they even have to be concurrent?21:10
cloudnullSo maybe its not my cloud is old and crusty :)21:11
mordredkevinbenton: we've only ever seen the problem occur when doing it concurrently21:11
mordreddoing the same thing serially does not seem to break (or at least not frequently enough for us to catch it)21:11
armaxcloudnull: you happen to see any trace in the L3 agent for this?21:11
cloudnullno i've not.21:12
*** ddieterly[away] is now known as ddieterly21:12
armaxmordred: I would be surprised if we were that broken21:12
kevinbentonmordred: but if you request a floating IP for a port that already has a floating IP, don't you get an error?21:12
pabelangermordred: clarkb: quick rax question, do we know on average how long ImageTaskCreate takes? First time I have looked into it21:12
clarkbpabelanger: when I did initial investigation forever ago a single one run without other uploads happening took 30-45 minutes21:12
clarkbpabelanger: but I have seen it take many hours21:13
mordredkevinbenton: the opposite thing is what's happening - we're requesting floating ips on two different ports and they're each getting the same floating ip21:13
*** lakshmiS has quit IRC21:13
clarkbpabelanger: its also tricky because shade will silently retry due to known errors that aren't really actionable by the user21:13
kevinbentonmordred: ooooh21:13
armaxkevinbenton, mordred: bug 158122021:13
openstackbug 1581220 in neutron "same floating ip can be associated to separate ports" [Undecided,New]
mordredkevinbenton: look at and look for f1169479-7d88-4526-a60a-82f18b61707721:13
pabelangerclarkb: Ya, we upload surprisingly fast in DFW, 9mins for 15GB21:13
pabelangerclarkb: but ya, the ImageTaskCreate is now waiting21:14
clarkbpabelanger: oh my numbers were for the total thing, but the task for image create is the one that gets retried21:14
*** ldnunes has quit IRC21:15
kevinbentonarmax, mordred: this doesn't seem possible. the floating IP datamodel has a single column for which port it's associated to21:15
*** HeOS has joined #openstack-infra21:15
kevinbentonarmax, mordred:
kevinbentonand it's been that way basically since floating IPs were introduced21:16
mordredkevinbenton: yup. each entry has a distinct value there21:16
mordredkevinbenton: but there are two fips with the same id21:16
mordredso I think it IS two different objects - but somehow two objects are getting the same uuid21:16
kevinbentonmordred: oh, sounds like DB retry logic leaked21:17
clarkbmordred: do they have the same ip addr too?21:17
mordredclarkb: no, they don't21:17
*** mixos has joined #openstack-infra21:18
mordredthey look like properly formed things in all ways except for sharing the same id21:18
openstackgerritMerged openstack-infra/puppet-logstash: Properly support logstash conf.d
*** Sukhdev has joined #openstack-infra21:18
armaxkevinbenton: what’s possible is that the last FIP association wins21:18
kevinbentonarmax: based on the output in the bug, they looks like quite separate floating IPs21:19
*** regXboi has quit IRC21:19
openstackgerritMerged openstack-infra/system-config: Use logstash filters from filters repo
kevinbentonmordred: is that listing a result of a standard floatingip-list call to Neutron directly?21:20
armaxkevinbenton: you looking at the same gist?21:21
mordredkevinbenton: not quite - it's the output of that but then run through a normalization function we have21:21
kevinbentonmordred: ack21:21
kevinbentonarmax: yeah, i'm just wondering why the messed up one would even come back from the API21:22
*** hashar has quit IRC21:22
armaxmordred: I think that from Neutron you’d only see the last FIP association21:22
armaxnova may as well get confused and see two21:22
mordredthat function takes the output of the standard neutron list floating ips call21:23
kevinbentonmordred: yeah, just ensuring it's not the proxied one through nova21:23
openstackgerritJef Mallal proposed openstack-infra/jenkins-job-builder: XUnitDotNetTestType xUnit publisher support
*** sdake_ has joined #openstack-infra21:24
mordredkevinbenton: yah - shouldn't be that one21:24
pabelangerclarkb: Ya, so increasing upload-workers will have an impact for us, because nodepool blocks that worker waiting for ImageTaskCreate to finish21:25
kevinbentonmordred: because i there were cases where that could become stale because it was based on some cache IIRC21:25
pabelangerclarkb: which is good, because we are done uploading all images about 45mins ago21:25
armaxmordred: so just to be clear you’d the the same FIP associated to two separate servers using the Nova API21:26
clarkbjesusaur: we shoul dsee if this is drop in replacement in the next 25 minutes or so21:26
mordredkevinbenton: oh yeah - we have tons of fun with stale info21:26
clarkber 20 minutes21:26
pabelangerwith that in mind, I'm happy to bump our image-workers to 16:
mordredarmax: nope. we have not used the Nova API for any of this21:26
mordredit's all neutron21:26
*** baumann has quit IRC21:26
*** thorst_ has quit IRC21:27
jesusaurclarkb: yeah, I just noticed that it merged, but wasn't picked up by the puppet run a couple minutes ago21:27
armaxmordred: ok, but say after you booted the servers and went to see the port associated to the offending FIP21:27
*** thorst_ has joined #openstack-infra21:27
mordredyah - sorry - was unclear ...21:27
armaxmordred: any trace of one association would be gone21:28
armaxmordred: if not in the server object that has the same FIP as another server21:28
mordredwe boot the server with nova. then we do the fip stuff all with neutron. then once the fip reports active we go backto nova and do a nova server get to get a refreshed copy of the server dict from it21:28
mordredarmax: yah. which would then explain why nova does not see a fip assocated with the server21:28
*** sdake has quit IRC21:28
mordredand the overall process fails, due to nova not knowing about it21:29
armaxmordred: because connection to it fails, right?21:29
*** jerryz has joined #openstack-infra21:29
mordredno - we actively report it as an error if we can't get a nova server record with the ip information attached21:29
openstackgerritJef Mallal proposed openstack-infra/jenkins-job-builder: XUnitDotNetTestType xUnit publisher support
mordredwe've been polling nova for an hour, btw, and it never shows up in these cases21:30
mordredwhich I think is consistent with what you're saying21:30
kevinbentonmordred: but it always shows up from neutron, right?21:30
armaxmordred: ok, I sense that it’s plausible under the existing code that two FIP association requests interfere, the last one wins and everything goes out the window21:30
*** sdake has joined #openstack-infra21:31
anteayaso has anyone ever seen folks introduce a question with 'long complicated question' or something similar? I ask because I see people use the phrase 'quick question' before they ask a question and it usually isn't21:31
anteayathis isn't related to anything, just wondering21:31
*** thorst_ has quit IRC21:31
mordredkevinbenton: yes21:32
*** sdake_ has quit IRC21:32
cloudnullanteaya: Real quick, I have a question :)21:32
anteayacloudnull: I bet you do21:32
anteayahow fast can you say it21:32
armaxkevinbenton: it’s possible that the two requests interleave when it’s time to check if the FIP is associated already21:32
*** thorst_ has joined #openstack-infra21:33
kevinbentonarmax: a FIP ID is a primary key so it's not dupable, so def same IP. and it's associated fixed_port_id is a single column entry21:33
armaxkevinbenton: anyway I need more time to think about this, I am rambling21:33
*** dtantsur|afk has quit IRC21:33
kevinbentonarmax: i mean it must be one FIP record21:34
*** csaba has quit IRC21:35
*** matt-borland has quit IRC21:35
*** csaba has joined #openstack-infra21:35
*** dtantsur has joined #openstack-infra21:35
*** thorst_ has quit IRC21:36
*** thorst_ has joined #openstack-infra21:36
armaxkevinbenton: right, something else must be happen where two requests get hold of the same FIP record21:36
armaxkevinbenton: I would need to take the time to go through shade to see the exact sequence of steps involved to see what’s truly happening21:37
*** ffrank has joined #openstack-infra21:37
mordredarmax, kevinbenton but it looks like two different records, no?21:37
mordredbecuae there are two different floating_ip_address values21:37
armaxthe FIP uuid is prmary key on the FIP table21:38
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add zuul launcher hosts
mordredarmax: wow21:38
kevinbentonmordred: same floating address21:38
armaxmordred: the floating_ip_address is the same21:38
kevinbentonmordred: only difference is which port is associated21:38
mordredit is the same address21:38
kevinbentonmordred: so it smells like stale data someshow21:38
*** ddieterly is now known as ddieterly[away]21:38
kevinbentonmordred: and we don't cache queries or anything like that on neutron21:39
mordredwe do cache some things - but we cache the entire list output, not individual bits21:40
mordredbut I'll also go through our stuff and make sure we're not double-writing something21:40
clarkbjesusaur: ok logstash worker20 was updated but the logstash daemon wasn't running when I logged in21:40
clarkbjesusaur: and if I restart it dies, investigating but hunch is we need to delete the old conf.d file21:41
*** ddieterly[away] has quit IRC21:41
kevinbentonmordred: you don't happen to have the neutron server debug logs for one of these runs do you?21:41
clarkbjesusaur: complaining about addr already in use possibly because we have two identical inputs now? I am going to remove that file manually and try restarting21:41
jesusaurclarkb: oh, hm21:42
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Add repo/project
clarkbjesusaur: that seemed to be it, so in your if else to handle backward compat I think we can add an ensure absent for the old file in the else block21:43
mordredkevinbenton: I do not - cloudnull might be able to help (this is running aginst the OSIC cloud)21:43
jesusaurclarkb: ya, i'll write that up now21:43
*** doug-fish has joined #openstack-infra21:43
kevinbentoncloudnull: only if it's really convenient21:43
cloudnullkevinbenton:  i dont have debug logs for it. I could go turn debug on and capture some things.21:44
cloudnullits kinda a pain, but i can make it happen if you think itll help21:44
*** kjorgensen has joined #openstack-infra21:45
*** brucet has joined #openstack-infra21:45
*** gordc has quit IRC21:45
kevinbentoncloudnull: ah, don't worry about it yet21:45
cloudnullok. you let me know though21:46
openstackgerritJames E. Blair proposed openstack-infra/zuul: Ansible launcher: Support configurable site roots for ftp/scp
*** earlephilhower has quit IRC21:46
openstackgerritMerged openstack/requirements: Add networking-generic-switch to project list
openstackgerritMerged openstack/requirements: Add python-monascaclient to the global requirements
*** bswartz has joined #openstack-infra21:48
*** doug-fish has quit IRC21:48
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-logstash: Remove old config when switching to split configs
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Exercise the launcher in beaker tests
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Add support for additional launcher config options
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Add support for zuul launcher
clarkbjesusaur: yup reviewing21:48
clarkbjesusaur: it is what I expected, I have approved it and we can have puppet run on logstash-worker19, but that should be it and we can deploy it across the board21:49
*** dprince has joined #openstack-infra21:49
*** matrohon has quit IRC21:49
*** smarcet has quit IRC21:49
*** maishsk has quit IRC21:50
jesusaurclarkb: ++21:50
openstackgerritJames E. Blair proposed openstack-infra/puppet-zuul: Install extra packages for ansible launcher
kevinbentonmordred: is this following the _neutron_create_floating_ip path in shade?21:51
openstackgerritJef Mallal proposed openstack-infra/jenkins-job-builder: XUnitDotNetTestType xUnit publisher support
kevinbentonmordred: or is it _neutron_attach_ip_to_server ?21:52
jeblairmordred, clarkb, pabelanger, fungi, Shrews, morgan, jhesketh: i think "topic:v25" is ready, and i'd like to try deploying a zuul server tomorrow [we said we would just use a jenkins server during the summit, but i think now that we are not under that pressure, a new server makes sense]21:53
*** jang has quit IRC21:53
morganMakes sense.21:54
*** earlephilhower has joined #openstack-infra21:55
pabelangerjeblair: exciting21:57
pabelangerjeblair: we should be able to use the current jenkins credentials too, that's what I did when I was testing locally.21:58
*** rkukura has quit IRC21:58
pabelangerneed to finish up the zuul-slave DIB element too21:58
jeblairpabelanger: yeah, that's the way i have staged everything; we can change them once all of jenkins is gone21:58
jeblairpabelanger: ooh, that'll be handy, thanks21:59
*** thorst_ has quit IRC21:59
*** asilenkov_ has quit IRC21:59
pabelangerjeblair: we might need libselinux-python for our centos-7 / fedora images. However, I was running into that issue when ansible tried to do yum stuff22:00
*** sigmavirus24 is now known as sigmavirus24_awa22:00
*** dmk0202 has quit IRC22:00
pabelangeror we limit our initial testing to ubuntu22:01
pabelangereither way, exciting!22:01
jeblairpabelanger: i don't follow the selinux train of thought.22:01
jeblairpabelanger: and we have no choice about what we run this on -- it's all or nothing22:01
pabelangerjeblair: let me see if I can find a log file explaining the issue22:02
*** thorst_ has joined #openstack-infra22:05
*** thorst_ has quit IRC22:05
*** thorst_ has joined #openstack-infra22:06
*** ccamacho has quit IRC22:06
*** brucet has quit IRC22:06
clarkbdid kolla ever get back to us on their potential selinux issue?22:06
*** brucet has joined #openstack-infra22:07
*** Sukhdev has quit IRC22:07
lbragstaddoes anyone know off-hand what version of mysql we're running in the gate?22:07
clarkblbragstad: its captured in the logs,22:08
clarkblbragstad: it is whatever version is available for that particular distro release22:08
*** ccamacho has joined #openstack-infra22:09
*** jsavak has quit IRC22:09
clarkbI lbragstad in the case of ubuntu trusty it is 5.5 based on a quick seach on my local machine22:09
lbragstadclarkb awesome - i assume it's in the mysql logs?22:10
lbragstadclarkb specifically looking here -
lbragstadand here -
clarkblbragstad: no, for unittests it is in the console log when we install the packages, and for devstack type things it iwll be in the devstakc log and in the dpkg.log file22:10
lbragstadclarkb perfect - looks like 5.5.49-0ubuntu0.14.04.122:11
lbragstadclarkb thanks!22:11
openstackgerritMikhail S Medvedev proposed openstack-infra/puppet-bandersnatch: Ignore "bandersnatch mirror" non-zero return status
*** dimtruck is now known as zz_dimtruck22:12
*** mixos has quit IRC22:13
*** aysyd has quit IRC22:13
*** tosky has left #openstack-infra22:14
*** vhosakot has quit IRC22:15
pabelangerjeblair: Oh, I think I fixed it by installing it into my DIB.22:15
pabelangerjeblair: but I can test it again22:15
pabelangerjeblair: this was my last error, that I have debugged yet:
*** thorst_ has quit IRC22:16
pabelangerjeblair: failure in zuul_runner task22:16
jeblairpabelanger: for selinux, i don't understand how this will be any different than what jenkins is doing today22:16
*** brucet has quit IRC22:16
jeblairpabelanger: that paste looks like a bug in the callback module22:17
pabelangerjeblair: ansible will try to setup security contexts on some files, which it need python-libselinux for22:17
pabelangerjeblair: without it, ansible bombs and says install the missing package22:17
kevinbentoncloudnull, mordred: added a couple of comments to the bug report. can you reply there?
openstackLaunchpad bug 1581220 in neutron "race leads same floating ip to be associated to separate ports" [High,Confirmed]22:17
pabelangererr libselinux-python22:18
jeblairpabelanger: interesting; it seems like ansible ought to take care of copying that over itself22:18
jeblairpabelanger: well, i guess that covers it.  :)22:19
jeblair"at least it's documented"22:19
cloudnullkevinbenton: will do22:19
pabelangerjeblair: Ya, so maybe we need a bootstrap playbook for zuul to add dependencies like this22:19
pabelangeror a way to call a ready script like we do in nodepool22:19
jeblairpabelanger: i think putting it in the dib makes sense; sounds like we should go ahead and put that in now.22:19
pabelangerjeblair: sorry, I have to run (family) but interested in continue the discussion22:19
pabelangerjeblair: Ya, that's what I did to fix it22:19
clarkbodd that ansible doesn't copy that over and add it to its run path22:20
*** brucet_ has joined #openstack-infra22:22
*** thorst_ has joined #openstack-infra22:22
*** vhosakot has joined #openstack-infra22:22
*** jpr has quit IRC22:22
*** andymaier has joined #openstack-infra22:22
openstackgerritMorgan Fainberg proposed openstack/requirements: Add pyldap to g-r
*** kzaitsev_mb has quit IRC22:24
*** lakshmiS has joined #openstack-infra22:24
*** jamielennox|away is now known as jamielennox22:24
*** ningy__ has quit IRC22:24
*** ayoung has joined #openstack-infra22:25
openstackgerritMorgan Fainberg proposed openstack/requirements: Add ldappool to projects.txt for g-r updates/checking
*** markvoelker_ has quit IRC22:25
*** thorst_ has quit IRC22:27
*** tpsilva has quit IRC22:27
openstackgerritMorgan Fainberg proposed openstack/requirements: Add pyldap to g-r
*** brucet_ has quit IRC22:30
clarkbjesusaur: we are just going to miss this puppet run, will have to be the next one22:30
jesusaurclarkb: :(22:30
openstackgerritMerged openstack-infra/puppet-logstash: Remove old config when switching to split configs
*** brucet_ has joined #openstack-infra22:30
jesusaurclarkb: did you start puppet back up already?22:31
clarkbjesusaur: not on 19, will get it once the run is properly started22:31
clarkbotherwise there is racyness between modules being updated and when the hosts lists for ansible is generated (I think we still do those as separate steps)22:32
*** weshay has quit IRC22:33
jesusauroh right, I forgot that puppet is triggered by ansible22:34
openstackgerritJames E. Blair proposed openstack-infra/project-config: Remove openstack/requirements from #-infra
jeblairfor discussion ^ :)22:34
*** mriedem has quit IRC22:34
clarkbdiscussion we should just approve :)22:34
*** mriedem has joined #openstack-infra22:34
*** mriedem has quit IRC22:35
jeblairgood discussion22:35
*** jerryz has quit IRC22:37
openstackgerritDavid Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job
*** MarkAtwood has quit IRC22:38
cloudnullkevinbenton: i've updated the LP issue with a bit more data.22:38
clarkbwe can continue to help with emergencies but I don't think we need to alert this channel of every requirements update22:38
cloudnulllet me know if you need anything else or want me to poke something with a stick22:38
clarkbjesusaur: 19 is removed fro mthe exclusion list and should get updated at the 45 minute run22:39
*** Qiming_ has joined #openstack-infra22:39
*** brucet_ has quit IRC22:39
*** kfarr has quit IRC22:41
*** amitgandhinz has quit IRC22:44
*** asselin has joined #openstack-infra22:44
openstackgerritDavid Lyle proposed openstack-infra/project-config: Update horizon dsvm tempest job
*** pahuang has joined #openstack-infra22:47
*** bpokorny_ has joined #openstack-infra22:47
*** hongbin has quit IRC22:47
*** vhosakot has quit IRC22:49
pabelangerjeblair: does this look correct for zuul-launcher playbook?
*** bpokorny has quit IRC22:51
zarojeblair: sorry, i got held up a bit this week with gerrit gc investigation22:52
kevinbentoncloudnull: but once it's in a broken state can you actually see if neutron floatingip-list returns duplicates?22:52
jeblairpabelanger: heh, wow, i've never actually looked at the output :)22:52
kevinbentoncloudnull: i want to rule out some race condition interally in shade or the neutron client as shade is using it22:52
zarojeblair: i just imported the projects to review-dev for testing.  running into a bit of an issue with the disk space.  seems like it's really low on space now.22:53
pabelangerjeblair: Ya I am thinking the !!python/unicode is not correct22:53
clarkbjesusaur: that worked, I am going to let puppet run on al of them shortly22:54
jeblairpabelanger: yeah, probably not necessary.22:54
jesusaurclarkb: \o/22:55
zaroclarkb, jeblair :what is the process to add more disk space to review-dev?22:57
*** dprince has quit IRC22:57
*** dprince has joined #openstack-infra22:57
*** jcooley has joined #openstack-infra22:58
*** thiagop has quit IRC22:59
zaroor maybe clean up files to clear up space?22:59
clarkbzaro: we don't seem to be space constrained on that host23:00
nikhilfungi: clarkb : hi, we are stuck on these patches as the #openstack-quota channel has lost the only op it had. Is there a way to get back an op for this channel or somehow fix the chanserv issue? (I did not create/op that channel)23:00
*** lakshmiS has quit IRC23:01
clarkbnikhil: if you get everyone to exit the channel then you can register it properly (assuming it isn't already registered)23:01
clarkbif it is already registered then you should talk to the person that has registered it23:01
nikhilclarkb: it's not registered. so that's the only way?23:01
clarkbif you don't have a founder present I think so23:01
zaroclarkb: i am trying to test performance of gerrit gc on nova project on review-dev.  i've got the project there and am trying to clone it locally.  when I clone i get "No space left on device" error23:01
*** sdague has quit IRC23:01
nikhilclarkb: the founder is present but he lost the op (possibly after reconnect) :)23:02
clarkbzaro: which file path, / has 13GB and /opt has 33GB23:02
zaroi'm cloning kdtest/nova-no-gc project into my home folder.23:03
zaroi blieve that's /23:03
zarohere's the error: Cloning into 'nova-no-gc'... fatal: failed to copy file to 'nova-no-gc/.git/objects/85/d6ae81b95f6baa56f41c5a13d57c4e44463d0d': No space left on device23:03
*** ayoung has quit IRC23:03
clarkbwhat command I can try to reproduce23:04
clarkbis kdtest/nova-no-gc hosted by gerrit?23:04
zaroclarkb: git clone /home/gerrit2/review_site/git/kdtest/nova-no-gc.git23:04
zaroclarkb: yes, i imported that project23:05
*** sridhar_ram has quit IRC23:05
*** sridhar_ram has joined #openstack-infra23:06
zaroi was able to clone it onto my local machine23:06
*** xyang1 has quit IRC23:06
*** Sukhdev has joined #openstack-infra23:09
*** tonytan4ever has quit IRC23:09
*** |-paul-| has joined #openstack-infra23:11
clarkbzaro: it is cloning for me so far but down to les than 7GB disk now, lets see if it will complete23:11
clarkbzaro: in any case you can probably use /opt if / doesn't have enough. We can't really expand / anyways and can only attach volumes that are mounted elsewhere23:11
clarkbzaro: it failed for me, how big is that repo?23:12
zaroclarkb: i believe each clone is 13G23:12
* clarkb runs some du23:12
clarkbdu says 7GB23:12
*** gildub has joined #openstack-infra23:12
*** asselin has quit IRC23:13
*** mjblack has joined #openstack-infra23:13
*** andymaier has quit IRC23:16
zarohmm, after clone onto my local machine du says only 672M for me.  not sure why the error happens.23:17
clarkbit may need extra tmp space to verify files or something23:17
*** HeOS has quit IRC23:18
openstackgerritMerged openstack/requirements: Bump pyghmi version to 1.0.3
openstackgerritMerged openstack/requirements: Only install unicodecsv on Python < 3.0
zaroclarkb: I guess it'll probably still fail if i attempt to clone to /opt? since /tmp is the thing that clone is having issues with?23:20
clarkbzaro: it may not use /tmp. I would try using /opt as it may use the local dir23:21
clarkbif it is /tmp being used then ya would be the same23:21
zarook. giving it a try now23:21
*** yuanying_ has joined #openstack-infra23:22
*** r-mibu has quit IRC23:23
jlvillalWhy does Gerrit warn about messages over 70 characters when the docs says 72 characters is the width to use.23:23
*** yuanying has quit IRC23:23
*** Swami has quit IRC23:24
jlvillalremote: (W) 86186d1: too many commit message lines longer than 70 characters; manually wrap lines23:24
zarojlvillal: we probably just need to adjust that.  it's just a warning correct?23:25
jlvillalzaro: Correct. It did not reject23:25
*** dingyichen has joined #openstack-infra23:25
*** markvoelker has joined #openstack-infra23:26
zarojlvillal: thanks for letting us know.  i'll make a note to either adjust the gerrit param (or the docs :)23:26
jlvillalzaro: I think 72 is a well known value.  The default that VIM uses.23:26
jlvillal80 - 8 (tab) = 7223:27
*** r-daneel has quit IRC23:27
zarojvrbanac: sounds good to me.23:28
*** bpokorny_ has quit IRC23:28
zarooops that was meant for jlvillal23:28
*** j_king has quit IRC23:29
jlvillalzaro: :)23:29
*** bpokorny has joined #openstack-infra23:29
clarkbthat bug is more than 5 years old23:29
*** j_king has joined #openstack-infra23:29
jlvillalclarkb: Thanks :)23:30
zarohaha, i asume the bug wanted 72 to be the default23:30
*** mahatic has quit IRC23:31
*** markvoelker has quit IRC23:31
*** claudiub has quit IRC23:31
*** mahatic has joined #openstack-infra23:32
clarkbjesusaur: at this point it may make sense sending mail to jj and or the ops list and say hey its a thing please to use it23:32
zaroclarkb: ok, it worked in /opt   thanks23:33
*** chlong has quit IRC23:35
*** liusheng has quit IRC23:36
*** liusheng has joined #openstack-infra23:36
jlvillalclarkb: zaro
jlvillalI'm guessing OpenStack Gerrit is running that plugin23:37
*** johnny_ has quit IRC23:38
*** cbader_ has quit IRC23:39
*** amitgandhinz has joined #openstack-infra23:45
*** r-mibu has joined #openstack-infra23:45
*** mriedem has joined #openstack-infra23:46
*** brucet_ has joined #openstack-infra23:47
*** Sukhdev has quit IRC23:48
*** amitgandhinz has quit IRC23:49
*** asettle has joined #openstack-infra23:56
*** SumitNaiksatam has quit IRC23:59

Generated by 2.14.0 by Marius Gedminas - find it at!