Wednesday, 2016-05-11

anteayanot in projects.yaml00:00
anteayawell first they have to go through jenkins job builder00:00
anteayain order for the jobs to be build in xml00:00
anteayaand to be honest I don't think I have ever asked the exact question you are asking00:00
JayFThe arrangement seems to be pseudo-arbitrary, but I'm sure there's some rhyme or reason to it00:01
anteayabut I believe that the way jenkins job builder builds the jobs it is configured to read certain kinds of information from certain yaml files00:01
JayFeither way I think I have this patch fixed, we'll see ...00:01
anteayathe other thing is that projects.yaml is very very large00:01
anteayait would be even larger if we also allowed templates to be defined inside of it too00:02
*** julim has joined #openstack-infra00:02
JayFyeah I think splitting more out of it would've made what I'm doing a lot easier00:02
*** thorst_ has quit IRC00:02
openstackgerritJay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs
anteayawell there are an exceptional amount of jobs called on the ironic repo in projects.yaml00:03
anteayamost other projects dont' have nearly that many jobs on them00:04
*** johnny___ has quit IRC00:04
anteayaI don't know why tripleo jobs are defined on the ironic repo in projects.yaml00:04
*** aeng has quit IRC00:04
*** camunoz has quit IRC00:04
anteayaI need to stop talking about having something to eat and have something to eat00:05
*** sflanigan has quit IRC00:05
anteayastepping away00:05
*** pahuang has quit IRC00:05
JayFsame, ty for the help and would appreciate reviews on ^00:06
pabelangerjeblair: I don't believe so.00:07
jeblairpabelanger: yah, i think we tracked it down -- seems we always needed 'admin' privs to delete00:07
pabelangeranteaya: JayF: yes, they should be moved into tripleo-ci project if people are discussion reordering00:08
*** MarkAtwood has joined #openstack-infra00:08
pabelangeranteaya: JayF: I can submit a follow up patch in the morning for your review if you'd like00:08
pabelangerjeblair: ack00:08
*** ffrank has quit IRC00:09
*** sridhar_ram has quit IRC00:10
krotscheckEmilienM: Sorry I wasn't around. My concern is about the actual mirror file structure on disk- AFS has a practical folder size limit of ~32K records, and if the mirror is structured to put all packages into one dir, that'll blow up on us.00:10
*** kzaitsev_mb has joined #openstack-infra00:11
*** camunoz has joined #openstack-infra00:12
pabelangerclarkb: ianw: I have deleted bare-precise from nodepool, we are ready to land
*** ddieterly has joined #openstack-infra00:16
*** grauzikas has quit IRC00:17
*** IlyaG has quit IRC00:17
openstackgerritPaul Belanger proposed openstack-infra/system-config: Remove bare-precise from nodepool-builder logging
*** kzaitsev_mb has quit IRC00:17
pabelangerand updated logging for nodepool-builder ^00:17
*** sflanigan has joined #openstack-infra00:18
*** aeng has joined #openstack-infra00:20
ianwpabelanger: ok, cool.  probably someone who can log in should do final approval.00:22
*** chlong_ has joined #openstack-infra00:22
*** chlong_ has quit IRC00:22
*** pahuang has joined #openstack-infra00:22
ianwdoes that mean we are officially off snapshots everywhere?00:22
pabelangerianw: everywhere minus tripleo00:22
pabelangerianw: however I have a patch to fix that
pabelangerhoping to land that this week00:23
pabelangerthen plan the migration away from fedora-22 next00:23
pabelangerianw: then we can close out:
*** jamesmca_ has joined #openstack-infra00:26
*** chlong_ has joined #openstack-infra00:27
*** salv-orlando has joined #openstack-infra00:28
*** chlong_ has quit IRC00:28
*** Swami has quit IRC00:29
*** jamesmca_ has quit IRC00:30
anteayapabelanger: thanks yes, I think having tripleo jobs defined on tripleo repos would help avoid future confusion00:30
*** salv-orlando has quit IRC00:33
tonybfungi: what is the process for requesting kilo-eol tags in the projects are are now EOLd?00:34
anteayahello tonyb00:36
tonybanteaya: hey there!00:37
anteayatonyb: there you are being stable again00:37
anteayaI just love that you're the stable one00:37
anteayaha ha ha00:37
tonybanteaya: I do try to hold up my end of the bargin from time to time :)00:37
anteayayou do a great job00:37
fungisdake: i'm around again for a few minutes, but protracted discussion should probably wait for tomorrow00:38
fungitonyb: first make sure there are no open changes for those branches00:38
tonybfungi: cool I can do that00:38
ianwpabelanger: 2 years, 7 months, 28 days since .00:38
*** amitgandhinz has joined #openstack-infra00:39
fungitonyb: abandon any you find with a consistent message about the branch going away00:39
fungitonyb: then push a "kilo-eol" tag to the stable/kilo branches of any projects under official stable branch management00:39
pabelangerianw: Ya, I'm pretty happy how well its been coming together over the last few weeks00:40
tonybfungi: Oh I can do that? neat.00:40
fungitonyb: or get a gerrit admin to do it. i've been doing it since we eol'd diablo i think00:40
fungiyou might not have tag permissions on them00:41
pabelangerfungi: mind doing the honors on: ? I can restart all the nodepool bits00:41
fungitonyb: anyway, he biggest help will be tracking down open changes and putting together the list of repos which need kilo-eol treatment00:41
tonybfungi: okay it looks like I don't have tag permission so I'll do the other work and then email openstack-infra with the request.00:43
fungipabelanger: no logging config adjustment needed to go along with that?00:43
tonybfungi: Thanks for putting me on the right path.00:43
fungitonyb: sounds good. i'm happy to continue doing that piece. it's easy to do a shell loop one-liner00:43
*** hichihara has joined #openstack-infra00:44
pabelangerfungi: we do!
tonybfungi: cool.00:44
fungipabelanger: cool, all lgtm. thanks for working on that!00:45
pabelangerfungi: Ya, I'll very pleased once we move tripleo-ci to a DIB00:46
*** jpr has quit IRC00:47
*** mtanino has quit IRC00:48
*** timl0101 has quit IRC00:49
*** asselin has joined #openstack-infra00:53
*** bpokorny_ has joined #openstack-infra00:56
*** madhu_ak has quit IRC00:56
*** bpokorny has quit IRC01:00
*** bpokorny_ has quit IRC01:01
*** crinkle has joined #openstack-infra01:03
*** esikachev has joined #openstack-infra01:03
anteayafound out my osteopath who got sick summer of 2014 is alive!!01:03
*** yamamot__ has joined #openstack-infra01:04
anteayafinally so good to know something about her01:04
anteayaand it is good news!!01:04
anteayahappy happy happy happy01:04
*** esikachev has quit IRC01:08
*** tphummel has quit IRC01:10
*** zhurong has joined #openstack-infra01:11
*** pvaneck has quit IRC01:11
*** kzaitsev_mb has joined #openstack-infra01:12
*** ramishra_ has joined #openstack-infra01:12
*** tqtran has quit IRC01:12
fungiguess we're going to have to stop avoiding toml soon...
dstufftwhy were you avoiding toml01:15
fungisome of us were turned off by the author basically stating that he wrote toml because he didn't feel like reading the entire yaml spec01:16
fungiwhich seemed like a terrible reason to go invent yet another serialization standard01:16
dstufftfungi: it is not the greatest reason to make yet another thing.01:18
jeblairthat pep seems to echo that meme :(01:18
*** kzaitsev_mb has quit IRC01:18
*** asselin has quit IRC01:19
jeblair"86 pages if printed on letter-sized paper"01:19
dstufftI mean, the spec is long, and full of bits people rarely use. Conceptual overhead is a thing01:19
fungii bet if i printed it double-spaced it would be even longer01:19
jeblairfungi: and in large type (though the 86 page thing did not specify type size or margins)01:19
*** mixos has joined #openstack-infra01:20
jeblairit also ends with a note about how yaml could be used in a way that addresses concerns about it.  and then ends.01:21
fungithough i confirm, having more than once tried to figure out how to sanely overload pyyaml to raise exceptions on duplicate keys, that its particular parser implementation is a bit of a bear01:21
dstufftjeblair: that was basically, I spent a few hours today ripping out the things in pyyaml that make it really hard to vendor inside of pip01:21
openstackgerritliusheng proposed openstack/requirements: bump aodhclient upper-constraints to version 0.5.0
fungiand i can understand not waning to vendor pyyaml inside pip01:21
dstufftbut like, I really really don't want to maintain a yaml parser01:21
fungier, wanting01:22
dstufftlike really really01:22
pabelangerSo, does anybody have any thoughts to the following dependency issue with pypi?01:24
jeblairdstufft: well, thanks for doing that, it seems useful.  i'm sorry to hear that it won't be used01:24
fungifor the use case in that pep, i have a feeling the toml config representation will basically look about the same as if it had just used configparser anyway01:24
pabelangerI'm guessing there is an issue with rax-iad mirror?01:24
*** kengo_sakai has joined #openstack-infra01:24
pabelangeror maybe because we are still using the read write mirror and don't have atomic updates?01:25
jeblairpabelanger: we're on read-write until lands01:25
dstufftjeblair: It's on github if anyone wants it :] ( I might finish fleshing it out with the other things I was going to change in it if I get bored01:25
jeblairdstufft: cool :)01:26
pabelangerjeblair: why yes, let me +A01:26
*** piet has joined #openstack-infra01:28
jeblairpabelanger: however, the upshot of that is that we should be even more up to date than normal01:28
dstufftFWIW I think it's a bit silly to focus on the "N pages long!!!" aspect of things, but I think it's a real concern that YAML is a fairly lengthy spec that contains a lot of things that most people will just flat out never need. Now it's true that if you don't need X feature, you don't have to use X feature, but you still tend to end up coming across people using all the little corners of it and having to know them to interact with01:29
dstufft other people's projects01:29
pabelangerjeblair: great01:29
*** amotoki has quit IRC01:30
fungidstufft: that stripped-down (i prefer using its pure python implementation anyway) and modernized pyyaml fork looks useful01:30
jeblairyeah; if we add an option to detect duplicate keys, we could have a winner :)01:31
dstufftit has some code to do that already I think01:31
dstufftit's commented out though01:31
jeblair(iirc, the pyyaml maintainers don't want to add that because it's so 'easy' to extend the parser)01:31
fungiwell, also it had something to do with the yaml spec not forbidding duplicate keys (which results in an undefined behavior, so arguably worse)01:32
*** ramishra_ has quit IRC01:32
jeblairyeah, but making it an _option_ is a good idea, methinks01:32
jeblairpabelanger: so i don't know why that error happened; satisfactory files exist... could try checking the apache logs maybe01:34
fungidstufft: wow, it really is just in there and commented out01:34
dstufftNote: I'm not 100% sure what the "Composer" file actually does01:35
dstufftthe lay out of the code is confusing01:35
dstufftbut that seems to do what you're looking for anyways01:35
fungiindeed, aside from similarly not wanting to maintain my own yaml parser ;)01:36
*** mjblack has joined #openstack-infra01:36
fungibut yeah, i'm curious to uncomment and try that out sometime when i'm not about to turn in for the night01:36
pabelangerjeblair: Ya, I might dig into it in the morning, see what happened01:36
openstackgerritMerged openstack-infra/system-config: Revert "Start using read/write AFS volume for PyPI mirror"
*** yamamot__ has quit IRC01:40
*** amotoki has joined #openstack-infra01:41
*** yamamot__ has joined #openstack-infra01:41
*** mriedem has quit IRC01:45
anteayawe have the sprint channel for infra trusty upgrade:
*** yamahata has quit IRC01:49
anteayafungi: I listed you as the event contact for that01:49
*** amotoki has quit IRC01:50
*** timl0101 has joined #openstack-infra01:51
*** tphummel has joined #openstack-infra01:51
*** timl0101 has quit IRC01:52
*** thorst_ has joined #openstack-infra01:53
*** piet has quit IRC01:54
*** anteaya has quit IRC01:54
*** timl0101 has joined #openstack-infra01:55
*** thorst__ has joined #openstack-infra01:55
*** amotoki has joined #openstack-infra01:56
*** thorst___ has joined #openstack-infra01:57
*** thorst___ has quit IRC01:57
*** thorst___ has joined #openstack-infra01:57
*** thorst_ has quit IRC01:57
EmilienMkrotscheck: maybe we need a second dir?01:58
*** salv-orlando has joined #openstack-infra01:58
EmilienMkrotscheck: is it useful if I tell how ~ gems we would have?01:58
*** thorst__ has quit IRC01:59
*** DevBox has joined #openstack-infra02:03
*** jpr has joined #openstack-infra02:04
*** salv-orlando has quit IRC02:05
*** kzaitsev_mb has joined #openstack-infra02:14
*** ddieterly is now known as ddieterly[away]02:15
*** hdd has joined #openstack-infra02:17
*** amotoki has quit IRC02:19
*** gyee has quit IRC02:20
*** kzaitsev_mb has quit IRC02:20
*** jianghuaw has quit IRC02:24
*** fitoduarte has quit IRC02:26
*** jamesmca_ has joined #openstack-infra02:28
*** tlian has quit IRC02:28
*** yamahata has joined #openstack-infra02:29
*** winggundamth has quit IRC02:29
openstackgerritLana Brindley proposed openstack-infra/irc-meetings: Updating Install Guide meeting details
openstackgerritQiming Teng proposed openstack-infra/project-config: api-ref gate for senlin
*** cody-somerville has joined #openstack-infra02:31
*** cody-somerville has quit IRC02:31
*** cody-somerville has joined #openstack-infra02:31
*** jpr has quit IRC02:32
*** amotoki has joined #openstack-infra02:32
openstackgerritQiming Teng proposed openstack-infra/project-config: api-ref gate for senlin
*** jamesmca_ has quit IRC02:32
*** dingyichen has joined #openstack-infra02:33
*** zhurong has quit IRC02:36
*** zhurong has joined #openstack-infra02:36
*** baoli has joined #openstack-infra02:38
*** yamamot__ has quit IRC02:38
*** aeng has quit IRC02:44
*** camunoz has quit IRC02:45
*** sflanigan has quit IRC02:45
*** piet has joined #openstack-infra02:45
*** tphummel has quit IRC02:46
stevemarlooks like a bunch of jobs just failed :\02:53
stevemarNo matching distribution found for oslo.messaging===5.0.002:53
stevemarNo matching distribution found for oslo.messaging>=4.5.002:54
*** aeng has joined #openstack-infra02:56
*** sflanigan has joined #openstack-infra02:57
*** camunoz has joined #openstack-infra02:57
*** apoorvad has quit IRC02:58
*** tphummel has joined #openstack-infra03:00
*** binbincong has quit IRC03:00
*** Daisy has joined #openstack-infra03:03
jamielennoxstevemar: they are all py3 jobs - and looking at it has py version == py203:04
*** ccamacho has quit IRC03:04
*** esikachev has joined #openstack-infra03:04
stevemarjamielennox: maybe?03:04
jamielennoxstevemar: it's a guess03:04
stevemarjamielennox: it was released yesterday, weird that it would start today03:05
jamielennoxbut it looks like it's not packaged for 303:05
dstufftIt's pip's fault.03:05
stevemarjamielennox: maybe
stevemarNo matching distribution found for lazr.restfulclient>=0.9.19 (from launchpadlib->releases==0.0.1.dev1488)03:05
stevemarlooks like it's not just oslo.messaging03:05
*** ccamacho has joined #openstack-infra03:05
stevemardstufft: whats the dealio?03:05
*** dimtruck is now known as zz_dimtruck03:06
*** zz_dimtruck is now known as dimtruck03:06
jamielennoxoh, then i've no idea :)03:06
*** piet has quit IRC03:07
*** Daisy has quit IRC03:08
*** esikachev has quit IRC03:09
*** binbincong has joined #openstack-infra03:12
*** amitgandhinz has quit IRC03:12
*** jamesmca_ has joined #openstack-infra03:12
dstufftstevemar: pip 8.1.2 (just released) pulls in a newer version of setuptools which fully normalizes names as per PEP 50303:13
dstufftand apparently bandersnatch (and others) don't handle that correctly when there is a . in the name03:13
dstufftit won't affect PyPI, but mirrors it will.03:14
dstufftand I think python-pip on Debian and ubuntu are going to have the same issue03:14
*** ddieterly[away] has quit IRC03:15
*** kzaitsev_mb has joined #openstack-infra03:17
*** jamesmca_ has quit IRC03:17
*** Daisy has joined #openstack-infra03:18
stevemardstufft: ouch, i guess we have to cap the pip version we use for now? hopefully someone from infra can solve this, i'm just the messenger :)03:19
dstufftI'm thinking.03:19
dstufftwe upgraded setuptools because of $ISSUES03:19
dstufft(the one we bundle in pip)03:19
dstufftbut that pulls in this change too03:19
*** yamamot__ has joined #openstack-infra03:19
dstufft(which this change is correct... I thought we already made it but apparently we missed a spot)03:20
*** jianghuaw has joined #openstack-infra03:20
jamielennoxdepends, if pip suddenly can't handle a . in a project name that will probably need to be fixed in a wider scope than us capping the version03:20
*** pahuang has quit IRC03:20
jianghuawhi all. Is this a bug with nodepool: Per the nodepool doc, floating ip pool is supported. But it seems this parameter doesn’t take effective anymore.03:20
jamielennoxoh, not pip, bandersnatch03:20
jianghuawThe ‘pool’ is generated from the configuration, but it is not used anywhere in the nodepool.
openstackgerritFei Long Wang proposed openstack-infra/irc-meetings: Propose changing Zaqar meeting time
*** links has joined #openstack-infra03:21
dstufft fixes it in bandersnatch03:21
*** hdd has quit IRC03:22
jianghuawPer my understanding: The function supporting the floating ip pool was previously in nodepool; but the relative functions had been moved to shade and then the following patch changed nodepool to use shade for all Openstack interactions. In this commit, the floating pool function was removed from nodepool but was not switched to use shade even though which supports the concept of ip_pool indeed. https://review.openstack.or03:22
*** yamamot__ has quit IRC03:22
*** kzaitsev_mb has quit IRC03:23
*** yamamot__ has joined #openstack-infra03:23
*** mjblack has quit IRC03:24
jianghuawI think we should change nodepool to pass the floating ip "pool" to shade APIs to support this documented function. Any ideas?03:24
*** salv-orlando has joined #openstack-infra03:24
*** mjblack has joined #openstack-infra03:24
*** Daisy_ has joined #openstack-infra03:24
*** yamamot__ has quit IRC03:25
*** Daisy__ has joined #openstack-infra03:25
*** signed8bit_Zzz is now known as signed8bit03:26
*** armax has quit IRC03:27
*** yamamot__ has joined #openstack-infra03:27
*** Daisy has quit IRC03:27
*** Daisy_ has quit IRC03:29
*** salv-orlando has quit IRC03:30
*** pahuang has joined #openstack-infra03:31
openstackgerritQiming Teng proposed openstack-infra/project-config: api-ref gate for senlin
*** signed8bit has quit IRC03:32
*** Daisy__ has quit IRC03:34
*** Jeffrey4l has joined #openstack-infra03:35
*** pahuang has quit IRC03:35
*** salv-orlando has joined #openstack-infra03:36
*** binbincong has quit IRC03:36
*** hichihara has quit IRC03:36
*** tphummel has quit IRC03:36
*** camunoz has quit IRC03:39
ianwdstufft: so you're saying that bandersnatch has created a broken mirror?  that's why stuff like is failing on "oslo.config" (due to .)?03:39
*** salv-orlando has quit IRC03:41
*** camunoz has joined #openstack-infra03:41
*** sree has joined #openstack-infra03:42
*** sree has joined #openstack-infra03:43
*** amitgandhinz has joined #openstack-infra03:43
*** aeng_ has joined #openstack-infra03:46
*** amitgandhinz has quit IRC03:48
*** aeng has quit IRC03:49
*** sflanigan has quit IRC03:49
*** camunoz has quit IRC03:50
*** gildub has joined #openstack-infra03:52
*** hdd has joined #openstack-infra03:53
*** fedexo has joined #openstack-infra03:55
*** binbincong has joined #openstack-infra03:59
*** hongbin has joined #openstack-infra04:00
*** hongbin has quit IRC04:01
*** mjblack has quit IRC04:01
*** baoli has quit IRC04:01
*** sflanigan has joined #openstack-infra04:02
*** pcaruana has joined #openstack-infra04:02
*** camunoz has joined #openstack-infra04:03
*** cody-somerville has quit IRC04:04
*** Qiming has quit IRC04:09
*** amitgandhinz has joined #openstack-infra04:14
*** weshay has quit IRC04:16
*** mjblack has joined #openstack-infra04:16
*** tphummel has joined #openstack-infra04:19
*** kzaitsev_mb has joined #openstack-infra04:20
*** jamesmca_ has joined #openstack-infra04:23
*** esker has joined #openstack-infra04:24
*** kzaitsev_mb has quit IRC04:26
*** esker has quit IRC04:28
*** jamesmca_ has quit IRC04:28
*** timl0101 has quit IRC04:30
*** rfolco has quit IRC04:30
*** amitgandhinz has quit IRC04:42
*** psachin has joined #openstack-infra04:42
*** yuanying has quit IRC04:44
*** Qiming has joined #openstack-infra04:46
*** amotoki_ has joined #openstack-infra04:46
*** mrunge has quit IRC04:46
*** salv-orlando has joined #openstack-infra04:47
*** claudiub has joined #openstack-infra04:48
*** amotoki has quit IRC04:48
*** markvoelker_ has joined #openstack-infra04:50
*** salv-orlando has quit IRC04:52
*** javeriak has joined #openstack-infra04:53
*** dimtruck is now known as zz_dimtruck04:57
*** zz_dimtruck is now known as dimtruck04:57
*** rcernin has joined #openstack-infra04:58
*** icey has quit IRC05:02
*** rlandy has quit IRC05:03
*** jaosorior has joined #openstack-infra05:04
openstackgerritsandhya proposed openstack/diskimage-builder: Procedure to build UEFI based images
*** fawadkhaliq has joined #openstack-infra05:05
*** esikachev has joined #openstack-infra05:05
*** dimtruck is now known as zz_dimtruck05:07
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
*** yamamot__ has quit IRC05:08
*** sdake has quit IRC05:08
*** yamamot__ has joined #openstack-infra05:09
*** kzaitsev_mb has joined #openstack-infra05:09
*** amitgandhinz has joined #openstack-infra05:09
*** esikachev has quit IRC05:10
*** yamamot__ has quit IRC05:12
*** javeriak has quit IRC05:12
*** yamamot__ has joined #openstack-infra05:13
*** savihou has joined #openstack-infra05:13
*** kzaitsev_mb has quit IRC05:14
*** fitoduarte has joined #openstack-infra05:14
*** abregman|afk is now known as abregman05:14
*** gildub has quit IRC05:22
*** roxanaghe has joined #openstack-infra05:25
*** salv-orlando has joined #openstack-infra05:28
*** javeriak has joined #openstack-infra05:31
*** shashank_hegde has joined #openstack-infra05:31
*** gomarivera has joined #openstack-infra05:33
*** camunoz has quit IRC05:33
*** hdd has quit IRC05:33
*** nadya has quit IRC05:38
*** roxanaghe has quit IRC05:39
*** javeriak has quit IRC05:41
*** burgerk has quit IRC05:41
*** fitoduarte has quit IRC05:42
*** burgerk has joined #openstack-infra05:42
*** javeriak has joined #openstack-infra05:47
*** roxanaghe has joined #openstack-infra05:48
*** nadya has joined #openstack-infra05:49
dstufftianw: sort of, the /simple/ API wasn't fully defined prior to PEP 503 and was super implenation specific. PEP 503 defines it but as part of that we had to make some decisions around which implementation's behavior we defined as standard. Bandersnatch attmpted to implement PEP 503 (via me doing it) and mistakenly missed an edge case in the normalization (that pio also happened to miss in some scenarios--- but which didn't matter05:50
dstufft because PyPI redirected) and the last N released of setuptools implemented things as PEP 503 said to do, so pip pulled that in with 8.1.2 as part of updating our bundled deps05:50
*** vhosakot has quit IRC05:50
*** yamamot__ has quit IRC05:50
dstufftianw: so there isn't a great fix here, pip could roll back the setuptools upgrade, but we'll need to do it at some point and given it only affects mirrors, and people are going to need to upgrade their bandersnatches in the next month or two ish anyways, seems reasonable to push it out that direction05:51
dstufftanyways, bed ->05:51
*** yamamot__ has joined #openstack-infra05:52
*** gomarivera has quit IRC05:52
*** nmagnezi has joined #openstack-infra05:52
*** sflanigan has quit IRC05:52
*** dchen has joined #openstack-infra05:53
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack
*** amotoki_ has quit IRC05:56
*** roxanaghe has quit IRC05:57
*** javeriak has quit IRC05:57
*** roxanaghe has joined #openstack-infra05:57
*** zz_dimtruck is now known as dimtruck05:57
*** amitgandhinz has quit IRC06:00
*** roxanaghe has quit IRC06:01
*** yuanying has joined #openstack-infra06:01
*** pahuang_ has joined #openstack-infra06:02
*** gildub has joined #openstack-infra06:04
*** tphummel has quit IRC06:04
*** dingyichen has quit IRC06:05
*** dchen has quit IRC06:05
*** dingyichen has joined #openstack-infra06:06
*** hdaniel_ has joined #openstack-infra06:07
*** piet has joined #openstack-infra06:16
*** nadya has quit IRC06:18
*** mikelk has joined #openstack-infra06:19
*** david-lyle has quit IRC06:23
*** dimtruck is now known as zz_dimtruck06:24
*** piet has quit IRC06:25
zqfanNo matching distribution found for oslo.messaging>=4.5.0, this issue blocks all CI, I think ,
*** aeng_ has quit IRC06:26
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints
*** david-lyle has joined #openstack-infra06:26
*** amitgandhinz has joined #openstack-infra06:27
zqfan it seems something wrong with the pypi mirror CI used06:29
*** zz_dimtruck is now known as dimtruck06:30
*** javeriak has joined #openstack-infra06:34
*** shashank_hegde has quit IRC06:36
*** camunoz has joined #openstack-infra06:37
*** woodster_ has quit IRC06:38
skraynevzqfan: +1 . I have the same issue, but with another requirements06:38
skraynevAJaeger: hi. could you help or redirect on right person about mentioned issue above ^06:39
*** aeng_ has joined #openstack-infra06:40
*** oanson has joined #openstack-infra06:42
rakhmerovhi, same is happening to me06:42
*** fawadkhaliq has quit IRC06:44
*** fawadkhaliq has joined #openstack-infra06:44
AJaegerskraynev, zqfan: We need an infra-root to look at this.06:45
AJaegerjhesketh, are you around to help?06:46
jheskethyep, just reading scrollback06:46
AJaegerthanks, jhesketh !06:46
skraynevAJaeger: thank you.06:46
*** fedexo has quit IRC06:47
*** dimtruck is now known as zz_dimtruck06:49
*** mixos has quit IRC06:49
*** MarkAtwood has quit IRC06:52
AJaegerjhesketh, skraynev, zqfan : ianw did some analysis:
*** vgridnev has quit IRC06:54
jheskethAJaeger: oh interesting06:54
openstackgerritQiming Teng proposed openstack-infra/project-config: api-ref gate for senlin
AJaegerjhesketh: that looks like there's no way to fix it right now ;(06:54
jheskethfwiw the oslo packages do exist in the simple mirror
openstackgerritEvgeny Antyshev proposed openstack-infra/zuul: Set keepalives for gerrit connections
AJaegerdstufft and ianw discussed this above ^06:55
*** vgridnev has joined #openstack-infra06:56
AJaegerjhesketh: at 03:21 there's a change for bandersnatch...06:57
*** andreas_s has joined #openstack-infra06:57
AJaegerSo, pinning pip to < 8.1.2 should be a temporary fix.06:58
jheskethAJaeger: are you referring to ?06:59
AJaegerjhesketh: yes - and looking at ianw's comment, it's not complete ;(06:59
jheskethhmm I think a pin is a good idea06:59
AJaegerjhesketh: backscroll starts at 2:53 with StevenK mentioning it - and then see discussion by dstufft and ianw.07:00
AJaegerjhesketh: Unfortunately I cannot help you any further on this.07:00
*** zz_dimtruck is now known as dimtruck07:01
*** salv-orlando has quit IRC07:02
AJaegerjhesketh: I suggest that you give a #status alert pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved07:02
AJaegerOr something like that...07:02
jheskeththanks, will do that as is07:03
jhesketh#status alert pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved07:03
openstackstatusjhesketh: sending alert07:03
*** amotoki has joined #openstack-infra07:04
*** liusheng has joined #openstack-infra07:04
*** matrohon has joined #openstack-infra07:04
-openstackstatus- NOTICE: pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved07:06
*** ChanServ changes topic to "pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved"07:06
*** esikachev has joined #openstack-infra07:08
*** hparekh has joined #openstack-infra07:08
openstackstatusjhesketh: finished sending alert07:08
*** rcernin has quit IRC07:09
*** acruz has joined #openstack-infra07:10
*** rcernin has joined #openstack-infra07:10
ianwjhesketh: yeah ... i think the thing is that do we have one canonical place to pin pip?07:10
jheskethianw: currently trying to figure that out... I think we might need to do it as a dib element07:10
openstackgerritsandhya proposed openstack/diskimage-builder: Add support for building images capable of UEFI
*** e0ne has joined #openstack-infra07:11
ianwjhesketh: is the first place07:12
ianwbut ISTR last time pip broke, there were a couple of other places that might overwrite it.  devstack for example07:12
ianwjhesketh: the other thing that might be done quickly is an apache rewrite rule07:13
jheskethhmm interesting thought07:14
*** markus_z has joined #openstack-infra07:14
jheskethpip will be installed on test nodes from dib, not install_puppet afaict... as you mention though devstack may do something else with it again07:14
ianwjhesketh: puppet dib calls that script, unless something has changed07:15
*** thorst___ has quit IRC07:15
AJaegerchange Id12f866f577c3ca2405a6049084c3cb0af82fde5 was what we did for system-config last time07:16
AJaegerNot sure what else was changed ;(07:16
*** thorst_ has joined #openstack-infra07:16
ianwjhesketh: i'm thinking the redirect option might work, i *think* matching everything after /simple/ and s/-/./ might work07:17
ianwohhh, unless it is supposed to have a '-' in it07:17
*** yuanying has quit IRC07:18
*** amotoki has quit IRC07:18
ianwso maybe could limit it to s/oslo-*/oslo./ ... urgh07:19
jheskethianw: It looks like puppet dib is only on centos-7 and fedora-23 unless it's pulled in elsewhere or I'm missing something... the pip-and-virtualenv element is pulled in by simple-init so I think that's where most nodes will have it from07:19
jheskethianw: we could have a condition on the apache rule to only be if the requested filename doesn't work07:20
jheskethbut if it has a mixture of .'s and -'s it'll be difficult07:20
*** dimtruck is now known as zz_dimtruck07:21
ianwhmm, yeah "if ! directory" might be good enough ... maybe the corner cases don't hit us07:21
jheskethAJaeger: that change possibly still applies, but I'm not sure if it'll be ran on the test nodes07:21
*** zeih has joined #openstack-infra07:22
ianw$ less simple | grep '-' | grep '\.' | wc -l07:22
ianwthat's packages with a "-" and a "."07:22
ianwout of 8033907:23
*** thorst_ has quit IRC07:24
*** vincentll has joined #openstack-infra07:24
*** amotoki has joined #openstack-infra07:25
*** kzaitsev_mb has joined #openstack-infra07:26
*** daemontool_ has joined #openstack-infra07:26
*** gabriel-bezerra has joined #openstack-infra07:27
*** nmagnezi has quit IRC07:30
*** markvoelker_ has quit IRC07:30
*** yaume has joined #openstack-infra07:34
*** nmagnezi has joined #openstack-infra07:34
*** yolanda has joined #openstack-infra07:37
*** ifarkas has joined #openstack-infra07:37
*** hferenc has joined #openstack-infra07:37
*** jiggalator has quit IRC07:37
hferenchi all! can i bother you with a question: ?07:38
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503
jheskethianw, AJaeger: something like ^ (note untested)07:39
*** netsin has joined #openstack-infra07:39
*** jlanoux has joined #openstack-infra07:40
AJaegerjhesketh: sorry, can't get my head around that. I love the idea but can't check the conditions ;(07:41
jheskethI think I have the logic backwards actually... we should replace -'s with .'s from the request URL as it's the -'s that are requested and teh .'s that exist07:42
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503
jheskethAJaeger: this seems to be a useful tool
jheskethusing the new url becomes
openstackgerritRyo Tagami proposed openstack-infra/jenkins-job-builder: Fix Multiple SCMs Plugin child XML format
*** kzaitsev_mb has quit IRC07:46
AJaegerjhesketh: so, that looks fine ;)07:48
jheskethI'm actually thinking it should be a passthrough...07:48
* jhesketh reads up on mod_rewrite some more07:48
*** chlong has quit IRC07:49
*** claudiub has quit IRC07:50
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503
jheskethianw: if you have a second to sanity check ^07:51
jheskethotherwise I'll apply it to a mirror manually and see how it goes07:51
*** amotoki has quit IRC07:54
*** amotoki has joined #openstack-infra07:55
*** hashar has joined #openstack-infra07:58
*** oanson has quit IRC07:58
openstackgerritSergey Gotliv proposed openstack-infra/project-config: Enable translation for manila-ui
senksomeone able to +2 this one?
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** binbincong has quit IRC08:01
*** jordanP has joined #openstack-infra08:01
*** binbincong has joined #openstack-infra08:02
openstackgerritYushiro FURUKAWA proposed openstack/requirements: Add networking-fujitsu into projects.txt
*** vincentll has quit IRC08:03
*** claudiub has joined #openstack-infra08:03
*** markus_z has quit IRC08:03
*** amotoki has quit IRC08:04
*** ifarkas has quit IRC08:04
openstackgerritChangBo Guo(gcb) proposed openstack-dev/cookiecutter: Don't include openstack/commmon in flake8 exclude list
*** weshay has joined #openstack-infra08:07
*** dmk0202 has joined #openstack-infra08:08
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quotas support
*** pavel_bondar has quit IRC08:10
*** pavel_bondar has joined #openstack-infra08:11
*** oanson has joined #openstack-infra08:11
*** zz_dimtruck is now known as dimtruck08:12
craigeThere be dragons in mod_rewrite jordanP08:14
craige^^ jhesketh08:14
*** asettle has joined #openstack-infra08:14
*** fawadkhaliq has quit IRC08:14
*** fawadkhaliq has joined #openstack-infra08:15
ianwjhesketh: LGTM, but i'm just going by reading the man page :)  i feel like that's the simplest place to start08:17
*** vincentll has joined #openstack-infra08:17
jheskethcraige: as in you found something with my rules?08:17
jheskethianw: so I just tried applying it and it appeared to have no effect... trying to figure out why08:18
ianwurggh, mod_rewrite debugging is the worst08:18
*** markus_z has joined #openstack-infra08:18
*** daemontool_ has quit IRC08:18
*** salv-orl_ has joined #openstack-infra08:19
*** ifarkas has joined #openstack-infra08:19
jheskethgoing to try with LogLevel warn rewrite:debug turned on08:20
*** daemontool_ has joined #openstack-infra08:20
*** amotoki has joined #openstack-infra08:21
*** javeriak has quit IRC08:21
*** haaj has joined #openstack-infra08:22
*** amotoki has quit IRC08:22
*** vgridnev has quit IRC08:22
jheskethsigh, I need to restart apache to up the log levels... that might make a few jobs fail08:23
*** vgridnev has joined #openstack-infra08:24
jheskethprobably better than the amount of jobs about to fail though08:24
*** fawadkhaliq has quit IRC08:25
*** fawadkhaliq has joined #openstack-infra08:25
*** amotoki has joined #openstack-infra08:26
*** dingyichen has quit IRC08:26
*** haaj has quit IRC08:29
*** yamahata has quit IRC08:29
*** Hal has joined #openstack-infra08:30
*** Hal is now known as Guest4073908:30
openstackgerritGhe Rivero proposed openstack-infra/shade: Add quotas support
*** dimtruck is now known as zz_dimtruck08:31
*** haaj has joined #openstack-infra08:31
*** patrick_k has joined #openstack-infra08:34
*** yuanying has joined #openstack-infra08:34
patrick_khi, I want to add new experimental job for nova, how can I proceed?08:34
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
jheskethianw: okay, now I'm confused... shows the rewrite rule working but I'm still getting 40408:35
rcarrillocruzpatrick_k: look for your project in project-config repo, zuul/layout.yaml08:35
rcarrillocruzadd the job in the experimental stanza08:35
*** Na3iL has joined #openstack-infra08:35
jheskethand /var/www/mirror/pypi/simple/o/oslo.config/index.html exists fyi08:35
patrick_kracrrillocruz: ok08:36
patrick_krcarrillocruz: is there any reference patch for the same?08:37
rcarrillocruzpatch reference?08:37
patrick_klike any experimental job example?08:37
*** yuanying has quit IRC08:37
ianwjhesketh: wait, it's "." to "-" ... unless i'm getting myself confused08:38
*** jistr has joined #openstack-infra08:38
jheskethianw: that's what I thought at first... but the request comes in as '-' from the new pip, and bandersnatch still mirrors with '.'s08:38
jheskethso we're converting - to .'s from the mirrors point of view08:38
*** yuanying has joined #openstack-infra08:38
rcarrillocruzwell, you could check on git log story of project-config08:39
ianwjhesketh: yeah, ok, pip has canonicalised it08:39
*** vgridnev has quit IRC08:39
*** sambetts|afk is now known as sambetts08:39
patrick_krcarrillocruz: thank you, I will have a look08:40
ianwjhesketh: ahh, hang on, it's rewriting the URL, right?  there's something else for file level IIRC08:41
jheskethianw: oh? that might be more useful08:41
*** yuanying has quit IRC08:41
ianwor maybe actually it's the PT in that rule that's doing it08:42
*** shardy has joined #openstack-infra08:43
*** yuanying has joined #openstack-infra08:43
*** rossella_s has joined #openstack-infra08:44
*** yuanying has quit IRC08:47
openstackgerritSylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project
openstackgerritSylwester Brzeczkowski proposed openstack-infra/project-config: Add fuel-extension-volume-manager project
*** javeriak has joined #openstack-infra08:48
*** markvoelker has joined #openstack-infra08:48
*** amotoki has quit IRC08:49
*** amotoki has joined #openstack-infra08:50
fricklerjhesketh: ianw: what about names like "django_compressor"? IIUC the new pip would normalize that to "django-compressor", too?08:50
*** derekh has joined #openstack-infra08:51
ianwunderscore too, yeah08:51
jheskethyeah, just did a test and it looks like it :-(08:51
openstackgerritYaroslav Lobankov proposed openstack-infra/project-config: [Rally] Adding two new non-voiting jobs for `rally verify`
*** haaj has quit IRC08:52
*** yuanying has joined #openstack-infra08:52
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
*** vincentll has quit IRC08:54
openstackgerritMerged openstack-infra/tripleo-ci: Add planet blog aggregator feeds to website
*** markvoelker has quit IRC08:55
openstackgerritGiulio Fidente proposed openstack-infra/tripleo-ci: Add Emilien Macchi blog to TripleO blogs feed
jheskethfrickler, ianw: looks like the old pkg safe_name already replaced underscores and bandersnatch handles those fine08:56
openstackgerritMerged openstack-infra/tripleo-ci: Add Emilien Macchi blog to TripleO blogs feed
*** yuanying has quit IRC08:57
*** vkramskikh_ has joined #openstack-infra08:57
ianwjhesketh: plain "RewriteRule /simple/(.*)-(.*) /simple/$1.$2" seems to work in local testing08:57
jheskethianw: but we actually need to rewrite into pypi/simple/a/alpha08:59
jheskethwhich I guess we could do as multiple rewrite rules rather than a passthrough (which is what I had first)08:59
*** HeOS has joined #openstack-infra09:02
jheskethianw: thanks! I think that works now :-)...09:04
jheskethno idea what was wrong with my passthrough rules09:04
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503
jheskethianw: with ^ applied09:05
*** markus_z has quit IRC09:05
jhesketherr, without the extra logging09:05
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Work around bandersnatch not supporting pep503
*** nijaba has quit IRC09:06
ianwjhesketh: nice, pip 8.0.2 installs oslo.config from the mirror now09:06
*** electrofelix has joined #openstack-infra09:06
openstackgerritDmytro Dovbii proposed openstack-infra/project-config: Make gate-murano-python34-db not running on stable/mitaka
*** vincentll has joined #openstack-infra09:07
*** andreas_s has quit IRC09:07
jheskethAJaeger: do you want to review 314898 before I merge09:08
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add non-voting docs job for fuel-web
jheskethyolanda, SergeyLukjanov or if either of you are around ^09:09
*** jang has joined #openstack-infra09:10
penguinolog - merge failed because of curretn topic (pip 8.1.2 broke our local python mirror, some jobs will fail with "No matching distribution found". We're investigating. Do not "recheck" until the issue is solved)...09:10
penguinologWhat should I do?09:10
*** javeriak has quit IRC09:11
jheskethpenguinolog: hold tight... we're working on a fix and will update the topic when finished09:11
yolandajhesketh, i'm not available today sorry... still at the hospital09:11
jheskethyolanda: no worries. Sorry to interrupt09:11
penguinologjhesketh, no, my question is different: am I correct, that after repair I should rebase the patch?09:12
*** nijaba has joined #openstack-infra09:13
jheskethpenguinolog: no need to rebase... after it's fixed a recheck should do it09:13
ianwpenguinolog: just a "recheck" will work09:13
*** hashar has quit IRC09:13
jheskethokay lacking any other infra-root, I'm going to merge 31489809:14
*** hashar has joined #openstack-infra09:14
ianwjhesketh: probably CI on will be a good test.  should be a precise nodes on rax we run project-config tests on09:16
*** markvoelker has joined #openstack-infra09:16
openstackgerritIsaac Beckman proposed openstack-infra/puppet-zuul: enable customize layout file name in zuul.conf
jheskethianw: yep09:17
ianwjhesketh: ahh, ran in IAD09:18
ianw might be09:19
ianwquite annoying to not have the live jenkins logs ATM09:19
jheskethI'm not sure my hot fix will stayed applied anyway... since I think puppet will reload apache when it notices the vhost is wrong09:19
openstackgerritMerged openstack-infra/system-config: Work around bandersnatch not supporting pep503
*** markus_z has joined #openstack-infra09:19
jheskethso we need to wait for the change to apply09:19
jhesketh(the alternative to disable puppet on each mirror and apply manually)09:19
jhesketh(but puppet runs are quick, so it can wait a few more minutes)09:20
openstackgerritIgor Belikov proposed openstack-infra/project-config: Add non-voting docs job for fuel-web
penguinologjhesketh, thank you! Just previously gates was not started, if previously failed by some error and recheck was used.09:20
*** markvoelker has quit IRC09:20
*** tesseract has joined #openstack-infra09:20
*** daemontool_ has quit IRC09:21
*** andreas_s has joined #openstack-infra09:21
AJaegerjhesketh: a recheck will only help once the issue is fixed. And rebase is never needed, since rebase is done as part of the job run09:21
AJaegerpenguinolog: wanted to address you ^09:22
AJaegersorry, jhesketh h09:22
*** daemontool_ has joined #openstack-infra09:22
ianwjhesketh: yeah, that also ran in IAD.  we'll soon find out :)09:22
*** zz_dimtruck is now known as dimtruck09:23
*** thorst_ has joined #openstack-infra09:26
*** zeih has quit IRC09:28
*** sree__ has joined #openstack-infra09:28
skraynevjhesketh: hi. could you please say update about current issue with oslo- ? I have seen your patch. does it enough for fix?09:29
jheskethskraynev: it worked when manually tested... just waiting for it to apply09:30
jheskethabout to run the puppet manually09:30
*** dtantsur|afk is now known as dtantsur09:30
jheskethah, it's already running09:31
*** sree has quit IRC09:32
*** fawadkhaliq has quit IRC09:32
*** fawadkhaliq has joined #openstack-infra09:32
*** dimtruck is now known as zz_dimtruck09:33
*** javeriak has joined #openstack-infra09:34
*** thorst_ has quit IRC09:35
*** amotoki has quit IRC09:35
skraynevjhesketh: ok. let's wait a merge :)09:36
jheskethit's merged... just waiting for puppet to finish09:36
jheskethlooks to be applied successfully
*** daemontool_ has quit IRC09:40
rakhmerovare we good now? Can recheck our jobs?09:40
AJaegerrakhmerov: we're testing right now...09:40
AJaegerjhesketh: worked!09:43
*** hparekh has quit IRC09:44
AJaegerrakhmerov: feel free to recheck one or two changes and tell us whether it works, please09:44
vponomaryovit works09:44
jheskethAJaeger: yay09:44
rakhmerovAJaeger: ok, thanks a lot guys!09:44
jheskethI'm also rechecking a few things before removing the notice09:44
AJaegerjhesketh: I noticed - and did the same ;)09:45
*** yaume has quit IRC09:47
*** nadya has joined #openstack-infra09:47
*** pcrews has quit IRC09:47
*** tosky has joined #openstack-infra09:47
*** zeih has joined #openstack-infra09:50
*** mescanef_ has joined #openstack-infra09:50
*** mescanef has quit IRC09:52
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Complex priorities UI in stories
AJaeger"No matching distribution found for backports.functools-lru-cache>=1.0.3"09:52
*** camunoz has quit IRC09:52
*** camunoz has joined #openstack-infra09:53
*** yamamot__ has quit IRC09:54
*** sree__ has quit IRC09:54
AJaegerand "No matching distribution found for openstack.nose-plugin>=0.7"09:54
*** zeih has quit IRC09:54
rakhmerovAJaeger: I recheck one of our things and it's working fine (
AJaegerSo, on project-config the irc-access job is still failing - and on other repos there are still failures.09:55
*** yamamot__ has joined #openstack-infra09:55
*** hparekh has joined #openstack-infra09:56
AJaegerjhesketh: is the combination of "." and "-" a problem?09:56
ianwhmm, the rewritecond should just pass it through if there's a dir though?09:57
jheskethwell we thought that might be an edge case09:57
openstackgerritMerged openstack-infra/project-config: Add non-voting docs job for fuel-web
AJaeger exists09:57
openstackgerritMerged openstack-infra/project-config: fuel-devops: Add jobs
apuimedoAJaeger: from when is merged, how long can it take to have the repos created in gerrit so we can start pushing the changes?09:58
AJaegerapuimedo: it needs a puppet run - so rule of thumb: Wait one hour after merge09:58
ianwoh, yeah, we've lost the information09:59
ianwpip is asking for
apuimedoAJaeger: do you know who I can beg for a second r+2?10:00
AJaegerapuimedo: right now nobody ;) We're firefighting  - see topic10:01
AJaegerapuimedo: somebody will review later the week for sure...10:01
*** DevBox has quit IRC10:01
AJaeger(for sure -> I hope)10:01
openstackgerritMerged openstack-infra/project-config: Make gate-murano-python34-db not running on stable/mitaka
ianw[iwienand@fedora19 tmp]$ less simple | grep 'openstack\.'10:02
ianw<a href='openstack.nose-plugin'>openstack.nose_plugin</a><br/>10:02
ianwopenstack.nose-plugin is literally the only one with a "." and "-" in it ... i wonder if it's worth just a hard-code in the regex10:02
ianw(in the openstack namespace...)10:03
ianwbackports.* has a few more10:03
AJaegerThere's backports.functools-lru-cache as well10:04
apuimedoAJaeger: good luck with the pypi mirror10:04
AJaegerif it's only the two of them - hardcode ;)10:04
*** zhurong has quit IRC10:04
AJaegerthanks, apuimedo10:04
jheskethI'm confused why it's not actually working though...10:04
*** sdague has joined #openstack-infra10:05
*** sdague has quit IRC10:05
jheskethoh, I think I see10:05
ianwyeah, i feel like this will have to be solved quickly tomorrow US time by pypa, comments building in
*** sdague has joined #openstack-infra10:05
jhesketh>>> canonicalize_name('openstack.something-else')10:05
ianwso quick hard-codes might be best10:05
jheskethso that's the name pip will request... which means it's impossible to tell where periods or dashes should be10:05
ianwjhesketh: yep; by the time it's got to apache we've lost the data10:06
ianwmatches for openstack.nose-plugin (seems to be an outlier) and backports.* might be enough10:07
openstackgerritDaniel Berrange proposed openstack/requirements: Require oslo.concurrency >= 3.8.0
jheskethianw: possibly... alternatively we could try replacing one dash at a time10:08
jheskethworking left to right10:08
jhesketh(this is all very hacky)10:08
*** Qiming has quit IRC10:09
*** signed8bit has joined #openstack-infra10:09
*** gildub has quit IRC10:09
*** yaume has joined #openstack-infra10:09
*** markvoelker has joined #openstack-infra10:10
*** zeih has joined #openstack-infra10:12
*** dteselkin has quit IRC10:13
*** stevemar has quit IRC10:13
*** asilenkov has quit IRC10:13
openstackgerritMerged openstack-infra/project-config: puppet/proposal: add missing publishers
*** markvoelker has quit IRC10:15
*** stevemar has joined #openstack-infra10:15
*** andreas_s has quit IRC10:16
*** asilenkov has joined #openstack-infra10:16
*** daemontool has joined #openstack-infra10:18
*** dteselkin has joined #openstack-infra10:18
ianwjhesketh: i think you'd have to try all permutations?10:19
jheskethianw: to be correct, yes, but to match the common pattern of 'namespace.package-name', less so10:19
jheskethso long as namespace never has a dash in it10:20
jheskethdoing one at a time only really works with a passthrough though and we didn't get that working10:21
*** claudiub has quit IRC10:21
jheskethso maybe special casing is the idea10:21
ianwyou name it, pypi has it :)10:21
jheskethI'm now curious what exactly that package is...10:22
jheskethStatus build badges for Sloth CI apps, powered by
*** haaj has joined #openstack-infra10:23
ianwjhesketh: gotta head out sorry, not that i was doing much.10:24
*** zz_dimtruck is now known as dimtruck10:24
jheskethianw: no worries.. Your help was very valuable, thank you!10:25
ianwone other thought was to stop bandersnatch, write a quick script to run over the mirror directory and symlink normalised names10:25
tonybI was about to suggest that10:25
jheskethyeah I had considered that but didn't like the idea of stopping bandersnatch10:25
ianwprobably would work, but has to keep the old names for old python, so "if normalized_name != regualar_name; ln normalized_name regular_name" i guess10:26
tonybor run s script over g-r to gernerate a reverse rewrite rule so that .../openstack.nose.plpugin get rewritten to .../openstack.nose-plugin10:26
tonyband then load that into apache10:27
* tonyb is fully aware he's missed hours of scrollback and these may have been rejected for $reasons10:27
*** esikachev has quit IRC10:27
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Allow lists of boards and worklists to be paged
jheskethtonyb: I think you'd just do everything not just g-r10:28
tonybjhesketh: Hmm okay that's a much bigger rewrite config, but it could still be generated form the 'simple' index right?10:31
*** andreas_s has joined #openstack-infra10:32
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Further fixes for bandersnatch
jheskethianw,  tonyb: ^ for specific workaround10:33
jheskethtonyb: yes I don't see why not10:33
*** hdd has joined #openstack-infra10:34
tonybjhesketh: aww man you missed sloth-ci.ext.shields-io ;P10:35
jheskethheh, I probably missed a bunch10:35
*** hdd has quit IRC10:36
*** haaj has quit IRC10:37
*** amrith is now known as _amrith_10:38
*** kmartin has quit IRC10:38
*** dimtruck is now known as zz_dimtruck10:39
AJaegerjhesketh, tonyb: the change covers the two failures I saw so far - let's see whether that one was blocking more...10:39
*** kushal has joined #openstack-infra10:39
tonybAJaeger: fair call.10:40
AJaegertonyb: I would try this now as well - but if more issues show up, we might do one of the other suggestions - but that's more involved...10:41
*** amitgandhinz has quit IRC10:42
*** markus_z has quit IRC10:43
*** vkramskikh_ has quit IRC10:43
*** jed56 has quit IRC10:44
AJaegerjhesketh: since there's no other infra-root around, do you want to self-approve 314956?10:45
jheskethokay if we're happy with this hack I'll test it manually then self-approve10:46
jheskethbut I do agree if there are more edge cases doing symlinks may be the way to go10:46
*** claudiub has joined #openstack-infra10:46
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack
*** daemontool has quit IRC10:48
AJaegerWE have at least 12 changes in the gate queue - and they're all green. So, your first change seems to have covered a large set of packages...10:49
*** ri0 has joined #openstack-infra10:50
*** zeih has quit IRC10:51
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Improve the worklist filtering functionality
*** kushal has quit IRC10:52
*** yamamot__ has quit IRC10:53
*** yamamot__ has joined #openstack-infra10:53
*** nadya has quit IRC10:54
*** zeih has joined #openstack-infra10:54
*** esikachev has joined #openstack-infra10:54
*** ri0 has quit IRC10:55
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add tempest-dsvm-ironic-pxe_ipa_tinyipa-smoke job
*** ri0 has joined #openstack-infra10:59
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Further fixes for bandersnatch
* jhesketh is struggling to get the rule to work :-s10:59
jheskethany idea what I'm missing ^10:59
*** jistr is now known as jistr|mtg11:00
*** ccamacho has quit IRC11:00
jheskethwith that applied still 404's11:00
*** winggundamth has joined #openstack-infra11:02
sdaguebackports-functools-lru-cache => backports.functools-lru-cache is what you are going for?11:04
*** markvoelker has joined #openstack-infra11:04
sdaguejhesketh: the apache access logs typically show the file it tried to read, right?11:04
jheskethsdague: yes & no11:05
*** zeih has quit IRC11:06
*** dprince has joined #openstack-infra11:06
jheskethit just shows the GET request, not the actual files it looks up11:06
*** amotoki has joined #openstack-infra11:07
jheskethI can turn on more verbose logging on the rewrite module but that requires a restart of apache (rather than reload)11:07
jheskethand shouldn't be on for long as it's very verbose11:07
*** ccamacho has joined #openstack-infra11:08
Davieyfungi: Hey.  Would you be happy to push kilo-eol tag and delete the branch?  Ta11:08
*** amitgandhinz has joined #openstack-infra11:08
*** tellesnobrega is now known as tellesnobrega_af11:08
*** markvoelker has quit IRC11:09
jheskethah, I figured it out... fix incoming11:10
*** baoli has joined #openstack-infra11:10
sdaguejhesketh: RewriteRule (.*)-(.*) $1.$2 also seems weird, and is going to by default change the last - to .11:11
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Further fixes for bandersnatch
*** Qiming has joined #openstack-infra11:12
*** baoli_ has joined #openstack-infra11:12
jheskethsdague: yeah, that's the workaround we currently have but there are some edge cases that ^ is going to address... if there are more edge cases we may undo that and symlink the new canonicalized names11:12
jheskethAJaeger: ^ 314956 applies and tests okay now FYI11:12
jheskethsdague: either way these are just workarounds until bandersnatch is fixed11:13
sdaguejhesketh: the [N] is something I'm unfamiliar with11:13
jheskethsdague: basically keep applying the rule until it can't any further11:14
jheskethsdague: it's so the (.*) aren't greedy and only replace one - as a result11:14
jheskethie, replace all -'s with .'s11:14
*** jlanoux_ has joined #openstack-infra11:14
jhesketh(as an aside I had initially tried to do this as a passthrough which would have allowed us to replace one dash at a time and avoid repeating the sub-directory rules but getting that to work was trickier than it was worth)11:15
AJaegerthanks, jhesketh11:15
*** baoli has quit IRC11:16
*** rraja has joined #openstack-infra11:16
jheskethsdague: I was about to self-approve the extra workaround unless you had any thoughts/objections?11:17
sdaguenope, seems as good as anything to me11:17
*** jlanoux has quit IRC11:18
*** mrunge has joined #openstack-infra11:20
*** rodrigods has quit IRC11:20
*** rodrigods has joined #openstack-infra11:20
sdaguejhesketh: we good to recheck?11:20
jheskethsdague: yep, most jobs are passing, just these edge cases11:20
jheskethI'll remove the notice once this merges and applies11:20
* jhesketh lines up an email also11:21
*** nadya has joined #openstack-infra11:21
*** baoli_ has quit IRC11:21
*** baoli has joined #openstack-infra11:22
*** jaosorior has quit IRC11:22
*** jaosorior has joined #openstack-infra11:22
*** icey has joined #openstack-infra11:23
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Add api-ref jobs for neutron
openstackgerritMerged openstack-infra/system-config: Further fixes for bandersnatch
*** baoli has quit IRC11:25
*** baoli has joined #openstack-infra11:25
openstackgerritCarlos Camacho proposed openstack-infra/tripleo-ci: Fix validation for stable branch
*** nadya has quit IRC11:26
dstufftianw: good catch re: bandersnatch patch11:27
dstufftI fixedorized it11:27
*** tpsilva has joined #openstack-infra11:28
dstufft(That's what I get for writing patches when I'm half asleep)11:28
*** ldnunes has joined #openstack-infra11:31
*** kushal has joined #openstack-infra11:33
openstackgerritAkihiro Motoki proposed openstack-infra/project-config: Add api-ref jobs for neutron
*** thorst_ has joined #openstack-infra11:37
ansiwenAJaeger: can I see the semi-automatic changes resulting from the requirements change?11:37
ansiwenAJaeger: or better said: how can I find it?11:37
*** rhallisey has joined #openstack-infra11:38
skraynevAJaeger: hi. looks like I can not catch Anita Kuno to re-review my patch. could you say who else can help me and have rights for approve in project-config?11:39
AJaegeransiwen: shows all reviews produced11:39
ansiwenAJaeger: thanks!11:40
AJaegeransiwen: and you can check the post file from requirements like
rcarrillocruzjust randomly saw it on twitter11:40
AJaegerskraynev: please have some patience - and do not rebase your changes! - see topic, we currently have to fix infra...11:40
AJaegerskraynev: reviews here typically take a week to merge...11:41
*** salv-orl_ has quit IRC11:43
*** ccamacho is now known as ccamacho|lunch11:43
openstackgerritSergey Gotliv proposed openstack-infra/project-config: Enable translation jobs for manila-ui
*** thorst_ has quit IRC11:44
*** |-paul-| has joined #openstack-infra11:45
*** fawadkhaliq has quit IRC11:46
*** fawadkhaliq has joined #openstack-infra11:47
skraynevAJaeger: Opps. My apologize . I have not know about such estimates. Then I will wait. thank you. I thought, that it happens quickly.11:48
*** ociuhandu has joined #openstack-infra11:49
AJaegerskraynev: sometimes it happens much faster but the one week is more realistic11:49
AJaegerjhesketh: irc-access job just passed, so one step forward ;)11:50
*** kgiusti has joined #openstack-infra11:50
jheskethyep, just watched that :-)11:50
jheskethjust waiting for 314435 to go through11:50
*** _amrith_ is now known as amrith11:51
*** nadya has joined #openstack-infra11:51
*** hdd has joined #openstack-infra11:51
*** lucasagomes is now known as lucas-hungry11:52
AJaegerhorizon-python34 passed on - so, hope it passes on 314435 as well11:53
jheskethyep, and pep8 pulled in openstack.nose-plugin 31443511:54
jheskethwhich is good enough for me :-)11:54
jhesketh#status ok We have a workaround for our mirrors to attempt to translate package names if a match isn't immediately obvious. A more complete fix is yet to come. It is now safe to 'recheck' any jobs that failed due to "No matching distribution found". Please join #openstack-infra if you discover more problems.11:55
openstackstatusjhesketh: sending ok11:55
AJaegerjhesketh: great, thanks!11:55
jheskethAJaeger: Thank you for your help :-)11:56
*** yamamot__ has quit IRC11:56
ansiwenAJaeger: and how to proceed, if there is a verify failure:
sdaguejhesketh: hmmm.... Jenkins is no longer rolling up in hideci -
sdagueany thoughts as to why?11:56
*** yamamot__ has joined #openstack-infra11:56
jheskethsdague: appear to for me11:57
jheskethtry clearing your cookies/cache?11:57
jheskethor just refreshign the page11:57
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"11:57
-openstackstatus- NOTICE: We have a workaround for our mirrors to attempt to translate package names if a match isn't immediately obvious. A more complete fix is yet to come. It is now safe to 'recheck' any jobs that failed due to "No matching distribution found". Please join #openstack-infra if you discover more problems.11:57
sdaguejhesketh: Jenkins shows up?11:57
dimsjhesketh : nice work with the rewrite rules! just catching up11:57
*** psilvad has joined #openstack-infra11:58
jheskethsdague: oh sorry, I see.. I thought you meant in the comment (with the toggle CI).. no I don't see it in the summary11:58
*** markvoelker has joined #openstack-infra11:58
sdagueright, I see the others, not jenkins11:58
jheskethsdague: ditto11:58
sdagueand this has been true for a whole bunch of changes from last night11:59
*** tosky has quit IRC11:59
ttxGerrit slowing down, might be the hundreds of people showing up to recheck changes12:00
openstackstatusjhesketh: finished sending ok12:00
*** yamamot__ has quit IRC12:01
*** yamamot__ has joined #openstack-infra12:01
*** yamamot__ has quit IRC12:01
AJaegeransiwen: if there's a Jenkins -1, always check the log file first for the cause and figure out what it is.12:01
*** salv-orlando has joined #openstack-infra12:01
AJaegeransiwen: if it was the "No matching distribution" found, add a recheck. Otherwise debug it ;)12:02
*** yamamot__ has joined #openstack-infra12:02
openstackgerritMerged openstack-infra/project-config: Add new project Higgins to openstack repo
*** markvoelker has quit IRC12:03
*** zeih has joined #openstack-infra12:03
*** javeriak has quit IRC12:03
AJaegerjhesketh: Looks great so far!12:03
*** zeih has quit IRC12:07
*** claudiub has quit IRC12:08
jheskethsdague: the javascript is parsing the comments and building the data structures correctly... I'm not sure why it isn't rendering it12:08
jhesketh*well adding it to the dom12:08
*** claudiub has joined #openstack-infra12:08
*** ddieterly has joined #openstack-infra12:10
jheskethsdague: oh I see... somehow Jenkins and Intel NFV have been rolled up into the same group...12:10
jheskethsdague: that '1 rechecks' you see on the Intel CI is actually counting Jenkins, not 2 votes from intel12:11
*** jistr|mtg is now known as jistr12:11
sdaguejhesketh: huh, any idea why?12:11
jheskethstill poking12:11
sdagueyeh, I turned on the debug parsing (so glad I left that in there)12:11
sdagueso I could see it's finding the entries12:11
*** zeih has joined #openstack-infra12:11
openstackgerritVasyl Saienko proposed openstack-infra/project-config: Add ironic cleaning job
sdaguejhesketh: so... notably, it's the only CI that we're getting a pipeline for12:13
sdagueI wonder if there is a bad fold here12:13
jheskethsdague: not sure what you mean by that?12:14
openstackgerritPranav Salunke proposed openstack-infra/irc-meetings: Updating training-labs team meeting.
jheskethsdague: oh, I see, on the parse comments12:14
jheskeththat is interesting...12:15
mordredmorning all12:15
* mordred is in an all day meeting - but lurking here becuase here is more interesting12:16
*** zeih has quit IRC12:16
sdaguejhesketh: yeh, I think that's the issue12:16
jheskethsdague: jenkins has the check pipeline too... they are also the only CI bots posting the pipeline in the format we expect12:17
sdaguejhesketh: right12:17
AJaegermordred: you missed the excitement - jhesketh has done apache rewrite voodoo to fix pip 8.1.2 ;)12:17
*** patrick_k has quit IRC12:17
mordredwhat did 8.1.2 do?12:17
mordred(also, nice)12:18
jheskethsdague: right, I think I understand the issue now... perhaps we need to group by ci_name + pipeline?12:18
sdaguejhesketh: I think so12:18
AJaegermordred: and see ianw and dstufft discussing at around 3:00 UTC12:18
jheskethsdague: I can take a look at that if you like12:18
*** amitgandhinz has quit IRC12:18
mordredAJaeger: woot12:18
sdaguejhesketh: that would be awesome12:18
sdaguemordred: fwiw, it's not just 8.1.212:18
sdagueI have an 8.1.1 system that sees similar thing12:19
*** signed8bit is now known as signed8bit_Zzz12:19
sdaguepip search oslo.vmware12:19
dstufftpip search is kinda borken12:19
dstufftmostly because PyPI search is kind of borken12:19
dstufftan unrelated kind of borken12:19
openstackgerritThierry Carrez proposed openstack-infra/release-tools: change email topic tags based on summit feedback
*** asettle has quit IRC12:20
AJaegerjhesketh: if you fix this, please look also at which was reverted - so that "wrong requirements" shows up in the list as well. Right now, they are filtered out12:20
mordredjianghuaw: I agree - what you describe is a bug and we should fix it12:20
mordredjhesketh: nice rewrite12:20
*** hdd has quit IRC12:20
jheskethmordred: it's a total hack.. (my earlier version was slightly neater using passthroughs but turned out to be trickier than it was worth)12:21
*** acruz has quit IRC12:21
*** ddieterly is now known as ddieterly[away]12:21
jheskethAJaeger: sure, I'll tackle the summary box first though12:22
AJaegerjhesketh: sure! One thing after the other ;) But first some sleep...12:22
mmedvedesdague: jhesketh : we can also ask wznoinsk to remove  "(check pipeline)" from the Intel NFV CI comment12:22
*** anteaya has joined #openstack-infra12:23
*** kashyap has left #openstack-infra12:23
sdaguemmedvede: we could, but honestly there is no reason why they should12:23
sdaguethey have a real zuul now12:23
sdagueit's a bug in the parser12:24
*** marcusvrn_ has joined #openstack-infra12:24
*** acruz has joined #openstack-infra12:24
mmedvedesdague: it is a bug in the parser. Adding pipeline to the comment is not automatic, is it?12:25
jheskethmmedvede: was about to reply exactly what sdague already said12:25
jheskethmmedvede: zuul adds the pipeline to the comment.. so it's because they have configured zuul to have a pipeline called 'check'12:26
jheskethif they called it 'my_ci_pipeline' it'd work12:26
jheskethbut as pointed out, it's a bug in the parser12:26
*** ccamacho|lunch is now known as ccamacho12:28
mmedvedeok, I wrongly assumed the pipeline was manually added12:28
mordreddstufft, jhesketh: is there a time in the future when bandersnatch will normalize things on the disk?12:29
dstufftmordred: there's a PR already12:29
dstufftI talked to Christian, he's swamped with work stuff ATM but is going to try and merge it and get a release out as quickly as he can12:29
mordredso the hack is only needed for a little bit12:30
*** akijak has joined #openstack-infra12:30
*** vincentll has quit IRC12:30
mordreddstufft: while he's at it - it would be _great_ if he'd also land the hashed-directory patch we're running in production so we could start following his releases again12:30
mordreddstufft: but I know you don't have magic powers12:30
*** zz_dimtruck is now known as dimtruck12:31
openstackgerritGraham Hayes proposed openstack/requirements: Block dnspython 1.13.0
*** openstackgerrit has quit IRC12:33
*** openstackgerrit has joined #openstack-infra12:33
*** aysyd has joined #openstack-infra12:33
zigomordred: Hi there. Could you have a look at and let me know what is wrong? I'm not sure I understand pabelanger's comments.12:34
openstackgerritGraham Hayes proposed openstack/requirements: Block dnspython 1.13.0
*** signed8bit_Zzz is now known as signed8bit12:34
mordredzigo: lookig12:34
*** thorst has joined #openstack-infra12:34
wznoinskmmedvede I feel check pipeline is a good name for our main pipeline and most likely other deployments of downstream-puppet will think the same12:35
mmedvedesdague: actually, the pipeline name in zuul result is manually configured, it is not added automatically.12:35
mordredzigo: jenkins will only upload files that are inside of its "workspace" - so you can't, in the publisher, get at things that are up a dir12:35
zigomordred: How to work around this then?12:36
mordredzigo: you'll want to copy/move the output things that need to be uploaded into the workspace at the end of the build script12:36
wznoinskit's a zuul layout config (the pipeline name) and zuul layout pipeline's description that adds the '(check pipeline)' to the comment in gerrit12:36
mordredso that they're in a location you can reference here12:36
zigoThat's messy ... :)12:36
mmedvedewznoinsk: the parser confused your CI with jenkins due to the comment having similar pattern, i.e. '(check pipeline)'. I remember you changed both your failure and success comment yesterday. I thought you would change it to remove the pipeline name.12:36
*** ayoung has quit IRC12:36
mordredzigo: yes. this will be better in zuul v3 - but for now it's what we've got :)12:36
zigoDoing this now.12:37
mordredzigo: woot!12:37
*** Liuqing has joined #openstack-infra12:37
wznoinskmmedvede we also have different pipelines, and want to distinguish comments/results from different ones ... also 'check' is a good name as well12:37
wznoinskmmedvede if it breaks things and can't be fixed in a short time on your side I can change my configuration (the comments only, the pipeline will remain 'check') for the time being12:38
*** eharney has joined #openstack-infra12:39
*** thorst has quit IRC12:39
mmedvedewznoinsk: as been said, it is still hideci parsing problem. Your CI just seemed to be the first to actually have the similar pattern to jenkins12:39
*** yamamot__ has quit IRC12:40
*** yamamoto has joined #openstack-infra12:40
wznoinskmmedvede so you ok me leaving it like this ?12:40
wznoinskbtw. ci watch ( looks to be ok tome12:40
*** dimtruck is now known as zz_dimtruck12:41
mmedvedewznoinsk: consensus is the proper action is to fix hideci parser12:42
*** mrkelley has joined #openstack-infra12:43
*** amitgandhinz has joined #openstack-infra12:44
mmedvedewznoinsk: although ciwatch also uses the same pattern to distinguish jenkins from all other CI12:44
*** ddieterly[away] is now known as ddieterly12:44
mmedvedewznoinsk: so it needs to be fixed too12:44
*** jamielennox is now known as jamielennox|away12:45
*** abregman is now known as abregman|afk12:45
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Allow 3rd party CI's to have same pipeline names
jheskethsdague, mmedvede: ^12:45
*** vincentll has joined #openstack-infra12:45
*** amitgandhinz has quit IRC12:45
*** amitgandhinz has joined #openstack-infra12:46
pabelangerzigo: jenkins will not access files in ../ only ./, so if you try to point jenkins to ../build-area, the publisher will fail.12:46
zigopabelanger: Yup, I'm fixing deb-openstack-pkg-tools first.12:46
*** asettle has joined #openstack-infra12:47
zigopabelanger: About your comment: lets use 'tarballs/{project}' to be more consitent with other projects on tarballs.o.o12:47
zigopabelanger: Then thing is, that's supposed to be a temp folder which we would later on clean-up (every week?).12:47
zigoSo I thought it made sense to have a known folder name.12:48
zigoYour thoughts?12:48
sdaguejhesketh: that seems about as good as I can imagine it12:48
zigoOr is it that it's the case for every projects anyway?12:48
*** sree has joined #openstack-infra12:48
pabelangerrcarrillocruz: Ya, see that. Should see if grafyaml works still12:48
*** lucas-hungry is now known as lucasagomes12:49
pabelangerzigo: Ya, I'm fine with tarballs/{project} for now, we need to implement the clean-up stuff too12:50
*** daemontool has joined #openstack-infra12:51
cschwedesdague: Hi! if you have a few minutes, could you have a look at and let me know what you think about it? that would be great!12:51
*** links has quit IRC12:52
*** amitgandhinz has quit IRC12:52
*** markvoelker has joined #openstack-infra12:52
*** amitgandhinz has joined #openstack-infra12:52
*** xyang has joined #openstack-infra12:53
*** daemontool has quit IRC12:53
*** timl0101 has joined #openstack-infra12:53
*** jamielennox|away is now known as jamielennox12:53
*** daemontool has joined #openstack-infra12:53
*** bdemers has joined #openstack-infra12:54
*** smarcet has joined #openstack-infra12:54
*** psilvad has quit IRC12:55
*** sdake has joined #openstack-infra12:55
*** psilvad has joined #openstack-infra12:55
zigoWas web access to jenkins<num> removed?12:56
zigo <=== Canm'12:56
zigocan't get there12:56
sdaguecschwede: cool, I've got it starred I'll circle around on it when I can dump state12:56
*** regXboi has joined #openstack-infra12:56
cschwedesdague: thx a lot!12:56
*** rfolco has joined #openstack-infra12:57
*** markvoelker has quit IRC12:57
sdaguemordred / fungi - approving this would be extremely helpful for any project where Intel NFV CI is commenting, as right now it is typically masking out Jenkins votes12:57
*** Goneri has joined #openstack-infra12:58
*** xyang1 has joined #openstack-infra12:58
*** sdake_ has joined #openstack-infra12:58
*** tlian has joined #openstack-infra12:59
pabelangerzigo: yes, we firewall it off because of a security issue12:59
pabelangerso, no more public access to jenkins UI13:00
*** aysyd has quit IRC13:00
zigoI was expecting something like that.13:00
*** psilvad has quit IRC13:00
zigoJenkins is a @#$(%&*13:00
anteayajhesketh ianw sdague thanks for putting the workaround in place13:01
*** xyang1 has quit IRC13:01
*** dtardivel has joined #openstack-infra13:01
*** ddieterly has quit IRC13:02
*** sdake has quit IRC13:02
*** zqfan has quit IRC13:03
openstackgerritThomas Goirand proposed openstack-infra/project-config: Publish debian packages after build
daemontoolyolanda, ping at your convenience if we could get +A here: New repo for the Freezer Project13:03
*** edmondsw has joined #openstack-infra13:04
*** acruz has quit IRC13:04
openstackgerritMerged openstack-infra/storyboard-webclient: Removed explicit references to node_modules/.bin
*** thorst_ has joined #openstack-infra13:05
*** kengo_sakai has quit IRC13:06
*** amotoki has quit IRC13:08
*** mrkelley has left #openstack-infra13:08
*** Nakato has quit IRC13:08
*** Nakato has joined #openstack-infra13:08
*** rlandy has joined #openstack-infra13:08
openstackgerritMerged openstack-infra/tripleo-ci: Make the reviewday config file pluggable
*** matt-borland has joined #openstack-infra13:10
smarcetfungi: morning, how are u doing? still having problems with tars generation on openstackid13:10
*** andymaier has joined #openstack-infra13:10
openstackgerritMerged openstack-infra/project-config: Delete bare-precise
anteayaI'm mostly offline today, running errands13:12
pabelangeranteaya: enjoy13:12
anteayapabelanger: thanks13:12
*** gordc has joined #openstack-infra13:13
anteayapabelanger: I talked with shamail about the wiki, he would like to help if he can be shown/told what to do, he said he would pop into channel today13:13
openstackgerritMerged openstack-infra/tripleo-ci: Ignore bash errors when posting metrics
anteayapabelanger: if he does while I'm out can you give him a bit of an orientation?13:13
clarkbsmarcet: looks like the mcrypt enablement didnt take?13:14
*** jsavak has joined #openstack-infra13:14
*** aysyd has joined #openstack-infra13:14
clarkbor rather fixed it in one test buy not the other13:14
*** alikhya has joined #openstack-infra13:15
smarcetclarkb: weird, this error just arose in these days13:15
smarceton the past we didnt have that, should be related to the moving to trusty13:16
pabelangeranteaya: sure13:16
*** mriedem has joined #openstack-infra13:16
*** pabelanger has quit IRC13:16
*** pabelanger has joined #openstack-infra13:16
anteayapabelanger: thanks, I appreciate it13:16
ansiwenAJaeger: but I don't get that, so my change is merged into requirements, which creates a lot of followup changes in all the projects. there are a couple of failures. while I'm looking at these, other people will merge more stuff into requirements... shouldn't "requirements" then be gated with all the project merges its producing?13:17
*** alikhya has quit IRC13:18
*** amitgandhinz has quit IRC13:18
smarcetclarkb: weird thing is that ran ok13:19
smarcetclarkb: not sure if there is any other job to fix13:19
*** aysyd has quit IRC13:19
*** amitgandhinz has joined #openstack-infra13:20
clarkbsmarcet: the job that failed needs to be fixed13:20
*** ffrank has joined #openstack-infra13:20
*** ri0 has quit IRC13:20
smarcetclarkb: ok sure,13:20
smarcetclarkb: its located on project config ?13:21
clarkbyes, should be in the same dir as the previous one13:21
SotKhey folks, is there a way I can find out what versions of things are in use on
smarcetok thx13:21
SotKspecifically oslo_db, sqlalchemy, and sqlalchemy-fulltext-search13:21
mordredclarkb: I just sent up two patches to swift adding godep infrastructure: and
mordredsdague: lgtm - but I'd prefer someone else look at it too13:25
openstackgerritsebastian marcet proposed openstack-infra/project-config: Fix on OpenstackID job
*** arxcruz has joined #openstack-infra13:28
*** ddieterly has joined #openstack-infra13:28
*** vincentll has quit IRC13:28
*** hamzy has quit IRC13:29
smarcetclarkb: done :)13:29
*** _ari_ has joined #openstack-infra13:29
*** hamzy has joined #openstack-infra13:29
akijakhi, I'd like to create a new channel: #openstack-pl (general disscussions, support in Polish). I've read and here I am :) Can you help?13:29
*** sree__ has joined #openstack-infra13:29
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: Add networking-midonet grafana dashboard
fungiSotK: SQLAlchemy==0.9.8, SQLAlchemy-FullText-Search==0.2, sqlalchemy-migrate==0.9.413:31
SotKfungi: thanks!13:32
*** sree has quit IRC13:32
clarkbsmarcet: approved, once that gets applied someone will need to retrigger that job or wait for new merge to happen for new tarball13:33
*** eharney has quit IRC13:33
*** zz_dimtruck is now known as dimtruck13:33
fungizigo: jeblair is pretty close on a series of zuul patches to replace the limited bits of jenkins we still use13:33
*** aysyd has joined #openstack-infra13:34
*** daemontool has quit IRC13:34
zigofungi: That's super nice. I just hope we'll be able to see build logs in real time again! :)13:34
mordredzigo: we do too13:35
jheskethAJaeger: I'm not sure what you wanted to do with 308731... The only thing I can think of is group 3/4 being too greedy. Do you have an example of what you wanted to capture with it? Perhaps the easiest way would be to apply it again (perhaps to review-dev) and then tweak it13:35
*** baumann has joined #openstack-infra13:38
*** jed56 has joined #openstack-infra13:38
*** chlong has joined #openstack-infra13:38
*** tonytan4ever has joined #openstack-infra13:38
rcarrillocruzzigo: it seems we'll be able to see those via telnet (telnet is BACK)13:39
fungiwell, plaintext tcp socket stream anyway13:39
fungii doubt it will do telopt negotiation, 8-bit escapes, et cetera13:40
*** zeih has joined #openstack-infra13:40
fungithough if it needs them, i do have some experience writing a telnet option handler state engine in pure python ;)13:40
Zararcarrillocruz: do you know anything about the current status of this? just going through tidying up old stories:!/story/15313:41
* rcarrillocruz suggests telnet as April's Fool day for zuul job logs stream13:41
rcarrillocruzZara: whoa, i totally forgot about that!13:42
pabelangerrcarrillocruz: YES13:42
rcarrillocruzi started working on a prototype that used websockets with tornado to do that13:42
rcarrillocruzi'm not even sure if still have that around13:42
*** thorst has joined #openstack-infra13:42
rcarrillocruzZara: is that subscription thing still a thing?13:42
pabelangerI had some good successes with zuulv2.5 last week. I need to loop back to it this afternoon to finish up the new zuul-slave DIB elements13:42
rcarrillocruzi honestly abandoned it, cos my employer told me to focus on other things, but given that i have more freedom now i wouldn't mind hacking on it13:43
rcarrillocruzpabelanger: :P13:43
*** jpr has joined #openstack-infra13:43
pabelangernow that telnet is back, we need to stand up bbs.openstack.org13:43
pabelangerLORD ftw13:43
SotKrcarrillocruz: it would be a cool and useful thing to have, but I don't think any work has been done on it since you last looked13:44
Zararcarrillocruz: we have subscriptions in the api but I have no idea if it's the same thing, so probably not. :D13:44
rcarrillocruzso the idea would be to subscribe to storyboard over a websocket13:44
rcarrillocruzan events stream13:44
*** thorst_ has quit IRC13:44
rcarrillocruzso clients/users could do things on triggered events13:44
*** ddieterly is now known as ddieterly[away]13:45
*** eharney has joined #openstack-infra13:45
rcarrillocruzyou should probably check with krotscheck  for that13:45
* SotK goes to find the spec he's definitely seen for this13:45
rcarrillocruzif there's interest on that, i'm cool looking at it, just let me know13:45
*** tosky has joined #openstack-infra13:46
rcarrillocruzbut i guess SB needs to stabilize and adress more common use cases13:46
rcarrillocruzthat was more of a nice to have13:46
*** zeih has quit IRC13:46
SotKaha, I think it was this:
rcarrillocruz++, that was it13:46
*** markvoelker has joined #openstack-infra13:46
ZaraI'll link it from the story so it doesn't get lost again13:47
openstackgerritMerged openstack/requirements: Bump paramiko to version 2.0
* SotK is very interested in storyboard growing such functionality13:49
*** hdaniel_ has quit IRC13:50
*** fawadkhaliq has quit IRC13:50
openstackgerritPhilip Marc Schwartz proposed openstack-infra/shade: Don't fail getting flavors if extra_specs is off
*** fawadkhaliq has joined #openstack-infra13:50
phschwartzmordred: Shrews: ^^13:51
openstackgerritMatthew Treinish proposed openstack/os-testr: Add whitelist file to ostestr docs
*** hongbin has joined #openstack-infra13:53
mordredphschwartz: thanks!13:53
*** ddieterly[away] is now known as ddieterly13:55
Zararcarrillocruz: if you want to hack on it, I'd also be happy about that. :) it might take longer for me to get to reviewing things if I'm prioritizing more general stability, but I feel like if we wait for it we'll never work on any of the nice-to-haves... so mainly depends on whether SB's ready enough to work on for that, I guess.13:55
*** markvoelker has quit IRC13:55
*** dimtruck is now known as zz_dimtruck13:56
rcarrillocruzyeah, i suggest we leave it unassigned and the SB team asses when it's sensible to get back to it? I think it may be the wiser thing, given the SB team is small enough13:57
*** daemontool has joined #openstack-infra13:58
openstackgerritMerged openstack-infra/project-config: Fix on OpenstackID job
fungii can't find today's issue reported against seuptools, pip or bandersnatch... has anyone reported this somewhere upstream yet and if so, where?13:59
openstackgerritPaul Belanger proposed openstack-infra/project-config: Create a tripleo-ci project in JJB
*** Na3iL has quit IRC13:59
*** xyang has quit IRC14:00
phschwartzmordred: as to that change, I have tested it and it lets the os_server call work from ansible so it is a good fix till we decide on better handling of policy's blocking extra vars.14:01
*** |-paul-| has quit IRC14:01
*** xyang has joined #openstack-infra14:01
*** ayoung has joined #openstack-infra14:01
*** sigmavirus24_awa is now known as sigmavirus2414:01
*** links has joined #openstack-infra14:01
fungidstufft: oh, thanks!14:01
fungii didn't think to look at open patches14:01
fungithough i finally found your discussion from earlier in scrollback where you said you submitted a patch14:02
*** salv-orl_ has joined #openstack-infra14:04
*** kushal has quit IRC14:04
*** mhickey has joined #openstack-infra14:04
*** jed56 has quit IRC14:04
*** jed56 has joined #openstack-infra14:05
*** sree has joined #openstack-infra14:05
anteayamhickey: there you are14:06
anteayahey folks I'd like you to meet martin hickey14:06
mhickeyanteaya: Howdy! :-)14:06
*** cdent has joined #openstack-infra14:06
anteayamartin has been working on neutron things14:06
anteayaand now is told he is working on elk things14:06
anteayaclarkb: when you are online I'd like you to meet mhickey14:06
*** salv-orlando has quit IRC14:06
*** mariojv has joined #openstack-infra14:06
anteayaso it will take him a bit to get oriented to elk things14:07
*** jed56 has quit IRC14:07
anteayabut I told him he doesn't have to leave openstack to do his elk things14:07
*** jed56 has joined #openstack-infra14:07
mariojvhi infra - i'm here from #openstack-ironic and i'm curious about how the openstack proposal bot works. would it be possible to have it regenerate a sample configuration for the project in the same commit as a global requirements update?14:07
anteayaas we are doing elk things here and always are grateful for more opinions on the topic, as well as help14:07
mariojvthe command is a simple tox -egenconfig14:07
anteayamhickey: pabelanger and jesusaur also work on elk things as well14:08
mhickeyanteaya: thanks.14:08
anteayamhickey: thank you14:08
dstufftfungi: no problem :]14:08
Zararcarrillocruz: sure, I can't promise it'll be a priority for me any time soon, but then priority is also affected by 'what are people able to work on (and interested in working on)?' ... since a bunch of people can say 'x is important' but that doesn't matter if nobody will resource it.14:08
anteayamhickey: hope we can keep you working on openstack things while making management happy14:08
*** sree__ has quit IRC14:08
*** jed56 has quit IRC14:08
mhickeyanteaya: sure! :D14:09
*** jed56 has joined #openstack-infra14:09
anteayamhickey: wonderful14:09
anteayamhickey: have you spent much time using our logstash.o.o implementation?14:09
*** kushal has joined #openstack-infra14:10
mhickeyclarkb, pabelanger and jesusaurL look forward to chatting14:10
*** fawadkhaliq has quit IRC14:10
mhickeyanteaya: nope; didn't know about it.14:10
*** fawadkhaliq has joined #openstack-infra14:10
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Fix error in search due to multiple `title` columns in query
fungimariojv: amrith started a thread on the -dev ml about publishing sample configs (note, embedding generated content into your source code is ugly and best avoided). dhellmann pointed out that there's now a mechanism to generate sample configuration and embed it in your sphinx docs when publishing them14:11
*** xyang has quit IRC14:11
mariojvthank you. I'll look at the mailing list thread.14:11
openstackgerritPaul Belanger proposed openstack-infra/tripleo-ci: Build and upload instack.raw image
*** Na3iL has joined #openstack-infra14:13
*** daemontool has quit IRC14:13
mariojvfungi: amrith: do either of you know the subject of the thread or have a link to it handy? if not i will search myself14:14
*** Liuqing has quit IRC14:14
*** zz_dimtruck is now known as dimtruck14:14
fungimariojv: i don't recall it off hand, no. also dhellmann may just be able to point you to the config generation sphinx extension or whatever it is if he's around14:15
*** psilvad has joined #openstack-infra14:15
*** jed56 has quit IRC14:15
*** xyang has joined #openstack-infra14:15
*** jed56 has joined #openstack-infra14:15
*** mixos has joined #openstack-infra14:15
*** fawadkhaliq has quit IRC14:15
dhellmannfungi , mariojv : have a look at and and see how they are used in nova and glance respectively14:15
*** jaosorior has quit IRC14:16
* amrith reads ...14:16
*** mixos_ has joined #openstack-infra14:16
amrithmariojv, let me look for it14:17
*** Swami has joined #openstack-infra14:17
mariojvthank you amrith14:17
*** Qiming_ has joined #openstack-infra14:17
*** Qiming has quit IRC14:18
*** jed56 has quit IRC14:18
*** zeih has joined #openstack-infra14:18
*** baumann1 has joined #openstack-infra14:18
*** jed56 has joined #openstack-infra14:18
krotscheckEmilienM: It would. Once all the NPM things land I'll put your manifest on a VM and see what it looks like after running for a while.14:18
*** rbrndt has joined #openstack-infra14:18
EmilienMkrotscheck: thanks, feel fee to take over my patch if I did something wrong.14:19
mariojvamrith: i wouldn't worry about retrieving it now; we're not using oslo's config generator yet, so we would have to have that work finished before using this i believe14:19
*** baumann has quit IRC14:19
krotscheckYou did nothing wrong. There just might be more to do :)14:19
*** Liuqing has joined #openstack-infra14:19
*** baumann has joined #openstack-infra14:19
EmilienMI'm willing to help, just let me know14:19
krotscheckAny infra-core around available to shell into mirror_update and give me a status on the output of the latest registry static update?14:20
*** mixos has quit IRC14:20
krotscheckIt should be running in a screen.14:20
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Look for a passing periodic upgrades job before moving pin
*** thorst has quit IRC14:21
*** dizquierdo has joined #openstack-infra14:21
amrithmariojv, I'm not sure what fungi refers to ...14:21
*** thorst has joined #openstack-infra14:21
amrithI did post some comments on a review where someone wanted to use oslo config generator for Trove and it didn't work very nicely for reasons that are mostly Trove specific.14:22
mariojvok - the documentation is sufficient to answer my questions14:22
openstackgerritMatt Riedemann proposed openstack-infra/elastic-recheck: Add query for ceph resize bug 1580625
openstackbug 1580625 in OpenStack Compute (nova) "volume-backed resize intermittently fails with rbd imagebackend" [High,Confirmed]
openstackgerritDougal Matthews proposed openstack/requirements: Add tripleo-common to global requirements
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only
*** baumann1 has quit IRC14:23
*** mariojv has left #openstack-infra14:23
*** rossella_s has quit IRC14:25
*** jamesmca_ has joined #openstack-infra14:25
*** thorst has quit IRC14:26
openstackgerritGraham Hayes proposed openstack/requirements: Block dnspython 1.13.0
*** sdake_ has quit IRC14:28
*** mixos_ has quit IRC14:28
*** sdake has joined #openstack-infra14:29
AJaegerjhesketh: currently if a job fails the requirements check job, it does not show up in the list of job by Jenkins. ianw wanted to fix that with 308731.14:29
*** anteaya has quit IRC14:30
*** amotoki has joined #openstack-infra14:30
*** jsavak has quit IRC14:30
fungiamrith: oops, you're right, i was thinking of nikhil's thread on sample configs14:30
fungimea culpa14:30
*** jsavak has joined #openstack-infra14:30
amrithfungi, no worries.14:31
AJaegerjhesketh: just do a change against nova that updates requirements file in such a way that the test will fail - and then check for that failure... Example:
*** daemontool has joined #openstack-infra14:31
nikhilhemanthm: is r14:31
*** jed56 has quit IRC14:31
AJaegerjhesketh: you see the -1 by jenkins but only succeeding jobs and don't see the "Incompatible requirement found"14:31
nikhilPoc on config stuff14:31
*** jed56 has joined #openstack-infra14:32
*** cdent has left #openstack-infra14:32
fungiAJaeger: is there a retry for ianw's previous attempt to fix that?14:33
AJaegeransiwen: for any change to requirements, some smoke testing is done - we cannot test all jobs on all repos ;(14:34
AJaegerfungi, I haven't seen one - just wanted to point out the problem to jhesketh since he was looking at the intel nvf problem which is in the same area...14:35
*** xyang has quit IRC14:35
*** sdake has quit IRC14:35
*** pcrews has joined #openstack-infra14:36
*** xyang1 has joined #openstack-infra14:36
*** ajmiller has quit IRC14:36
*** Liuqing_ has joined #openstack-infra14:37
*** yamahata has joined #openstack-infra14:38
*** xyang has joined #openstack-infra14:38
*** asselin has joined #openstack-infra14:38
*** Liuqing has quit IRC14:38
pabelangerAJaeger: I replied to your ML thread about bindep last night, let me know what you think when you have a moment14:39
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.log to 3.7.0
*** nadya has quit IRC14:39
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.messaging to 5.1.0
*** Liuqing_ has quit IRC14:40
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.middleware to 3.10.0
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.privsep to 1.7.0
*** Liuqing has joined #openstack-infra14:41
AJaegerpabelanger: if we install the test profile, anything that is not marked, gets installed as well.14:41
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.serialization to 2.6.0
*** mixos has joined #openstack-infra14:41
AJaegerpabelanger: so, right now if we switch to test profile, it will not make a difference for package install at all.14:41
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.utils to 3.10.0
pabelangerAJaeger: Yup, and I tested that locally too.14:41
AJaegerpabelanger: only if you add entries "package [non-test]", those would be excluded14:41
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for taskflow to 1.32.0
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for tooz to 1.36.0
AJaegerpabelanger: in the end I don't care whether we use test or not - as long as we document what we have and have a clear recommendation...14:42
pabelangerAJaeger: Right, by only point, is we have no way to test that today in the gate.  So, if people want to use [test] they are happy too. We simply cannot enforce it14:42
*** ajmiller has joined #openstack-infra14:42
*** zeih has quit IRC14:42
AJaegerpabelanger: documentation
openstackgerritJulien Danjou proposed openstack-infra/release-tools: announce: make metatag optional
AJaegerpabelanger: correct, we cannot enforce it at all, we can just document it. And if we use test like does it will just work14:43
*** jaosorior has joined #openstack-infra14:43
AJaegerpabelanger: so, I'm fine with fungi reviewing and saying how he likes it's done...14:44
pabelangerAJaeger: Yup, so I am happy to do what you have documented.14:44
AJaegerfungi, what's your opinion on this?14:44
*** daemontool has quit IRC14:45
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Fix error in search due to multiple `title` columns in query
AJaegerstevemar: with publishing of the generated config file at do you still need to have the proposal job for the in-tree file? Or can we remove job and file?14:45
*** Liuqing has quit IRC14:45
mordredpabelanger, fungi: I'd like to restart nodepool with the fip cleaning patch applied - and concerns with doing that?14:46
*** thorst has joined #openstack-infra14:46
*** jed56 has quit IRC14:47
pabelangermordred: nodepool-builder not affected?14:47
*** jed56 has joined #openstack-infra14:47
mordredpabelanger: no, it would be affected - we're in the middle of image build/upload aren't we?14:47
*** zeih has joined #openstack-infra14:48
openstackgerritSergey Kraynev proposed openstack-infra/project-config: Add ci-cd-pipeline-app-murano project to openstack
*** anteaya has joined #openstack-infra14:48
AJaegermordred: we should be finished, aren't we? We start earlier now...14:48
pabelangermordred: Ya, won't be until later tonight until it finished. since we upload 56 images now14:48
*** thorst has quit IRC14:48
mordredI'll wait then14:48
*** thorst has joined #openstack-infra14:49
*** Liuqing has joined #openstack-infra14:49
pabelangermordred: if we land stack we can increase the number of upload workers to 12. Will need to land about 3 depends-on patches14:49
sdaguecould we get a promote of this - ?14:49
*** e0ne has quit IRC14:49
sdaguewe had a new test that is cratering ceph14:49
openstackgerritBeth Elwell proposed openstack-infra/project-config: Added openstack-server-publish-jobs to ironic-ui
openstackgerritDoug Hellmann proposed openstack/requirements: update constraint for oslo.service to 1.10.0
zigopabelanger: mordred: <--- I've fixed deb-openstack-pkg-tools, so you can review this one again.14:50
*** woodster_ has joined #openstack-infra14:50
mordredpabelanger, clarkb: any specific reasons we haven't landed that series?14:50
*** thorst has quit IRC14:50
*** jamesmca_ has quit IRC14:51
*** thorst has joined #openstack-infra14:51
pabelangermordred: clarkb: think we just need to schedule it at this point14:51
mordredpabelanger: maybe let's do it when the current uploads finish14:51
*** vincentll has joined #openstack-infra14:53
*** Qiming_ has quit IRC14:53
*** links has quit IRC14:54
*** thorst has quit IRC14:55
*** daemontool has joined #openstack-infra14:55
openstackgerritBeth Elwell proposed openstack-infra/project-config: Added openstack-server-publish-jobs to ironic-ui
*** jamesmca_ has joined #openstack-infra14:56
*** jamesmca_ has quit IRC14:56
*** jaosorior has quit IRC14:57
pabelangermordred: WFM14:57
*** mtanino has joined #openstack-infra14:58
*** DevBox has joined #openstack-infra14:58
*** ddieterly has quit IRC14:58
*** claudiub has quit IRC14:59
*** ddieterly has joined #openstack-infra15:01
*** electrofelix has quit IRC15:01
*** baumann1 has joined #openstack-infra15:02
*** thorst_ has joined #openstack-infra15:03
*** sdake has joined #openstack-infra15:03
*** baumann has quit IRC15:04
*** electrofelix has joined #openstack-infra15:06
openstackgerritMerged openstack-infra/project-config: Add api-ref jobs for Heat
*** thorst_ has quit IRC15:07
dtantsurhi folks! could someone approve a change to os-loganalyze: please? 5 months passed since its first and only +2 :)15:08
*** thorst_ has joined #openstack-infra15:09
*** jamesmca_ has joined #openstack-infra15:09
*** matrohon has quit IRC15:10
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: Test stack-update to new OPM
*** sree has quit IRC15:11
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Magnum functional tests in Identity v3-only
*** Liuqing has quit IRC15:12
*** jamesmca_ has quit IRC15:13
*** jistr has quit IRC15:13
*** thorst_ has quit IRC15:13
*** r-mibu has quit IRC15:14
*** r-mibu has joined #openstack-infra15:14
sdakefungi ping re vmt15:14
betherlyHi there! have a project docs question. I have added openstack-server-publish-jobs to ironic-ui to create a link at (hopefully)
betherlywhat happens next and how will i then directly edit the docs once they have been created?15:14
sdakebetherly you create docs in your project's doc dir15:15
sdakethe yget published by a tox publish job15:15
sdakeits automagic once you have the infra in place in your project repo15:15
sdakelike to see an example?15:15
betherlysdake: that would be great please!15:15
betherlyso i have a docs directory with minimum stuff atm but  can update. will that then publish with the patch i submitted on the next repo merge?15:16
clarkbfungi: you haven't retriggered that openstackid post job have you?15:16
clarkbfungi: smarcet I can do that shortly if not15:16
clarkbsdake: the docs target is actually not used at all15:16
sdakebetherly you need that - once that works locally  your publish jobs should go to docs.oo/developer/ironic-ui15:17
sdakeclarkb shows what I know ;)15:17
AJaegerbetherly: we call 'tox -e venv python sphinx_build' in your repo and publish the content of doc/build/html to
*** esker has joined #openstack-infra15:17
*** yamamoto has quit IRC15:17
sdakeAJaeger thanks for the correction15:17
clarkbsdake: betherly covers the testing interface for python projects which includes sphinx generation15:17
*** salv-orl_ has quit IRC15:17
sdakebetherly you can test that locally by copyiing that tox.ini code15:17
sdakeclarkb ack, just tyring to help ;)15:18
*** yamamoto has joined #openstack-infra15:18
AJaegerbetherly: sory, typo, it's build_sphinx15:18
AJaegerbetherly: so, if the that tox works, you should be set...15:18
sdakebetherly also you need sphinx in requirements.txt or test-requirements, forget which15:19
betherlyok cool thanks everyone!15:19
AJaegerbetherly: let me check for you and then review the change - if we're lucky, nothing else needs t obe done15:19
*** zeih has quit IRC15:19
betherlyAJaeger: that would be amazing thanks :)15:20
*** thorst_ has joined #openstack-infra15:20
*** salv-orlando has joined #openstack-infra15:21
*** winggundamth_ has joined #openstack-infra15:21
*** zeih has joined #openstack-infra15:21
fungiAJaeger: pabelanger: seems fine to me. if projects want to annotate specific deps as test-only then this solution works fine, and if they don't want to go to the trouble of separating out test-only from other dependencies then they can just ignore it... however the tox env invocation probably needs to pass the test profile too for parity15:21
fungimordred: no objection from me for a nodepool restart15:21
*** fawadkhaliq has joined #openstack-infra15:22
fungiother than image update interruptions, as noted15:22
*** claudiub has joined #openstack-infra15:22
*** rcrit has quit IRC15:22
fungisdague: 315106 is still in check. did you want it enqueued and then promoted or were you wanting to wait for it to pass check first?15:22
fungisdake: what did you want to discuss re: vmt things?15:23
*** rcrit has joined #openstack-infra15:23
sdakefungi we had discussed once a project achieves the vmt tagging, neew released (in this case neewton of kolla) could be covered by the ossa process15:23
*** smarcet has quit IRC15:24
sdakewhich totally works for me15:24
*** jamesmca_ has joined #openstack-infra15:24
sdakemy quesiton is - we ar eadding a repo this cycle15:24
fungiclarkb: i have not retriggered post jobs for that openstackid commit since the latest job fix merged, no. feel free if you haven't already15:24
pabelangerfungi: agreed15:24
sdakefungi and it won't undergo threat analysis until the end of the cycle15:24
AJaegerbetherly: everything looks fine. go ahead and add more documentation ;)15:24
sdakefungi it may not at all, as its an experimental repository15:24
*** vhosakot has joined #openstack-infra15:24
AJaegerfungi, yes, that's why I updated my swift change to use bindep test - and also updated the documentation15:25
sdakefungi just looking for some groudnrules around that point - if we don't release a final release because we don't think its ready, must we undergo TA for that repo as well?15:25
betherlyAJaeger: thanks thats great!15:25
*** thorst_ has quit IRC15:25
AJaegerfungi, see
sdaguefungi: yes15:25
sdagueit's a skip for a test15:26
*** kmartin has joined #openstack-infra15:26
sdagueso the safety level is high15:26
*** esker has quit IRC15:26
sdagueand there is a project-config change coming to prevent that kind of issue happening in the future15:26
openstackgerritChris Denneen proposed openstack-infra/jenkins-job-builder: gitlab trigger: Support new "trigger-open-merge-request-push" options
sdaguefungi: and, until it lands, a lot of stuff is going to be failing15:26
*** esker has joined #openstack-infra15:27
sdaguefungi: like the nova change in gate slot #215:27
*** jsavak has quit IRC15:27
fungisdague: i've started the enqueue, though it's currently behind zuul burning down the current events/results. as soon as the enqueue command returns i'll do a promote on it15:28
*** tonytan_brb has joined #openstack-infra15:28
sdaguefungi: coolio, thanks15:28
*** jsavak has joined #openstack-infra15:28
*** jamesmca_ has quit IRC15:28
fungisdake: the vulnerability:managed tag is per-deliverable, so the outcome depends somewhat on whether the new repo is part of the same covered deliverable or is a new, separate deliverable in the governance sense15:29
*** jsavak has quit IRC15:30
clarkbfungi: smarcet I have run the trigger job command, waiting now for job to run15:30
sdakefungi got it, so we could have kolla be a deliverable with vmt, an dkolla-kubernetes not be15:30
openstackgerritMatt Riedemann proposed openstack-infra/project-config: Gate tempest on - gate-tempest-dsvm-full-devstack-plugin-ceph
*** jsavak has joined #openstack-infra15:30
AJaegerbetherly: since you already gate on the doc job and can preview the contents (ever clicked on the gate-ironic-ui link?), everything is fine ;)15:30
fungisdake: correct. also one could be covered starting with newton while another could be covered starting with ocata, for example15:31
sdakefantastic nes15:31
fungiyou're welcome!15:31
*** amrith is now known as _amrith_15:31
sdakethe docs indicate everything must be covered at oncei n the governance repo15:31
*** tonytan4ever has quit IRC15:31
openstackgerritThiago da Silva proposed openstack-infra/project-config: Migrating liberasurecode and pyeclib projects
*** jistr has joined #openstack-infra15:31
sdakebut maybe i misread them15:31
sdakeits been a couple weeks since i looked at them (prior to summit)15:31
AJaegerpabelanger, fungi: Since you approved the bindep test change for project-config, could you review the docs change as well, please?
*** armax has joined #openstack-infra15:32
betherlyAJaeger: oooooo! well thats exciting!15:32
fungisdake: "if some repos in a deliverable are in good enough shape to qualify, their vulnerability management could be held back by other repos in the same deliverable"
sdakegot it - same deliverable15:32
sdakeok thanks :)15:32
clarkbmhickey: the current state of things is we are running logstash 2.0, elasticsearch 1.7.5, and kibana 3.something15:32
*** madhu_ak has joined #openstack-infra15:33
fungisdake: the documentation tries to speak in terms of deliverable instead of project, because it's a deliverable tag not a project tag15:33
*** _amrith_ is now known as amrith15:33
sdakeafter 4 years of openstack, still learning my way around the system :)15:33
fungiwe all are... we all are15:33
*** jistr has quit IRC15:33
mhickeyclarkb: Hey. Ok.15:33
*** jistr has joined #openstack-infra15:33
sdakethe sad part about that is im a "expert" :)15:33
clarkbmhickey: jesusaur has picked up the work to make our openstack logstash rules generically reconsumable in
clarkbwell not just that one change, but that repo15:33
*** zeih has quit IRC15:34
sdakefungi now that everyone has recovered from summit or mostly so, we will finish the job on ta15:34
clarkbwhich reminds me I need to finish reviewing those related changes and switch our logstash daemons over to those rulesets15:34
sdakeour 4 hours of ta at summit went extremely well15:34
clarkbjesusaur: ^ any chance you will be around tomorrow? I think I will have time to devote to that tomorrow15:35
fungisdake: absolutely. also tristanC can similarly answer questions (as can gmurphy if you find a channel he's in)15:35
jesusaurclarkb: yeah, i can make time tomorrow15:35
*** ifarkas has quit IRC15:35
mhickeyclarkb: ok. so how long is infra using elk?15:35
sdakefungi cool thanks15:35
clarkbmhickey: years now. at laest 2 I think15:35
clarkbI would have to check git logs to give you an exact time period15:36
*** xyang has quit IRC15:36
mhickeyclarkb: thats ok, a while is enough15:36
dirkdims: ping?15:36
dimsdirk : pong15:36
sdaguefungi: looks like event queue has drained15:37
fungittx: when you have a moment, trying to get control of in the hands of the current vmt members. i don't mind leaving you as the team owner but one or more of us probably need to be set to administrator15:37
*** xyang has joined #openstack-infra15:37
*** mikelk has quit IRC15:37
mhickeyclarkb: what are use cases for using the stack?15:37
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove keystone-propose-config-updates
sdaguealso infra-core: - Intel NFV CI is effectively hiding Jenkins votes in Hide CI rollup, which is making reviewing code challenging.15:38
fungisdague: 315106,1 should be at the front of the "integrated" gate queue as soon as the latest set of events/results are processed15:38
openstackgerritMerged openstack-infra/elastic-recheck: Add query for ceph resize bug 1580625
openstackbug 1580625 in OpenStack Compute (nova) "volume-backed resize intermittently fails with rbd imagebackend" [High,Confirmed]
fungiand there it goes15:39
sdaguefungi: awesome, thank you15:39
jesusaurmhickey: we collect all the test output and index that, we then automate searches of known bugs to find frequencies of e.g. race conditions15:39
*** bpokorny has joined #openstack-infra15:39
*** armax has quit IRC15:40
mhickeyjesusaur: Hey. Thanks; kewl!15:40
*** vincentll has quit IRC15:40
*** rraja has quit IRC15:41
*** winggundamth_ has quit IRC15:41
*** jamesmca_ has joined #openstack-infra15:41
*** xyang has quit IRC15:41
clarkbright the idea is to make the large volume of logs generated by the test jobs more accessible15:41
fungiinfra-root: once 315015 propagates to review.o.o, the ci results box will disappear from the gerrit web-ui until someone does a `sudo touch ~gerrit2/review_site/etc/GerritSiteHeader.html` on i'll try to catch it but if you don't, that's how to generally fix it15:41
fungier, if _i_ don't15:42
*** bpokorny has quit IRC15:43
ttxfungi: added you and tristanC as admins, and removed myself from team15:43
*** bpokorny has joined #openstack-infra15:43
mhickeyclarkb: nice :-)15:43
*** ramishra has quit IRC15:44
openstackgerritMerged openstack-infra/shade: Don't fail getting flavors if extra_specs is off
openstackgerritMadhumitha Viswanathan proposed openstack-infra/jenkins-job-builder: Add partial support for maven-release-plugin
tristanCttx: thanks!15:45
*** xyang has joined #openstack-infra15:45
*** ramishra has joined #openstack-infra15:46
sdaguefungi: thanks for the review on the js fix15:46
*** rcernin has quit IRC15:47
openstackgerritMerged openstack-infra/project-config: Use "bindep test"
pabelangerfungi: ack15:48
daemontoolanyone available for a +A here it is about a new repo creation, thanks15:48
clarkbmhickey: are you looking to run one for yourself or interested in how we use it upstream or?15:48
*** sshnaidm has quit IRC15:49
*** e0ne has joined #openstack-infra15:49
*** sshnaidm has joined #openstack-infra15:49
*** andymaier has quit IRC15:50
wznoinskinfra cores... is there an official requirement for the CIs that would like to have voting rights? I hear two locations for resiliency etc but is there a checklist?15:50
openstackgerritDirk Mueller proposed openstack/requirements: Add review guideline about projects.txt updates
wznoinsksdague: aren't cores interested in jenkins +1 changes so the issue intel nfv ci masking jenkins results in table is not actually that problematic? :-)15:51
clarkbwznoinsk: are you talking third party CI? if so that is entirely up to the projects they are voting on15:52
*** armax has joined #openstack-infra15:53
clarkbwznoinsk: if talking upstream CI then yes we require cloud redundancy for voting jobs15:53
wznoinskclarkb: yes, 3rdparty, then if has gating rights requires redundancy?15:53
mhickeyclarkb: going to be working upstream on elk in the next while. anteaya mentioned it is being used in OpenStack and then put me onto you folks! :)15:54
clarkbwznoinsk: third party cannot gate15:54
fungittx: thanks!!!15:54
sdaguewznoinsk: on a jenkins +1 there are still potentially non voting jobs that failed, so being able to see that fact is important15:54
wznoinskclarkb: I know, two thoughts in one sentence didn't go well here..15:54
*** Sukhdev has joined #openstack-infra15:54
openstackgerritMerged openstack-infra/system-config: Allow 3rd party CI's to have same pipeline names
*** lakshmiS has joined #openstack-infra15:55
fungiwznoinsk: are you asking whether a third-party ci system needs redundancy before projects will let it vote +1/-1 on changes?15:55
mhickeyclarkb: so when I get ramped up, maybe I can be of help?15:55
*** xyang has quit IRC15:55
*** xyang has joined #openstack-infra15:56
*** dtantsur is now known as dtantsur|afk15:56
wznoinskfungi: I think I take it as 'no' as per clarkb, I was trying to make another (shortcut) statement that if it gates it needs redundancy (for the OS infra gate only)15:56
fungiwznoinsk: if so, i don't know of any ecosystem-wide requirement for that. however, individual projects do get to decide which third-party ci systems vote on their changes and may impose such requirements on them as they see fit15:56
wznoinskfungi: ok15:56
clarkbwznoinsk: I think our use of "gate" may be clashing here. In our use of zuul and gerrit the upstream CI is the only thing that gates. We require redundant clouds for those jobs to be voting15:57
*** armax has quit IRC15:57
*** fitoduarte has joined #openstack-infra15:57
clarkbwznoinsk: third party CI does not have this requirement at an openstack level and they do not gate. Individual projects (like nova or neutron) can allow third party CI to vote on changes and that is entirely up to them15:58
wznoinskclarkb: and I understand only your OS infra zuul can gate which is the same thing?15:58
fungiwznoinsk: keep in mind that there are plenty of third-party ci systems which are run by the primary contributors for a given project and vote only on that project's repos (because perhaps they have some fundamental dependencies which cannot be tested upstream for various reasons), so we try to provide individual projects with as much levity as we can in these decisions15:58
wznoinskfungi: clarkb: understood15:58
*** xyang1 has quit IRC15:59
*** xyang1 has joined #openstack-infra15:59
fungihowever, as clarkb mentions, third-party ci systems don't have any ability to gate (i.e., automatically prevent changes from merging). only the upstream zuul gets to control that aspect16:00
pabelangerrcarrillocruz: crinkle: yolanda: was curious if there was any update on the IP address of infracloud?  Or an etherpad tracking the progress16:00
*** sridhar_ram_ has joined #openstack-infra16:01
*** jlanoux_ has quit IRC16:01
*** jsavak has quit IRC16:01
*** jsavak has joined #openstack-infra16:02
*** fawadkhaliq has quit IRC16:02
funginote that i don't particularly care whether there's a list of ip addresses, as long as they find their way into dns16:03
fungias _that_ is the proper place for a list of ip addresses16:03
*** cbader has quit IRC16:03
*** roxanaghe has joined #openstack-infra16:03
fungiso don't feel obligated to do a bunch of unnecessary double-entry bookkeeping to track them as long as they get into dns16:04
*** ffrank has quit IRC16:05
*** ddieterly is now known as ddieterly[away]16:06
*** dmk0202 has quit IRC16:07
crinklepabelanger: as far as I know rcarrillocruz is still working on it16:07
*** nmagnezi has quit IRC16:07
*** mjblack has quit IRC16:08
jeblairReleased volume mirror.ubuntu successfully16:08
jeblairreal    128m36.625s16:08
jeblairReleased volume mirror.npm successfully16:08
jeblairreal    567m34.443s16:08
*** esker has quit IRC16:08
pabelangercrinkle: thanks for the info16:08
clarkbjeblair: those are both from scratch to new afs02.dfw?16:08
pabelangerthat seems faster16:09
jeblairso we've gone from like a week to 10 hours for a full replication of npm16:09
mordredjeblair: I think that's a good from-scratch16:10
mordredjeblair: npm is non-small16:10
*** MarkAtwood has joined #openstack-infra16:11
pabelanger also re-enabled vos release for mirror.centos too, since our initial sync is finished16:11
clarkbpabelanger: are we ready to update the ready script in nodepool to enable the centos mirror then?16:11
openstackgerritJames E. Blair proposed openstack-infra/system-config: Enable vos release for mirror.centos
jeblairclarkb, pabelanger: uncommented a line pabelanger missed, and approved ^16:13
*** abregman|afk has quit IRC16:13
fungimordred: we're able to mirror lots more isinteger() this way16:13
*** gyee has joined #openstack-infra16:13
mordredfungi: woot!16:14
jeblairkrotscheck: the clean finished; i did a catchup run, and i'm vos releasing again (i expect it to have a lot to do and to take a while)16:14
krotscheckjeblair: Awesome! I'll come back... tomorrow?16:15
*** cbader has joined #openstack-infra16:15
krotscheckjeblair: I don't think that's accurate. The second run will have skipped any tarballs that it's already downloaded.16:16
krotscheck(to the above topic)16:16
jeblair#status log all afs mirror volumes have been moved to afs01.dfw and afs02.dfw (so they are no longer in ord) to speed up vos release times.  all are in regular service using read-only replicas except for npm.16:16
openstackstatusjeblair: finished logging16:16
jeblairkrotscheck: sorry, i meant the vos release will take a while.  the catchup already finished.  took only a few mins.16:16
clarkbpabelanger: is there a change up to reduce rax-ord max-servers yet?16:16
clarkbalso the osic timeout attaching floating IP seems to be persistent, we should probably send them mail if that hasnt happened yet16:17
jeblairkrotscheck: (but the vos release has all the changes that we made over the past 10+ hours)16:17
krotscheckRight. All the json files16:17
krotscheckPlus whatever tarballs we're missing.16:17
*** ddieterly[away] is now known as ddieterly16:17
krotscheckHonestly, I don't think it's going to take that long?16:17
* krotscheck could be wrong16:17
krotscheckLet me amend that.16:18
krotscheck"All the tarballs we're missing, plus whatever json files had bad versions in them (small)"16:18
jeblairkrotscheck: yeah, though if it ended up touching files that didn't actually change (like, writing out all the json files regardless) could still take a while16:18
jeblairshould be an upper bound of 10 hours though :)16:19
clarkbcloudnull: are you around? Seeing floating IP attachment take a long time (looks like close to an hour according to my logs) before the whole thing times out and is treated as an error in OSIC16:19
clarkbcloudnull: I can get instance UUIDs though it doesn't look like shade is currently logging the floating ip (or if it is I am not finding it iwth my grep yet)16:19
krotscheckAlright, I'll pester you at 7PM :)16:20
clarkboh sweet found it with better grepping16:20
clarkbcloudnull: 2016-05-11 16:15:48,128 ERROR shade: Timeout waiting for floating IP to become active. Floating IP was created for server 055f7ffb-9e39-405e-982a-710fa462aff5 but is being deleted due to activation failure. is one example16:20
*** armax has joined #openstack-infra16:21
pabelangerjeblair: thanks16:21
clarkbmordred: ^ logs actually show it hapepning in bluebox too...16:21
pabelangerclarkb: no, not yet. I didn't know we wanted to reduce them16:21
clarkbmordred: I wonder if this is a shade issue and not necessarily a cloud issue (but only basing that on two clouds having same issue)16:22
openstackgerritMerged openstack-infra/storyboard: Fix error in search due to multiple `title` columns in query
pabelangerclarkb: I was going to test the script today16:22
clarkbpabelanger: well I am not sure we do, but we continue to hit that overquota error16:22
*** mixos has quit IRC16:22
pabelangerclarkb: Ya, same. Not sure why we are hitting it16:22
*** claudiub has quit IRC16:23
*** hashar has quit IRC16:24
fungiinfra-root: just to follow up, it looks like our puppeted automation to touch GerritSiteHeader.html on a delay after hideci.js updates worked this time. maybe we got lucky? or maybe this is more robust in 2.1116:24
clarkbmordred: there are indeed recent changes to shade in this area...16:24
clarkbmordred: /me is rereviewing to see if maybe this is the problem16:25
clarkbpreviously we would leak the fip16:25
clarkbnow we are bit a more explicit about things16:25
clarkbok, so back to maybe clouds are broken (where clouds may also == neutron since mutiple clouds)16:27
*** thorst_ has joined #openstack-infra16:27
*** baumann1 has quit IRC16:28
krotscheckjeblair: I just ran `npm install` against horizon root with the new mirrors and it workred.16:28
krotscheckWell, with
pabelangerclarkb: so, this looks to be our default CentOS-Base.repo:
jeblairkrotscheck: yeah, the read-only ones are are at the state we were in right before running clean, so i would expect most things to work16:30
*** apoorvad has joined #openstack-infra16:30
pabelangerclarkb: we are currently missing extras in rsync16:31
krotscheckjeblair: But they didn't until the cleaned json files went in.16:31
*** apoorvad has quit IRC16:31
*** dizquierdo is now known as dizquierdo_afk16:31
*** thorst_ has quit IRC16:31
jeblairkrotscheck: i thought you said most json files were correct?16:31
krotscheckjeblair: They were, but horizon had bad dependencies which prevented it from working,.16:32
fungiper the -dev ml thread on "HW upgrade for tripleo CI" it sounds like we may need to keep an eye on nodepool in case we end up having to disable them in configuration16:32
fungiderekh: i take it from your e-mails that you're still on target to have it online again tomorrow?16:33
fungithe check-tripleo pipeline is at a ~5 hour backup right now with only 37 changes, but i expect that will grow quite a bit over the next day16:34
derekhfungi: yes that the plan, we should be back up and running some time tonight, the dcops guys are currently adding HW, once their done we should have everything back about 2 hours after16:34
fungiderekh: awesome. let us know if you hit any major issues that could cause significant delays to coming back online16:34
derekhfungi: will do16:35
*** mhickey has quit IRC16:35
*** tosky has quit IRC16:36
*** apoorvad has joined #openstack-infra16:36
*** tosky has joined #openstack-infra16:36
rcarrillocruzhey folks, any reason why hasn't gone thru the gate? it got workflow, but i don't see it on the zuul page16:37
rcarrillocruzand it has no dependencies16:37
*** baumann has joined #openstack-infra16:38
*** tphummel has joined #openstack-infra16:39
*** abregman|afk has joined #openstack-infra16:39
*** esikachev has quit IRC16:40
*** tellesnobrega_af is now known as tellesnobrega16:41
clarkbpabelanger: we can just update the rsync command and resync right?16:42
pabelangerclarkb: yup, just checking that now16:43
*** Jeffrey4l has quit IRC16:43
*** hashar has joined #openstack-infra16:43
clarkbrcarrillocruz: it needs +1 jenkins, try rechecking it16:43
*** mjblack has joined #openstack-infra16:44
rcarrillocruzoh...probably there was a glitch when i pushed it, that's from last week...16:44
clarkbfungi: still waiting for the openstackid-release-branch job to run but it is triggered16:44
fungigate-tempest-dsvm-neutron-identity-v3-only-full sure seems to take a while16:46
openstackgerritLucas Alvares Gomes proposed openstack/requirements: Bump pyghmi version to 1.0.3
clarkbfungi: and of course as soon as I say that it is done16:47
clarkbsmarcet is no longer here though16:47
clarkbfungi: does puppet automatically pick up that new tarball or do we need to approve a change to deploy it to openstackid-dev?16:47
fungidid we get an updated branch tarball this time?16:47
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints
krotscheckIs configure_mirrors run on the release slave?16:48
fungiyeah, puppet retrieves and autodeploys the latest tarball each time it runs16:48
*** hashar has quit IRC16:48
openstackgerritPaul Belanger proposed openstack-infra/project-config: Add CentOS support to
clarkbfungi: yes there are two new tarballs listed16:48
pabelangerclarkb: that should work ^16:48
pabelangerclarkb: want to test it on a node first16:48
clarkbfungi: see;O=D16:48
fungikrotscheck: the long-lived job workers aren't built by nodepool and so don't get nodepool ready scripts run on them. they need their configuration managed by puppet where required16:49
pabelangerclarkb: holding a node now16:49
clarkbpabelanger: we need extras first though?16:49
pabelangerclarkb: Ya, doing that now16:49
pabelangerclarkb: will also add a Depends-On16:49
krotscheckfungi: Oh good, that makes my life easier.16:50
*** claudiub has joined #openstack-infra16:50
*** Swami has quit IRC16:50
openstackgerritLucas Alvares Gomes proposed openstack/requirements: Bump pyghmi version to 1.0.3
openstackgerritPaul Belanger proposed openstack-infra/system-config: Also mirror extras for
pabelangerclarkb: once we land that, I'll grab the lock and manually rsync / vos release16:51
fungisdague: looks like that tempest skip just merged16:52
clarkbwe also have the occasional no valid host found in ovh-gra1, which I think means the nova scheduler couldn't place that instance16:52
*** abregman|afk has quit IRC16:52
clarkbpabelanger: ok16:52
*** amotoki has quit IRC16:52
*** fawadkhaliq has joined #openstack-infra16:53
openstackgerritDan Smith proposed openstack/requirements: Update oslo.versonedobjects to 1.9.1 and block 1.9.0
*** thorst_ has joined #openstack-infra16:53
jeblairpabelanger: how large is extras?16:54
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Configure npm mirror for nodepool slaves.
pabelangerjeblair: just checking that now16:54
*** asettle has quit IRC16:55
*** derekh has quit IRC16:55
openstackgerritvenkatamahesh proposed openstack/requirements: Remove kolla-mesos from projects.txt
*** javeriak has joined #openstack-infra16:56
*** baumann has quit IRC16:56
mordredclarkb: good! so we're actually doing the log and delete properly16:56
clarkbmordred: I think we are16:56
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Magnum functional tests in Identity v3-only
*** claudiub has quit IRC16:57
*** mat128 is now known as mat128|lunch16:57
*** psachin has quit IRC16:58
clarkbmordred: we are functionally broken in better ways now \o/16:58
*** thorst_ has quit IRC16:58
mordredclarkb: yay!16:58
mordredclarkb: it's all we can ask for16:58
*** andreas_s has quit IRC16:58
AJaegersdague: could you review , please - the TEMPEST_PLUGIN is unusual..16:58
clarkbmordred: would be good if we can get someone from bluebox (jlk, timrc?) and someone from osic (cloudnull?) to look at the service logs for these FIPs16:58
mordredclarkb: agree16:58
mordredclarkb: I'd love to know more about the situatoin16:58
clarkb`grep 'is being deleted due to activation failure' /var/log/nodepool/debug.log` is the magic incantatiion for all the datas16:59
clarkb169.45.92.x is bluebox and the rest is osic16:59
*** javeriak_ has joined #openstack-infra16:59
openstackgerritMerged openstack/requirements: Remove Ceilometer from projects
*** thorst_ has joined #openstack-infra16:59
mordredclarkb: OH!!!!!17:01
mordredclarkb: there is still a leak17:01
mordredclarkb: 2016-05-11 16:56:05,471 ERROR shade: Timeout waiting for floating IP to become active. Floating IP was created for server 0dbe1ec7-e914-4530-864c-480c56288fc7 but is being deleted due to activation failure.17:01
*** javeriak has quit IRC17:01
*** fernnest has quit IRC17:01
*** abregman has joined #openstack-infra17:01
mordredclarkb: I show f27f46b9-0dfd-4e66-8253-89e3d5d1db31 in the leak list for bb17:01
sdaguefungi: yep, thanks17:01
clarkbmordred: interesting17:01
mordredclarkb: so it seems that perhaps the delete call is failing17:01
mordredclarkb: we should log that too17:01
*** jistr has quit IRC17:02
*** keedya has joined #openstack-infra17:02
clarkbits like that floating ip goes off into lalala land17:02
clarkbtimeout attaching, can't delete, boom leaked17:02
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove keystone-propose-config-updates
*** thorst_ has quit IRC17:04
pabelangerjeblair: looks like 1.8GB for extras17:05
*** nadya has joined #openstack-infra17:06
jeblairpabelanger: i don't think i'd bother with a manual relase, that'll take just a few minutes17:06
pabelangerjeblair: okay, I'll monitor17:06
*** dizquierdo_afk is now known as dizquierdo17:08
*** xyang has quit IRC17:09
*** xyang has joined #openstack-infra17:09
*** roxanaghe has quit IRC17:10
AJaegerfungi, is a new project by krotscheck for Infra, please review. LGTM17:10
*** xyang has quit IRC17:11
krotscheckjeblair: Where would be the right place to document common mirror troubleshooting steps?17:11
krotscheckjeblair: For example-> Testing against storyboard-webclient, it appears that there's a missing package on the mirror.17:11
*** fawadkhaliq has quit IRC17:12
*** yaume has quit IRC17:12
AJaegersdague: and similar - could you put that on your review queue as well, please?17:12
*** coreyob has quit IRC17:12
jlkclarkb: mordred: so I'd like to get y'all off of Juno before digging too deep into why floating IPs are leaking.17:14
pabelangerlooks like jenkins07 is down17:14
mordredjlk: ++17:14
pabelangergoing to start the clean up process17:14
*** jed56 has quit IRC17:14
*** roxanaghe has joined #openstack-infra17:15
*** xyang has joined #openstack-infra17:15
*** wajdi has joined #openstack-infra17:15
clarkbpabelanger: ok, you can run the ansible playbook and limit it to that host17:16
clarkbpabelanger: it runs against localhost so even with firewall in place should be fine17:16
pabelangerclarkb: okay, deleting jenkins07 nodes in nodepool first17:16
wajdiWould this be the proper place to ask about issues trying to dial in to a meeting using the conferencing system?17:16
*** johnny__ has joined #openstack-infra17:16
clarkbwajdi: it would17:16
fungiwajdi: definitely17:17
pabelangerI wonder if chan_sip is deadlocked again17:17
wajdithanks clarkb . Last week I used hangouts and I got in fine. Currently trying and receiving "this is an invalid conference number"17:17
AJaegerthanks, fungi for review. Could you put the governance change also on your review queue?
wajdithe number works, but the access code is not working17:17
* AJaeger approves now krotscheck's new repo...17:17
clarkbwajdi: can we start with what conference tool you are using? maybe a link to the details?17:17
fungiwajdi: the number at is working for me17:17
*** tqtran has joined #openstack-infra17:18
fungiwajdi: are you trying to connect with sip instead?17:18
wajdiclarkb: fungi Certainly. I'm using Hangouts, dialing in to +1 512-808-575017:18
jeblairkrotscheck: i don't think there are any common troubleshooting tasks17:18
fungiwajdi: i'm not sure what hangouts is17:18
*** arxcruz has quit IRC17:18
wajdiGoogle Hangouts17:18
krotscheckjeblair: Meta17:18
pabelangerclarkb: the restart_jenkins_masters.yaml right?17:19
pleia2wajdi: it works for me in google hangouts17:19
fungiwajdi: i didn't know google hangouts allowed you to dial into arbitrary conference call systems17:19
wajdithe number works. But the Conference code, e.g. 6555 does not17:19
rcarrillocruzinfra-cloud update:17:19
rcarrillocruzthe servers were racked up17:20
pleia2wajdi: ah, let me try that17:20
rcarrillocruzbut iLOs are pulling from wrong VLAN17:20
wajdifungi: I didn't think so either last week. But it did! :)17:20
jeblair'that is not a valid conference number, please try again' is what i get if i dial in and do not enter a conference number17:20
rcarrillocruztalking with network folks now17:20
wajdiwas pleasantly surprised17:20
pleia2wajdi: I was able to dial into that call, people are talking17:20
rcarrillocruzi'll update when i get updates :-)17:20
nibalizerrcarrillocruz: neat17:20
pleia2wajdi: use 6555#17:20
pleia2on google hangouts17:20
wajdi@pleia2 thanks for looking in to that.17:21
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Swift functional tests in Identity v3-only
pleia2rcarrillocruz: good luck :)17:21
wajdipleia2: definitely something really messed up on my end. I'm going to try jitsi. That should work.17:22
*** ddieterly is now known as ddieterly[away]17:22
*** xyang has quit IRC17:22
pleia2wajdi: hope so17:22
wajdithanks anyway! :)17:22
*** simondodsley has joined #openstack-infra17:23
rcarrillocruzwho do i need to bribe for reviews ?17:23
pabelangerrcarrillocruz: thanks for the info17:23
* rcarrillocruz shakes the review tin17:23
rcarrillocruzpabelanger: yeah, even though is not a great info17:23
*** jsavak has quit IRC17:23
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run glanceclient functional tests in Identity v3-only
pabelangerwajdi: are you using a SIP client at all?17:23
rcarrillocruzi'll paste the exact thing:17:23
pabelangersounds like DTMF is not reaching the far end17:23
rcarrillocruz"we need to adjust a few vlans on a switch, the problem is that the documentation RU of that switch and the physical don't match up, so we were working on confirming it's the right switch last i checked"17:24
rcarrillocruz^ from HPE net engineer17:24
*** f1ller is now known as filler17:24
pabelangerokay, jenkins07 back online17:25
*** electrofelix has quit IRC17:25
fungimaybe they just need to drain some more water out of the linecards17:25
pabelangerand ~65 nodes back in play in the gate17:25
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only
*** bpokorny has quit IRC17:26
*** esikachev has joined #openstack-infra17:27
*** bpokorny has joined #openstack-infra17:27
*** lucasagomes is now known as lucas-dinner17:27
openstackgerritMerged openstack-infra/project-config: Add js-generator-openstack to OpenStack-Infra
*** coreyob has joined #openstack-infra17:27
wajdijitsi was the winner17:27
*** ociuhandu has quit IRC17:27
*** mriedem1 has joined #openstack-infra17:27
*** dizquierdo is now known as dizquierdo_afk17:28
*** Swami has joined #openstack-infra17:28
pabelangerwajdi: which client wasn't working?17:28
*** mriedem has quit IRC17:28
pabelangerwajdi: check the dtmf settings and make sure rfc2833 is being use (out of band DTMF)17:28
wajdipabelanger: For some reason Google Hangouts was just not working for me when trying to use the conference number. THe phone number worked, but not the conference number, e.g. 655517:28
wajdi( works for me last week)17:29
pabelangerI'd say DTMF is coming from google inband, and is likely distorted17:29
pabelangerand asterisk cannot detect the tones17:29
*** rkukura has quit IRC17:30
wajdi@pabelanger that does make sense. I wonder if my VPN could be affecting this in any way17:30
*** oanson has quit IRC17:30
pabelangerwajdi: usually any sort of jitter on the line will cause DTMF not to work properly17:30
pabelangerdefault fix is always move to rfc283317:31
*** mriedem1 is now known as mriedem17:31
*** e0ne has quit IRC17:31
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run Ceilometer functional tests in Identity v3-only
*** openstackgerrit has quit IRC17:33
*** mat128|lunch is now known as mat12817:33
*** openstackgerrit has joined #openstack-infra17:33
*** claudiub has joined #openstack-infra17:34
krotscheckjeblair: Could you run this and paste me the output? It should add a new package that appears to be missing -> registry-static -d localhost -o /afs/ --blobstore afs-blob-store --hooks openstack-registry-hooks --one decompress17:36
*** Na3iL has quit IRC17:36
*** fitoduarte has quit IRC17:37
*** sileht has quit IRC17:37
jeblairkrotscheck: you checked it's missing from the rw volume?17:39
krotscheckjeblair: No, however the package history goes back far enough that I'm fairly certain it should have gotten picked up by now.17:39
krotscheckjeblair: /afs/
jeblairright, that's read-only; read-write is /afs/  and it also does not exist17:40
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Group DIB uploads by image name
*** esikachev has quit IRC17:41
*** esikachev has joined #openstack-infra17:41
*** matt-borland has quit IRC17:42
krotscheckjeblair: That _looks_ right...17:42
jeblairkrotscheck: that's kind of :( that the mirror didn't actually mirror it...17:42
*** abregman has quit IRC17:42
*** Na3iL has joined #openstack-infra17:43
openstackgerritMerged openstack-infra/system-config: Enable vos release for mirror.centos
krotscheckjeblair: Yeah, I know. I'm trying to figure out if it may have been a bad regex in the blobstore.17:43
krotscheckjeblair: There's a bunch of packages named decompress-** which... I dunno, maybe did a thing?17:43
krotscheckShouldn't have though17:43
krotscheckjeblair: did /afs/ get created? I don't think it's synced to me yet.17:44
*** sileht has joined #openstack-infra17:44
jeblairkrotscheck: that's read-only -- that happens with vos release, which means that it won't show up there until after the *next* vos release finishes (ie, not the one that is in progress)17:45
jeblairkrotscheck: /afs/ exists now though17:45
krotscheckoh good17:45
krotscheckWell, no. not good, because why did the mirror not mirror that?17:45
openstackgerritClaudiu Belu proposed openstack/requirements: Bumps os-win version to 0.4.2
*** e0ne has joined #openstack-infra17:46
*** tosky has quit IRC17:46
*** ccamacho is now known as ccamacho|away17:46
*** ociuhandu has joined #openstack-infra17:51
asselin_anyone konw if we should use bandersnatch 1.10 or
*** thorst_ has joined #openstack-infra17:53
asselin_I found the answer: is the one to use:
sdagueosic mirror issues - ?17:55
* AJaeger waves good bye for today...17:58
*** e0ne has quit IRC17:59
openstackgerritMerged openstack-infra/system-config: Also mirror extras for
*** austin81 has joined #openstack-infra18:02
openstackgerritBrian Rosmaita proposed openstack-infra/project-config: Add glance api-ref jobs
*** sridhar_ram has joined #openstack-infra18:05
*** kmartin has quit IRC18:06
*** signed8bit is now known as signed8bit_Zzz18:06
*** Na3iL has quit IRC18:07
*** jamesmca_ has quit IRC18:08
*** keedya has quit IRC18:09
*** signed8bit_Zzz is now known as signed8bit18:09
*** pcrews has quit IRC18:10
*** pcrews_ has joined #openstack-infra18:10
pabelangerclarkb: down to 5 image uploads, mainly because we only build fedora-23 and debian-jessie today, other failed because of missing repo.18:12
clarkbmissing ubuntu repos?18:13
clarkboh git repo18:13
*** rbrndt has quit IRC18:13
pabelangerrace condition with openstack/higgins18:13
pabelangerbut, we should be able to land our nodepool-builder workers patch18:14
*** IlyaG has joined #openstack-infra18:15
*** IlyaG has quit IRC18:16
openstackgerritJay Faulkner proposed openstack-infra/project-config: Use TinyIPA and iPXE for most Ironic jobs
dstufftfungi: asselin_ noticed it already, but yea, bandersantch is released now with the fixes needed for that (and for not breaking when Warehouse releases!)18:17
fungidstufft: thanks! i was just about to check back on that pr18:17
fungii'll make sure our mirror updater is upgraded18:18
clarkbsdague: that package downloads currently. whatever it was seems transient18:18
jeblairfungi: unfortunately, our afs patch is still being ignored18:18
clarkbI can check apache logs t o see if they shed more light18:18
jeblairfungi: so it looks like we may need to legitimately fork bandersnatch18:18
*** IlyaG has joined #openstack-infra18:18
JayF is a pretty easy review for ironic project-config, and is blocking us changing a devstack default; if someone has a moment and can review I'd appreciate it.18:19
dstufftjeblair: fwiw I think Christian is kind of overwhelmed with other stuff right now and struggling to find time to bandersnatch much18:20
fungijeblair: oh, good point. i forgot we were running a lightweight fork already18:20
asselin_dstufft, will be shown in pypi, or is not not uploaded yet?18:20
fungii'll hold off updating18:21
asselin_we're going to pull  the tag from hg18:21
*** mixos has joined #openstack-infra18:21
mtreinishjeblair: have you ever hit this before:
dstufftasselin_: I think that 1.10 was actually an OK release and 1.10.1 was not actually needed18:22
dstufftbut I'm not sure18:22
*** jsavak has joined #openstack-infra18:22
pabelangerjeblair: fungi: looks like there is a new comment 2hours old:
*** ddieterly[away] has quit IRC18:22
asselin_dstufft, he mentioned upload file failed ...??18:22
dstufftasselin_: yea, but then
jeblairdstufft: i sympathize, though i don't know what else to do.  the patch is about 1.5 years old, has passing tests (including some added in response to code review), and we've been using it in production for more than 6 months.  it's bitrotted sitting there twice.  :(18:23
asselin_dstufft, ok good to know. thanks18:23
jeblairmtreinish: neat.  is that repeatable?18:24
dstufftjeblair: yea I don't have a good answer for you, just saying that I'm pretty sure Christian isn't like, purposely ignoring y'all or anything (I end up having to email him to get him to poke at even things to fix broken mirrors)18:25
mtreinishjeblair: I haven't tried to repeat it yet, gertty just crashed when I went to open a change18:25
mtreinishI'll try opening the same change again and see what happens18:26
fungimtreinish: jeblair: i see that periodically too and keep meaning to try to recreate with debug enabled18:26
mtreinishjeblair: seems repeatable, crashed again18:26
mtreinishI forgot to run it in --debug, I'll do it now18:27
fungii think it's a result of git gc removing some patchsets. if i manually re-fetch those commits from the origin, gertty stops breaking18:27
*** xyang has joined #openstack-infra18:27
fungiso might be as simple as an extra check in gertty to see if a commit it expects is actually present in the local repo, and retry to retrieve it (or error appropriately when in offline mode)?18:27
jeblairthere are several things wrong though -- 1, all of our shas should have refs so that git gc doesn't remove them. 2, that is in a method that does exactly what fungi just describes18:28
*** chem has quit IRC18:28
jeblairso i wonder if there's a gitpython change that alters the behavior in that case18:28
*** nadya has quit IRC18:28
jeblairmaybe it raises a new kind of exception18:28
mtreinishso running with --debug didn't output anything extra and the log is empty18:28
clarkbsdague: no obvious errors and the only GET on that url at that time 200'd18:28
clarkbsdague: the byte count looks off though18:29
jeblairmtreinish: what gitpython do you have?18:29
sdagueclarkb: ok, I'll just assume blip18:29
*** Sukhdev has quit IRC18:29
fungijeblair: mtreinish: GitPython==2.0.2 for me18:29
mtreinishjeblair: GitPython==2.0.218:30
*** krotscheck_ has joined #openstack-infra18:30
*** fernnest has joined #openstack-infra18:30
*** Sukhdev has joined #openstack-infra18:31
*** krotscheck has quit IRC18:31
*** ddieterly has joined #openstack-infra18:32
*** jsavak has quit IRC18:32
*** krotscheck_ is now known as krotscheck18:33
*** xyang has quit IRC18:33
mtreinishjeblair: is there anything else I should check before I get distracted by something else?18:33
jeblairmtreinish: i'm about to push up a patch18:33
mtreinishoh, ok cool18:33
openstackgerritJames E. Blair proposed openstack/gertty: Handle ValueError on missing git commit
*** jsavak has joined #openstack-infra18:34
*** sridhar_ram1 has joined #openstack-infra18:34
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run novaclient functional tests in Identity v3-only
jeblairmtreinish: ^18:34
*** sridhar_ram has quit IRC18:35
openstackgerritJames E. Blair proposed openstack/gertty: Handle SNIMissingWarning requests exception
bswartzI'm having problems installing oslo packages -- I run my own devpi-server cache -- I saw the ML thread but I don't understand what I need to do to fix my own cache18:35
*** tonytan_brb has quit IRC18:36
mtreinishjeblair: seems to fix it, +1 (although it hangs while it's refetching the commits which was quite slow on the coffee shop wifi)18:36
*** esikachev has quit IRC18:37
jeblairmtreinish: yeah; if that's on a subscribed project, it likely means something went wrong already, so it's sort of a recovery operation at that point.  if it happens a lot, maybe there's something else going on....  if it's a new change (like you clicked on a link or searched for it) then it's a defacto synchronous task anyway.18:38
jeblairmtreinish: note, that if you, say, removed a git repo from under gertty, it will do that for all the changes in that repo; instead, you can run once with "--fetch-missing-refs", and it will try to pull in anything missing at startup.18:38
*** fitoduarte has joined #openstack-infra18:39
jeblair(that is a slow operation, which is why it's not the default)18:39
*** ddieterly is now known as ddieterly[away]18:40
fungibswartz: you need a new version of devpi i believe. there's a patch for it, though i don't know if there's been a release with that yet18:41
*** rvasilets__ has joined #openstack-infra18:41
dstufftdevpi has been released18:41
dstufftafaik you just need a new devpi-common18:41
*** weshay has quit IRC18:41
jeblairReleased volume mirror.npm successfully18:41
jeblairreal    71m19.766s18:41
jeblairkrotscheck: ^18:41
mtreinishjeblair: I think it was probably related to my recent laptop migration. I copied the gerrty db (and yaml) but just recloned all the working git dirs (which was part of my normal install scripts)18:42
bswartzdstufft: ty I'll try to update18:42
mtreinishso it probably lost things it had locally checked out before and was surprised when they disappeared18:42
jeblairmtreinish: ah, then you will probably want to run --fetch-missing-refs when you are on a good network connection.  until then, you'll have the slight delay when you open any old change18:42
jeblairmtreinish: but thanks, this case should have been handled, so i'm glad we found about the new behavior :)18:43
jeblairReleased volume mirror.npm successfully18:43
jeblairreal    1m12.752s18:43
jeblairkrotscheck: ^ we're all caught up18:43
*** e0ne has joined #openstack-infra18:44
*** ianychoi has quit IRC18:45
*** e0ne has quit IRC18:45
jeblairkrotscheck: i have released the mirror lock; so it should start running normally now18:46
*** ianychoi has joined #openstack-infra18:47
*** pcrews_ has quit IRC18:47
*** ddieterly[away] is now known as ddieterly18:47
*** pcrews has joined #openstack-infra18:47
jeblair#status log npm mirror sync finished; lock is released18:47
openstackstatusjeblair: finished logging18:47
*** xyang has joined #openstack-infra18:47
pabelangerjeblair: krotscheck: congrats!18:47
*** sdake has quit IRC18:48
*** xyang has quit IRC18:49
*** tesseract has quit IRC18:51
*** pt_15 has joined #openstack-infra18:51
*** rbrndt has joined #openstack-infra18:52
*** xyang has joined #openstack-infra18:52
*** nadya has joined #openstack-infra18:55
*** xyang has quit IRC18:55
openstackgerritamrith proposed openstack-infra/project-config: Adding new repository trove-image-builder
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in heat
*** vgridnev_ has joined #openstack-infra18:57
*** vgridnev_ has quit IRC18:57
*** ianychoi has quit IRC18:59
*** xyang has joined #openstack-infra18:59
*** tonytan4ever has joined #openstack-infra18:59
*** roxanaghe has quit IRC19:00
*** vgridnev_ has joined #openstack-infra19:00
pabelangerokay, just confirmed: centos mirror works correctly now that extras has been mirrored19:01
openstackgerritBrian Rosmaita proposed openstack-infra/project-config: Add glance api-ref jobs
clarkbpabelanger: I can approve now19:02
clarkbpabelanger: is that ok?19:02
pabelangerclarkb: sure, I can monitor19:02
krotscheckjeblair: WOO!19:02
* krotscheck does the ahppy dance19:02
krotscheckThanks so much for your hel on this, jeblair :)19:02
openstackgerritMerged openstack-infra/subunit2sql: Add db api tests for subunit2sql (1/3)
jeblairkrotscheck: the cron job just ran too19:03
*** pcrews has quit IRC19:03
openstackgerritMerged openstack-infra/subunit2sql: Add db api tests for subunit2sql (2/3)
clarkbpabelanger: btw I reall really really like that because we use rsync and centos doesn't aggressively delete things it means we can use the existing gpg trust chain19:03
jeblairkrotscheck: so i think it's all set19:03
openstackgerritMerged openstack-infra/subunit2sql: Add db api tests for subunit2sql (3/3)
*** baumann has joined #openstack-infra19:03
krotscheckI'll send out the "Hey go check your projects" email, and then we can flip the switch early next week.19:03
pabelangerclarkb: Ya, it works pretty nice.  Happy how it turned out.19:04
*** keedya has joined #openstack-infra19:04
*** sambetts is now known as sambetts|afk19:04
pabelangerclarkb: the last step is to write a simply python app, to validate we have all the files rsync'd (parse repodata info) before we do a vos release19:04
*** pcrews has joined #openstack-infra19:05
*** esikachev has joined #openstack-infra19:06
clarkbpabelanger: that is a confirmation that the index is complete?19:06
clarkber s/complete/correct/ given the files on disk19:06
pabelangerclarkb: ya, a validation step. Make sure all files in repo index exist, we can even check md5 hash if needed19:07
pabelangerneed to profile and see how long that takes19:07
clarkbtdurakov: any luck with libvirt cpu definitions on xenial?19:08
*** jamesmca_ has joined #openstack-infra19:08
*** dtardivel has quit IRC19:09
*** weshay has joined #openstack-infra19:10
clarkbfungi: any idea how to check what version of openstackid the dev server is running?19:11
* clarkb checks puppetboard19:12
clarkb that must be it19:12
*** jamesmca_ has quit IRC19:13
*** ddieterly is now known as ddieterly[away]19:13
*** ddieterly[away] is now known as ddieterly19:14
*** pvaneck has joined #openstack-infra19:15
openstackgerritMerged openstack-infra/project-config: Add CentOS support to
pabelangerclarkb: monitoring nodepool.log now19:15
*** amrith is now known as _amrith_19:16
*** ddieterly is now known as ddieterly[away]19:17
pabelangerclarkb: keep forgetting, we need new images for nodepool scripts19:18
clarkboh right19:18
pabelangerso, we're down to 2 DIB uploads on nodepool.o.o19:19
*** vgridnev_ has quit IRC19:19
pabelangershould be able to land the stack in the next 30mins19:19
*** ianychoi has joined #openstack-infra19:19
fungiclarkb: i think smarcet added some version query or maybe has it write out the version to disk or something? i'm not sure of the details19:20
*** IlyaG has quit IRC19:21
pabelangerclarkb: oh, and too please19:21
*** davideagnello has quit IRC19:21
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in ceilometer
clarkbpabelanger: one thing on
clarkbotherwise stack lgtm and we can merge them all19:25
openstackgerritPaul Belanger proposed openstack-infra/puppet-nodepool: Expose nodepool-builder worker settings
pabelangerclarkb: thanks!19:26
clarkbpabelanger: I approved ^, you will need to approve the child changes as their parents merge due to the depends on and no shared queue19:28
pabelangerclarkb: okay19:28
pabelangerstill uploading to rax-iad19:29
*** jsavak has quit IRC19:33
mtreinishkrotscheck: using the mirror to install openstack-health works for me locally. (although it was much slower) Just in case you were curious19:33
openstackgerritMerged openstack-infra/system-config: Remove bare-precise from nodepool-builder logging
openstackgerritMerged openstack-infra/subunit2sql: Capitalize first characters in param docs
*** jsavak has joined #openstack-infra19:33
*** rkukura has joined #openstack-infra19:34
*** jaosorior has joined #openstack-infra19:34
*** shardy has quit IRC19:36
fungimtreinish: slower? interesting. i wonder if you're hitting a closer cdn endpoint for lots of the deps when you don't pin to the mirror19:36
*** rkukura has quit IRC19:36
*** bnemec has quit IRC19:36
lifelessfungi: yes19:37
lifelessfungi: pypi is mirrored19:37
*** BobH_ has joined #openstack-infra19:37
*** DevBox has quit IRC19:37
*** lakshmiS_ has joined #openstack-infra19:38
fungilifeless: i think this was about the npm mirror, not the pypi mirror19:38
fungiso presumably github cdn endpoints or somethintg19:38
mtreinishfungi: that I can't say, it was just an anecdotal observation. Normally when I do a fresh npm install on it it's very fast that it doesn't make sense to go do something else19:38
*** ddieterly[away] is now known as ddieterly19:38
*** dmk0202 has joined #openstack-infra19:38
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in magnum
*** jamesmca_ has joined #openstack-infra19:39
fungimtreinish: local caching which was invalidated the first time you installed from the new location maybe? do subsequent reinstalls with the mirror go faster?19:39
mtreinishfungi: but in this case it took several minutes19:39
*** matrohon has joined #openstack-infra19:39
fungii guess we'll have some more objective data once we have horizon functional jobs running against the mirror19:39
*** e0ne has joined #openstack-infra19:39
*** lakshmiS has quit IRC19:39
mtreinishfungi: maybe, but I was thinking more about when I was setting up my new laptop last week19:40
mtreinishI'll test things more when I go home19:40
fungimtreinish: another possibility is cold cache on the mirror19:40
pabelangerfungi: mtreinish: that's what I was thinking19:40
fungisince our "mirrors" are just caches19:40
fungiso cache misses take teh slow boat over afs/udp19:40
*** e0ne has quit IRC19:41
jeblairoh yeah, cold caches on our mirrors are quite slow.  but that doesn't last long around these parts.19:41
openstackgerritRaildo Mascena proposed openstack-infra/project-config: Run identity v3 only job in barbican
daemontoolfungi, jeblair   if you have any spare minutes please, need +A, new repo creation, so tomorrow the Team can keep working on it
krotscheckmtreinish: Thanks! That's a good data point.19:42
*** bnemec has joined #openstack-infra19:44
smcginnisAnyone else getting stacking failures finding oslo.context 2.3.0?19:45
openstackgerritItisha Dewan proposed openstack-infra/project-config: Adding library Delimiter to the CI system
*** jsavak has quit IRC19:46
*** jsavak has joined #openstack-infra19:46
fungismcginnis: pip 8.1.2? using a pypi mirror (maybe via devpi)?19:47
*** rkukura has joined #openstack-infra19:48
smcginnisfungi: Looks like at least on one machine still using 8.0.2. Not local mirror, direct to pypi.19:49
*** r-daneel has joined #openstack-infra19:49
smcginnisINSTALLED: 2.2.019:49
smcginnisLATEST: 2.0.019:49
smcginnisfungi: On pypi I can go direct to the package, so it's there:
smcginnisfungi: But a search does not show it:
smcginnisMax is
fungismcginnis: doesn't sound familiar in that case. maybe the devstack devs in #opsnstack-qa have more of an idea as to what's going on with ut19:50
smcginnisI did get confirmation from at least one other person that they are getting the same thing.19:50
fungismcginnis: don't trust the "search" feature on, it's long broken19:50
fungiand pip doesn't depend on pypi search, it does its own index enumeration19:50
smcginnisfungi: Yeah, just interesting that the pip search for the package is reflecting the same thing.19:50
*** jaosorior has quit IRC19:51
fungismcginnis: weird,*2.3.0 says it's in the constraints list19:52
smcginnisfungi: Yep. My CI is failing to stack because of this.19:52
fungiwe also have tarballs/wheels for it archived at
fungioh, also shows 2.3.0 as available19:53
smcginnisJust got a report of the same situation with oslo.config19:53
*** tphummel has quit IRC19:53
smcginnisfungi: Yeah, the package is out there. The search just reflects the pip results of a max of
smcginnisPypi's index is messed up maybe?19:53
*** rkukura_ has joined #openstack-infra19:54
smcginnisPerhaps related:
clarkbhave a paste of the output of a failed pip install?19:54
*** tphummel has joined #openstack-infra19:54
mriedemAJaeger: should new jobs be using xenial nodes?
fungiyeah, sometimes fastly negative-caches some urls for no good reason19:55
bswartzUpdating my devpi-server cache to the latest version solved my oslo.config install issues19:55
bswartzI also did a rm -rf ~/.devpi19:55
fungismcginnis said he's not using devpi though, and is also on older pip (8.0.2) so presumably not related19:55
smcginnisfungi: Also tried updating pip to latest, same results.19:56
clarkbthe from versions list is empty19:56
clarkbI think that means it doesnt see the package at all19:56
*** HeOS has quit IRC19:57
smcginnisclarkb: pip search output:
*** rkukura has quit IRC19:57
*** rkukura_ is now known as rkukura19:57
clarkbya ignoring search because install is the best data for install19:57
*** mixos has quit IRC19:57
*** ccamacho|away has quit IRC19:58
*** thorst_ has quit IRC19:59
*** ccamacho has joined #openstack-infra19:59
*** vilobhmm11 has joined #openstack-infra20:00
clarkbI cant reproduce locallyfwiw20:00
smcginnisclarkb: DO you have a local cache?20:00
*** itisha has joined #openstack-infra20:01
*** thorst_ has joined #openstack-infra20:01
*** yamahata has quit IRC20:01
clarkbsmcginnis: the warnings from pip searchimply you are not talki g to pypi20:01
clarkbyou should not get the sni warning against pypi iirc20:02
openstackgerritPhilip Marc Schwartz proposed openstack-infra/system-config: Add zuul_launcher to openstack_project and run on jenkinsXX hosts.
vilobhmm11clarkb, fungii, anteaya, AJaegar : working on the effort to create a new lib for cross-project quota enforcement and here is the patch that itisha created to add "delimiter" to CI system can someone please have a look. This will help us to move quickly with this cross project effort20:02
vilobhmm11itisha, nikhil : ^^20:03
nikhilvilobhmm11: we also have CP spec ppl can refer20:03
*** jsavak has quit IRC20:03
*** salv-orl_ has joined #openstack-infra20:03
vilobhmm11sure CP spec for reference :
*** jsavak has joined #openstack-infra20:04
*** sdake has joined #openstack-infra20:05
*** thorst_ has quit IRC20:06
*** salv-orlando has quit IRC20:06
mtreinishfungi, krotscheck: I ran things locally again (with time this time)
mtreinishit definitely wasn't as slow as the first time I ran it (also the it was slower than I remembered without the mirror)20:09
*** hashar has joined #openstack-infra20:09
fungimtreinish: 7 seconds difference i would believe20:09
krotscheckmtreinish: I'm secretly having everyone run this so it warms up the AFS caches.20:09
krotscheckSo a lot of that initial lag is probably the cache warming up on dfw20:09
*** daemontool has quit IRC20:09
fungiour mirror servers aren't necessarily well-connected or high-performance either compared to what github et al are likely fronted with20:10
krotscheckWith that in mind, has its own CDN, so it's expected that using them will probably be faster.20:10
fungiahh, so there is a specific cdn for most npm stuff? even less surprising result in that case20:10
krotscheckfungi: Yep20:11
*** e0ne has joined #openstack-infra20:11
krotscheckfungi: I expect that number to be on-par or better when we're running it on a node in the same infra cluster.20:11
fungii think "takes 15% longer than" is a perfectly reasonable performance expectation20:11
*** vgridnev has joined #openstack-infra20:12
mtreinishyeah, that's probably fine20:12
*** vgridnev has quit IRC20:12
fungiand yes, most actual use of these mirrors will be over sub-milisecond latency links20:12
clarkbin region should be fast on warm cache20:12
clarkbya that20:12
mtreinishkrotscheck: well now I'm gonna run it against ord and other regions :)20:13
openstackgerritItisha Dewan proposed openstack-infra/project-config: Adding library Delimiter to the CI system
krotscheckmtreinish: Yay!20:13
*** e0ne has quit IRC20:13
* fungi wonders if a cold cache is more like gazpacho or more like borscht20:13
krotscheckmtreinish, fungi: I'm looking for packages that didnt' get synced. Apparenlty the mirroring script has problems with those :/20:13
krotscheckfungi: Gazpacho, I would say. Because you know there's a bunch of vegetables there, but you can't identify them.20:14
*** daemontool has joined #openstack-infra20:14
openstackgerritMerged openstack/requirements: Ignore more build temp dirs
krotscheckSo, now that i have javascript things in infra again that I'm the main developer on, who wants to volunteer to help land patches? :)20:15
pabelangerclarkb: okay, image uploads done. Starting the +A on nodepool-builder patches20:15
openstackgerritMerged openstack-infra/glean: Enable multiple VLANs per physical interface
*** _amrith_ is now known as amrith20:19
*** xyang has quit IRC20:19
mtreinishkrotscheck: just ord, i'm too impatient to wait ~6min again:
openstackgerritMerged openstack-infra/nodepool: Expose nodepool-builder workers settings
hongbinAJaeger: Hi, I wonder if you can add me to these two gerrit group: higgins-core, higgins-release20:26
*** mjblack has quit IRC20:27
*** earlephilhower has quit IRC20:28
*** annegentle has joined #openstack-infra20:28
*** kgiusti has left #openstack-infra20:30
openstackgerritgreghaynes proposed openstack-infra/glean: Enable support for configuring bonded interfaces
*** earlephilhower has joined #openstack-infra20:31
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Group DIB uploads by image name
pabelangerclarkb: okay, that should happy in the gate now^20:33
*** kien-ha has joined #openstack-infra20:33
*** esikache1 has joined #openstack-infra20:33
*** ddieterly is now known as ddieterly[away]20:34
*** kien-ha has quit IRC20:34
*** jamesmca_ has quit IRC20:34
openstackgerritMerged openstack-infra/puppet-nodepool: Expose nodepool-builder worker settings
cloudnullclarkb: was just reading scroll back im missed the ping and have been locked up in meetings most of the day sorry, is the floating IP situation figued out?20:34
jeblairpabelanger: oh, did we decide not to do the other approach?20:34
clarkbcloudnull: it is not20:35
cloudnullRE: OSIC problems ...20:35
clarkbcloudnull: they seem to periodically timeout on attach then fail to delete20:35
pabelangerjeblair: no, I think upload right away after build is the way. I just don't know how to do that yet :)20:35
clarkbcloudnull: did you see the example I pasted? should have uuids for ip and server would be great to see why that happened20:35
*** esikachev has quit IRC20:35
openstackgerritMerged openstack-infra/puppet-openstackci: Expose nodepool-builder workers settings
*** vilobhmm11 has quit IRC20:36
*** xyang has joined #openstack-infra20:36
cloudnullclarkb:  i see the log output from shade . do you have the like for the paste?20:37
* cloudnull catching up20:37
*** kien-ha has joined #openstack-infra20:37
*** baumann has quit IRC20:37
*** ddieterly[away] is now known as ddieterly20:37
*** mjblack has joined #openstack-infra20:37
clarkbcloudnull: wasnt a paste was just here in irc20:38
*** roxanaghe has joined #openstack-infra20:38
*** xyang has quit IRC20:38
cloudnullok, i see that log output, im logging into the env now and will start poking around20:38
pabelangerjeblair: If I understand, the issue is on 1635 of I guess we need to refactor waitForBuiltImages() to waitForBuildImageThenUpload()20:39
openstackgerritMerged openstack-infra/system-config: Switch to 12 upload-workers for nodepool
cloudnulland will post back regarding issues i find20:39
*** baumann has joined #openstack-infra20:40
*** thorst_ has joined #openstack-infra20:41
clarkbcloudnull: great thank you20:41
jeblairpabelanger: yes -- but you'll have a pool of things you're waiting for and won't know which will finish first (assuming there is more than one builder).  so i would just put each image build+upload into its own thread.20:41
*** ayoung has quit IRC20:41
*** kzaitsev_mb has joined #openstack-infra20:42
*** dimtruck is now known as zz_dimtruck20:42
*** thorst__ has joined #openstack-infra20:42
*** pcaruana has quit IRC20:42
pabelangerjeblair: okay, that makes sense. Going to take me a few tries to get it, threading not my strongest suit20:43
openstackgerritHongbin Lu proposed openstack-infra/devstack-gate: Add Higgins to PROJECTS list
jeblairpabelanger: a few lines up, updateImage launches a thread for each snapshot updater20:43
*** rcernin has joined #openstack-infra20:44
*** thorst___ has joined #openstack-infra20:44
*** icey has quit IRC20:44
pabelangerjeblair: Ya, I see that now. Okay, I'll use SnapshotImageUpdater as a reference20:45
*** thorst_ has quit IRC20:45
*** thorst_ has joined #openstack-infra20:45
*** icey has joined #openstack-infra20:46
*** _ari_ has quit IRC20:47
*** thors____ has joined #openstack-infra20:47
*** thorst__ has quit IRC20:47
*** thorst__ has joined #openstack-infra20:48
notmynamemordred: clarkb: FWIW, the rax team has a jenkins instance doing unit and functional tests against the golang branch
funginotmyname: ooh, that's a great point of reference to crib from (though would be better with set -x)20:49
*** mixos has joined #openstack-infra20:49
notmynamefungi: let's consider all of this as POC and examples :-)20:49
*** thorst___ has quit IRC20:49
*** jsavak has quit IRC20:49
funginotmyname: yeah, just meant i'm uncertain all the commands run are displayed in that log, so harder to crib from ;)20:49
notmynamefungi: I added a link to it on
*** yamahata has joined #openstack-infra20:50
notmynamefungi: the interesting part for me is the unit test running at the top20:50
*** thorst_ has quit IRC20:50
timothyb89krotscheck, mtreinish: on the topic of npm performance, have you seen ? I've seen dramatic speedups for running 'npm install', ~5m -> 30s20:50
mordrednotmyname: cool20:50
openstackgerritHongbin Lu proposed openstack/requirements: Add Higgins project to projects.txt
notmynamefungi: ie it makes a lot of sense to "build deploy test" when you're talking about functional tests, but the golang unittest stuff is new to me20:51
mordredERROR: object '/var/lib/jenkins/.java/' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.20:51
notmynameand the overal good thing is showing both python and golang code in the same project being tests20:51
funginotmyname: if you can tease a full job definition out of them, we might not have to guess at as much stuff20:51
*** thors____ has quit IRC20:51
mordredI do not want to be LD_PRELOADED20:51
mordredfungi: what fungi said20:51
fungimordred: thankfully it's not preloaded20:51
mordrednotmyname: what fungi said - I'd love to know what commands it's running20:51
notmynamemordred: again, don't worry so much about this particular thing and if it's exactly how you'd want it run :-)20:52
mordredlike - is that a script running go get ; go vet ; go test ?20:52
mordredor the make file20:52
*** thors____ has joined #openstack-infra20:52
notmynamemordred: fungi: I'll ask/investigate20:52
mordrednotmyname: totally - mostly just curious as to what's happening now20:52
funginotmyname: awesome--thanks!20:52
*** thorst__ has quit IRC20:52
fungiyeah, similar to mordred i mostly want to know what commands their job scripts are running, since that provides a much more solid starting point20:52
funginot that we would necessarily do exactly the same thing20:53
*** thorst__ has joined #openstack-infra20:53
*** thor_____ has joined #openstack-infra20:54
clarkbmordred I always ld preload jenkins for maximum CI20:54
mordredclarkb: you ldpreload everytihng though20:54
*** thorst___ has joined #openstack-infra20:55
*** thorst_ has joined #openstack-infra20:56
*** thors____ has quit IRC20:56
pabelangermordred: did you want to do something on nodepool.o.o were ready to do restarts20:56
*** xyang has joined #openstack-infra20:56
mordredpabelanger: ooh! yeah. did you land your patches?20:57
*** thors____ has joined #openstack-infra20:57
pabelangermordred: indeed, they are on nodepool.o.o now20:57
openstackgerritMerged openstack-infra/shade: Amend the valid fields to update on recordsets
mordredwoot. SO ...20:57
*** thorst__ has quit IRC20:57
mordredfungi, clarkb, pabelanger: - I'd love to apply this too, if y'all are ok with that.20:58
mordredbut we can also let pabelanger's changes land today and do that tomorrow20:58
openstackgerritMerged openstack-infra/glean: Enable support for configuring bonded interfaces
*** thorst__ has joined #openstack-infra20:58
*** jamesmca_ has joined #openstack-infra20:58
*** thor_____ has quit IRC20:58
*** weshay has quit IRC20:59
pabelangerI'm fine landing it today20:59
*** thorst___ has quit IRC20:59
*** xyang has quit IRC20:59
*** thorst_ has quit IRC21:00
*** mjblack has quit IRC21:01
*** aratus has joined #openstack-infra21:02
*** thors____ has quit IRC21:02
openstackgerritTim Buckley proposed openstack-infra/project-config: Create stackviz element and script in nodepool
*** davideagnello has joined #openstack-infra21:02
*** thorst__ has quit IRC21:02
cloudnullclarkb: i'm not seeing any smoking gun regarding vm build + floatingIP assignment.21:05
*** xyang has joined #openstack-infra21:05
*** sridhar_ram1 has quit IRC21:05
cloudnullmessaging had a fit on the 8 which caused some traces in the nova logs, but nothing notable since then21:05
*** abregman has joined #openstack-infra21:05
openstackgerritTim Buckley proposed openstack-infra/project-config: Create stackviz element and script in nodepool
cloudnullneutron seems to be happy at this time .21:06
*** mjblack has joined #openstack-infra21:06
*** aysyd has quit IRC21:06
*** hashar_ has joined #openstack-infra21:06
mtreinishtimothyb89: I haven't tried that before. I'll give it a look21:07
*** baumann has left #openstack-infra21:07
pabelangermordred: clarkb: I'm going to step out for a moment to get some food. Everything is in place for the new nodepool-builder worker flags.  I should be back shortly21:07
*** eharney has quit IRC21:08
cloudnullIm going to fire off a bunch of builds/assosiates to see if i can recreate it and trap something from the logs.21:08
*** hrybacki has quit IRC21:09
*** tonytan4ever has quit IRC21:10
*** hrybacki has joined #openstack-infra21:10
*** thorst_ has joined #openstack-infra21:10
*** psilvad has quit IRC21:10
*** hashar has quit IRC21:11
*** ekarlso has quit IRC21:11
*** thorst__ has joined #openstack-infra21:11
*** abregman has quit IRC21:12
*** thorst___ has joined #openstack-infra21:12
*** thors____ has joined #openstack-infra21:14
*** davideagnello has quit IRC21:14
*** ekarlso has joined #openstack-infra21:14
*** thor_____ has joined #openstack-infra21:15
*** thorst_ has quit IRC21:15
*** ayoung has joined #openstack-infra21:15
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
*** ldnunes has quit IRC21:16
*** davideagnello has joined #openstack-infra21:16
*** thorst__ has quit IRC21:16
*** thorst_ has joined #openstack-infra21:16
*** IlyaG has joined #openstack-infra21:17
*** jamesmca_ has quit IRC21:17
*** thorst___ has quit IRC21:17
*** thorst__ has joined #openstack-infra21:18
openstackgerritMerged openstack/requirements: Update oslo.versonedobjects to 1.9.1 and block 1.9.0
*** thors____ has quit IRC21:18
*** thorst___ has joined #openstack-infra21:19
*** thor_____ has quit IRC21:19
*** thors____ has joined #openstack-infra21:20
openstackgerritMerged openstack/requirements: Remove kolla-mesos from projects.txt
*** vilobhmm11 has joined #openstack-infra21:20
*** matrohon has quit IRC21:20
*** thor_____ has joined #openstack-infra21:21
*** thorst_ has quit IRC21:21
*** keedya has quit IRC21:22
*** thorst_ has joined #openstack-infra21:22
clarkbcloudnull: ok when I am not out running errands I can check our logs21:22
*** thorst__ has quit IRC21:22
*** thorst__ has joined #openstack-infra21:23
*** thorst___ has quit IRC21:23
*** jordanP has quit IRC21:23
*** thors____ has quit IRC21:24
*** xyang has left #openstack-infra21:24
*** thorst___ has joined #openstack-infra21:24
*** rhallisey has quit IRC21:24
*** thor_____ has quit IRC21:25
*** thors____ has joined #openstack-infra21:25
*** regXboi has quit IRC21:25
*** thor_____ has joined #openstack-infra21:26
*** thorst_ has quit IRC21:26
*** lakshmiS_ has quit IRC21:26
*** |-paul-| has joined #openstack-infra21:27
*** thorst_ has joined #openstack-infra21:27
*** thorst__ has quit IRC21:27
*** hashar_ has quit IRC21:27
*** thorst__ has joined #openstack-infra21:28
*** thorst___ has quit IRC21:28
*** thorst___ has joined #openstack-infra21:29
*** zz_dimtruck is now known as dimtruck21:29
*** thors____ has quit IRC21:29
*** nadya has quit IRC21:30
*** esikache1 has quit IRC21:30
*** thors____ has joined #openstack-infra21:30
*** thor_____ has quit IRC21:30
*** jamesmca_ has joined #openstack-infra21:30
*** baoli has quit IRC21:31
*** thorst_ has quit IRC21:31
*** thorst_ has joined #openstack-infra21:31
*** thor_____ has joined #openstack-infra21:32
*** thorst__ has quit IRC21:32
*** vilobhmm11 has left #openstack-infra21:33
*** thorst___ has quit IRC21:33
*** thorst__ has joined #openstack-infra21:33
*** thorst___ has joined #openstack-infra21:34
*** edmondsw has quit IRC21:34
*** thors____ has quit IRC21:35
*** thorst_ has quit IRC21:35
*** thors____ has joined #openstack-infra21:35
openstackgerritDirk Mueller proposed openstack/requirements: Clarify wording by mentioning the name of the gate job
*** thorst_ has joined #openstack-infra21:36
*** thor_____ has quit IRC21:36
*** thor_____ has joined #openstack-infra21:37
*** thorst__ has quit IRC21:38
*** Sukhdev has quit IRC21:38
*** thorst__ has joined #openstack-infra21:39
*** thorst___ has quit IRC21:39
*** savihou has quit IRC21:39
*** thorst___ has joined #openstack-infra21:40
*** thors____ has quit IRC21:40
*** thorst_ has quit IRC21:41
*** thors____ has joined #openstack-infra21:41
openstackgerritJoshua Harlow proposed openstack-infra/project-config: Add repo/project
*** jang has quit IRC21:41
openstackgerritTom Barron proposed openstack-infra/project-config: Enable translation jobs for manila-ui
*** thor_____ has quit IRC21:42
*** thorst_ has joined #openstack-infra21:42
*** thorst__ has quit IRC21:43
*** thorst__ has joined #openstack-infra21:44
*** thorst___ has quit IRC21:44
*** thors____ has quit IRC21:45
*** thorst___ has joined #openstack-infra21:45
*** ddieterly is now known as ddieterly[away]21:45
*** kzaitsev_mb has quit IRC21:45
*** thors____ has joined #openstack-infra21:46
*** mriedem has quit IRC21:47
*** jamesmca_ has quit IRC21:47
*** julim has quit IRC21:47
*** pradk has quit IRC21:47
*** tho______ has joined #openstack-infra21:47
*** gordc has quit IRC21:47
openstackgerritAndreas Florath proposed openstack/diskimage-builder: Fix apt-sources configuration for debian-minimal
*** thorst_ has quit IRC21:47
*** thorst__ has quit IRC21:48
*** thorst_ has joined #openstack-infra21:48
*** ddieterly[away] is now known as ddieterly21:48
*** HeOS has joined #openstack-infra21:48
*** Goneri has quit IRC21:48
*** kzaitsev_mb has joined #openstack-infra21:48
*** thorst__ has joined #openstack-infra21:49
*** thorst___ has quit IRC21:49
*** claudiub has quit IRC21:50
openstackgerritDirk Mueller proposed openstack/requirements: Removes posix-ipc from global-requirements
*** thors____ has quit IRC21:50
kevinbentonany gerrty devs around?21:51
*** tho______ has quit IRC21:52
pabelangermordred: clarkb: okay, I'm going to restart nodepool now minus 307755.21:52
*** jamesmca_ has joined #openstack-infra21:52
*** thorst_ has quit IRC21:53
*** thorst__ has quit IRC21:53
*** jsavak has joined #openstack-infra21:55
*** HeOS has quit IRC21:55
openstackgerritMorgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project
pabelanger#status log nodepool restarted to pickup
*** mkwiek has quit IRC21:56
openstackstatuspabelanger: finished logging21:56
*** Swami has quit IRC21:57
*** mkwiek has joined #openstack-infra21:58
openstackgerritMorgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project
*** austin81 has left #openstack-infra21:59
*** njohnston is now known as njohnston_afk22:00
*** njohnston_afk is now known as njohnston__22:00
*** thorst_ has joined #openstack-infra22:01
*** xyang1 has quit IRC22:01
openstackgerritMorgan Fainberg proposed openstack-infra/project-config: Import ldappool into gerrit and setup project
*** rcrit has quit IRC22:02
openstackgerritKien Ha proposed openstack-infra/jenkins-job-builder: Add support for http request plugin
hongbinHi, I have submitted a request to create a project: . I wonder if anyone can add me to these two gerrit groups: higgins-core, higgins-release22:04
*** ddieterly is now known as ddieterly[away]22:04
*** kien-ha has quit IRC22:05
*** javeriak_ has quit IRC22:05
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: add extra metadata to the tag messages
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: fix previous version detection for repos with no tags
openstackgerritDoug Hellmann proposed openstack-infra/release-tools: automatically submit constraint updates after tagging
*** jamesmca_ has quit IRC22:06
*** sridhar_ram has joined #openstack-infra22:07
pabelangerjeblair: fungi: clarkb: do you have a moment to help with nodepool.o.o, I am not sure things are coming back online as expected22:08
clarkbI am about 5 minutes from home can get on then22:09
pabelangerclarkb: great, thanks22:09
*** annegentle has quit IRC22:10
*** cloudtrainme has joined #openstack-infra22:10
*** jsavak has quit IRC22:12
clarkbok here now22:12
*** jordanP has joined #openstack-infra22:13
pabelangerclarkb: sure, so if you look at nodepool.log, it appear nodepoold is no longer doing anything22:13
pabelangerclarkb: it tried to mass delete some nodes but that's the last traffic22:13
*** thorst_ has quit IRC22:14
*** jordanP has quit IRC22:14
clarkbpabelanger: we have a thread dump signal handler, I think it is sigusr222:15
clarkblet me double check22:15
clarkbwe can run that to try an dsee what is going on22:15
pabelangerright, I forgot about that22:15
clarkbya sigusr2 so if you kill -USR2 $pid you should get that all in the debug log22:16
pabelangersure, 1 sec22:16
*** dimtruck is now known as zz_dimtruck22:16
*** davideagnello has quit IRC22:17
*** mixos has quit IRC22:17
pabelangerran command22:17
pabelangerclarkb: Ya, nothing in debug.log22:18
clarkbagainst 6862 ?22:18
clarkbif so then I think its probably really hosed22:18
pabelangerokay, how should we proceed?22:19
clarkbwhich changes did we add in?22:19
clarkbwas it just the builder worker change? that shouldn't affect nodepoold22:19
*** sigmavirus24 is now known as sigmavirus24_awa22:19
pabelangerya just 29433922:19
clarkbI have a strace against it and its doing some things22:20
clarkblooks like zmq socket is still alive and receiving things22:20
openstackgerritItisha Dewan proposed openstack-infra/project-config: Adding library Delimiter to the CI system
*** thorst_ has joined #openstack-infra22:20
clarkbpabelanger: we can try restarting it, my only fear with that is since the only change to nodepool itself should be indepdnent of this daemon that the issue is in a dep somewhere like sahde or paramiko or whatever22:21
pabelangerclarkb: maybe we pulled in a newer version of shade22:21
pabelangershade==1.8.0 is installed22:22
pabelangernot sure what we had before22:22
clarkbya I thought we were already running with that based on when it released and when jeblair restarted for the zuul 2.5 things22:22
clarkbanyways with no stack dumping I think only thing we really can do is restart and try to get better info if there is a next time22:23
pabelangerwe also have paramiko==2.0.022:23
clarkbits almost like the main thread died but we have a non daemon thread running keeping the process alive22:23
pabelangerclarkb: okay, did you want to restart it?22:23
clarkbpabelanger: you can do it, just stop then start22:23
clarkbif the main thread did die we might be able to see a traceback in the logs22:24
pabelangerokay stopped22:24
pabelangerand starting22:24
*** thorst_ has quit IRC22:25
clarkbit does while True: signal.pause() in the main thread22:25
JayFwhere are the logs for post jobs kept?22:25
clarkbso even if it gets signals it should go back to pausing, only thing would be an exception I Think22:26
clarkbbut there is no logging wrapped aroun dthat :/22:26
clarkbpabelanger: it seems stuck again22:26
pabelangerclarkb: yup22:26
JayFnevermind, jroll beat me to the punch22:26
*** yamamoto has quit IRC22:28
*** yamamoto has joined #openstack-infra22:28
pabelangerkill -USR2 not working still22:28
*** zz_dimtruck has quit IRC22:28
clarkbpabelanger: hrm we did update nodepoold for the workers change but its all hidden behind the if args.builder check22:28
*** dprince has quit IRC22:29
*** Goneri has joined #openstack-infra22:29
pabelangerclarkb: ya, pretty sure I tested it locally too22:30
pabelangerwe can try rolling back to see if that helps22:30
clarkband the tests would test it as well22:30
clarkbI don't think it is the problem22:30
clarkbI keep coming back to main thread is crashing and we have a non daemon thread keeping the process running somewhere22:31
clarkbwe could try a ninja try: except: around that signal.pause and log stuff22:31
clarkbor maybe it is crashing before that22:31
pabelangerwhat about nodepoold -d ?22:31
*** zz_dimtruck has joined #openstack-infra22:31
*** zz_dimtruck is now known as dimtruck22:31
*** sridhar_ram1 has joined #openstack-infra22:32
*** HeOS has joined #openstack-infra22:32
greghaynesI really don't see how that change would cause nodepoold to hang22:32
*** sridhar_ram1 has quit IRC22:32
*** ddieterly[away] is now known as ddieterly22:32
greghaynesAre you sure there wasnt another change sitting on disk that hadnt been used due to no restart?22:33
pabelanger/opt/nodepool looks good22:33
pabelangerwe install from there22:33
*** sridhar_ram1 has joined #openstack-infra22:34
clarkbgreghaynes: not 100% sure but reasonably sure22:34
*** roxanaghe has quit IRC22:34
*** sridhar_ram has quit IRC22:34
clarkbgreghaynes: jeblair restarted it during summit for zuul 2.5 stuff22:34
clarkbjeblair: ^ can you confirm22:34
kevinbentonjeblair: hey, do you have any thoughts on gertty tracking a pid to check for multiple instances talking to same DB? I've ruined my local gertty database several times by running multiple on accident22:35
*** ddieterly has quit IRC22:36
greghaynesYea, 294339 just doesnt hit any nodepoold code paths22:36
*** roxanaghe has joined #openstack-infra22:37
clarkbpabelanger: I also doule checked that pbr/pip claim to have hte currently checked out version installed and they do22:37
clarkbso we aren't running some random broken code22:38
clarkb(I don't think)22:38
pabelangerclarkb: Ya, I was hoping we logged pip install some place, but don't see it22:38
clarkbpabelanger: its in syslog for puppet22:38
*** thorst_ has joined #openstack-infra22:39
pabelangerclarkb: really? I cannot see the output of pip install -U /opt/nodepool22:39
clarkbpabelanger: oh you won't get the full output of that there, that goes in ~root/.pip22:39
clarkbyou do get it if it explodes I think22:39
pabelangerokay, checking that22:39
*** rbrndt has quit IRC22:39
clarkbhrm not ~root/.pip22:40
clarkbdoes puppet not set $HOME properly?22:40
pabelangernot sure22:40
*** thorst___ has joined #openstack-infra22:41
clarkbwrapping all of main() in a signal handler and logging that is my next best idea22:41
*** Swami has joined #openstack-infra22:41
clarkblet me write that patch really quick22:41
*** thors____ has joined #openstack-infra22:42
pabelangernot that it help, but here are the last changes in /usr/local/lib/python2.7/dist-packages22:42
greghaynesIs it starting, not logging anything, and just hanging?22:42
*** rbrndt has joined #openstack-infra22:42
clarkbgreghaynes: it is starting but most of the owkr happens in the pool thread22:43
*** thor_____ has joined #openstack-infra22:43
clarkbso it runs and then at some point stops doing things excet reading zmq events according to strace22:43
clarkbwhich is why I think its main thread dying and all daemon threads going iwth it22:43
greghaynesfun, I bet zmq is just a totally separate thread22:43
clarkbpabelanger: running with -d would be potentially anothre way to debug it22:43
greghaynesyea, or just blocking forever22:43
*** thorst_ has quit IRC22:43
clarkbgreghaynes: ya that is other possibility22:43
greghaynesthe next couple changes before this latest one deal with locks22:44
pabelangerclarkb: agreed22:44
*** thorst_ has joined #openstack-infra22:44
*** thorst__ has joined #openstack-infra22:45
openstackgerritClark Boylan proposed openstack-infra/nodepool: Log when nodepoold main excepts and dies
*** thorst___ has quit IRC22:45
clarkbso thats one option ^22:45
clarkbgreghaynes: so maybe jebalir didn't restart it with those changes? or we are somehow able to tickle a deadlock now when we were not ble to before22:46
*** thorst___ has joined #openstack-infra22:46
greghaynesclarkb: Yea, either of those things. I just really dont see it being caused by 29433922:46
*** thors____ has quit IRC22:47
*** thors____ has joined #openstack-infra22:47
clarkbpabelanger: so thats another option, do a wholesale revert install22:47
*** thor_____ has quit IRC22:47
*** tpsilva has quit IRC22:47
*** dmk0202 has quit IRC22:47
fungipabelanger: i'm back around now, sorry... was busy making dinner22:48
clarkb7c9c0516c54097159e1307fc5b5b48467d8faa8b maybe22:48
pabelangerclarkb: I'm find with reverting to confirm the issue is not related to 29433922:48
*** thor_____ has joined #openstack-infra22:48
clarkbuse ^ that commit as the checkout to install22:48
*** thorst_ has quit IRC22:48
clarkbgreghaynes: ^ is that one look right to you based on commit diffs?22:48
*** thorst_ has joined #openstack-infra22:49
*** thorst__ has quit IRC22:49
pabelangerclarkb: sure, /opt/nodepool or local clone22:49
pabelangernot sure what people do when we revert22:49
clarkbpabelanger: /opt/nodepool, just disable puppet22:49
greghaynesclarkb: as a safe starting spot? possibly, need to make sure that the changes in front are ok to back out on...22:49
clarkbpabelanger: my usual practice is stop puppet, git checkout manually, pip install manually22:50
clarkbthen based on that we can reflect it in tree and reenable puppet22:50
*** thorst__ has joined #openstack-infra22:50
pabelangerclarkb: okay, added nodepool.o.o to emergency file22:50
clarkbgreghaynes: the only thing I am worried about is we added a column22:50
clarkbgreghaynes: but that column should be fine as just being in the db unused I think22:50
*** |-paul-| has quit IRC22:50
*** HeOS has quit IRC22:50
*** thorst___ has quit IRC22:51
jeblairwe were running e0f before the latest restart22:51
fungiclarkb: pabelanger: what if it's trying to reach the tripleo cloud? tried commenting that out of the config?22:51
*** thorst___ has joined #openstack-infra22:51
pabelangerfungi: Oh, that is interesting. Since it is down22:51
clarkbfungi: I don't think it is, at least if that was the problem sigusr2 should function22:51
fungistill digesting what's been tried and the current symptoms, so ignore me if i'm in left field on that22:51
clarkbsince we install the signal handler before we start the pool22:52
*** thors____ has quit IRC22:52
clarkbjeblair: ok thanks for confirming22:52
fungibut yeah, just keep in mind that one of our providers is down hard and unreachable right now22:52
*** thors____ has joined #openstack-infra22:52
pabelangerthat should be easy to test22:53
*** thor_____ has quit IRC22:53
*** thorst_ has quit IRC22:53
clarkbya just remove it from the config temporarily22:53
greghaynesIf that doesnt do the trick, sounds like next best option is to try clarkb's debug patch22:54
jeblairyou can also run it in the foreground22:54
*** thorst_ has joined #openstack-infra22:54
pabelangerclarkb: remove or min-ready: -1?22:54
*** thorst__ has quit IRC22:54
fungisuddenly glad we locked down our jenkins masters and are working to excise them entirely...
*** thorst___ has quit IRC22:55
pleia2wow, that's worse than I thought :\22:56
clarkbpabelanger: you have to completely ocmment it out if you don't want it talking to it at all22:56
jeblairthe fist few are features, the last few are :(22:56
*** thorst___ has joined #openstack-infra22:56
clarkbpabelanger: -1 will still make clients and some connections just not do any real work22:56
*** thors____ has quit IRC22:56
pabelangerclarkb: okay, removed22:56
pabelangerstopping nodepool22:56
clarkbpabelanger: I think if that doens't fix it then we try running it in foreground22:56
*** HeOS has joined #openstack-infra22:57
pabelangerclarkb: sure22:57
clarkband if that doesn't help then we move on to the more drastic measures of debug patch or reverts22:57
pabelangerand starting now22:57
pabelangerrunning again22:57
*** thors____ has joined #openstack-infra22:57
*** thor_____ has joined #openstack-infra22:58
*** thorst_ has quit IRC22:58
*** Jeffrey4l has joined #openstack-infra22:59
*** thorst_ has joined #openstack-infra22:59
fungiaside from tracebacks about the unrecognized images from tripleo, the debug log looks sane22:59
greghaynesoh, that did it?22:59
pabelangernot sure, but still running22:59
*** kzaitsev_mb has quit IRC23:00
*** thorst__ has joined #openstack-infra23:00
clarkbthat is really weird if so, sigusr2 is installed before pool is started23:00
* fungi hates being right about this sort of thing23:00
clarkbmaybe we are doing more than we should on object instantiation23:00
*** thorst___ has quit IRC23:00
pabelangerfungi: I'll be happy if you are right!23:00
greghaynesYes, looks like fungi is the winner?23:00
*** thorst___ has joined #openstack-infra23:01
fungiyeah, debug log is pretty unhappy trying to delete some nodes too, all in tripleo cloud i think23:01
clarkbthis should be easy to test with the devstack plugin23:01
fungibut that should be benign just noisy23:01
clarkbmake cloud, turn off cloud, restart nodepool23:01
fungican call it "the tripleo cloud maintenance test"23:01
clarkbwe can likely just remove all those db roews since I doubt the new hardware will continue to host those23:01
*** thors____ has quit IRC23:01
*** HeOS has quit IRC23:01
*** rbrndt has quit IRC23:01
*** jamielennox is now known as jamielennox|away23:02
*** thors____ has joined #openstack-infra23:02
pabelangerI haven't noticed any new nodes coming online yet however23:03
*** thor_____ has quit IRC23:03
clarkbthat may just be nova boot lag23:03
pabelangerbut nodepool is still working23:03
*** thor_____ has joined #openstack-infra23:03
*** thorst_ has quit IRC23:04
greghaynesThat is really crazy that we can block the main thread if a cloud is down... that implies we are somehow connecting via the main thread23:04
*** thorst__ has quit IRC23:04
*** thorst_ has joined #openstack-infra23:04
pabelangerYa, it appears that way23:05
*** hongbin has quit IRC23:05
*** thorst__ has joined #openstack-infra23:05
clarkbits possible the setup for pool which happens before signal handlers is at fault for that23:05
clarkbif we do too much in setup23:05
fungithe only reason i even thought to mention it is that we've seen this behavior before. i thought it was fixed, but now i don't remember what the fix was supposed to be23:05
*** thorst___ has quit IRC23:05
clarkbwe should do setup logging then setup signal handlers, then everything else23:06
*** thorst___ has joined #openstack-infra23:06
greghaynesclarkb: set up? AFAICT its just the constructor for NodePool which is called and that only makes threads and does the statsd connection badness23:06
openstackgerritJoshua Harlow proposed openstack/requirements: Oslo releases for week of may 9th
greghayneser, not makes threads, makes thread conditions23:07
clarkbgreghaynes: ya the threads all connect to clouds23:07
*** thors____ has quit IRC23:07
clarkbbut let me double check23:07
*** thor_____ has quit IRC23:08
*** r-daneel has quit IRC23:08
*** thors____ has joined #openstack-infra23:08
clarkbah yup, it basically does nothing23:08
openstackgerritJoshua Harlow proposed openstack/requirements: Bump uc for oslo.service 1.10.0
greghaynesyea, everything is as a result of run()23:08
clarkbnow I am really confused, maybe python GIL isn't being released to let main() handle signal?23:08
greghaynesoh, you know23:09
clarkbgo down on a read or connect to cloud, block indepfinitely, never hand over thread control23:09
greghayneshrm, actually, no, I was thinking maybe it was blocking on dns resolution which might mess up the GIL23:09
*** thorst_ has quit IRC23:09
greghaynesbut dns should be fine right now23:09
*** thor_____ has joined #openstack-infra23:09
*** thorst__ has quit IRC23:09
clarkbya we have dns servers that should nxdomain if tripleo deleted records23:09
*** kmartin has joined #openstack-infra23:10
*** yuanying has joined #openstack-infra23:10
*** thorst___ has quit IRC23:10
*** thorst__ has joined #openstack-infra23:11
*** IlyaG has quit IRC23:11
clarkbin any case we should be able to reproduce easily enough using the devstakc plugin, I won't get to that now or today though23:12
*** thorst___ has joined #openstack-infra23:12
*** sdague has quit IRC23:12
pabelangerclarkb: do we need to manually purge the database now?23:12
*** gildub has joined #openstack-infra23:12
*** sambetts|afk has quit IRC23:13
*** SergK has quit IRC23:13
clarkbpabelanger: if tripleo doesn't preserve those instances and images, yes23:13
pabelangeror will nodepool recover the deleted nodes eventually (minus tripleo)23:13
fungipabelanger: yeah, just delete from various tables based on %tripleo% pattern23:13
*** thors____ has quit IRC23:13
*** thors____ has joined #openstack-infra23:13
clarkbthe non tripleo stuff should be cleaned up automagically23:13
pabelangerstill not sure why nodepool is not building any nodes yet23:13
*** thor_____ has quit IRC23:13
fungii don't think nodepool is going to clean this up until the tripleo cloud comes back into service otherwise23:13
clarkboh ya it may be dying on the tripleo ones23:13
*** SergK has joined #openstack-infra23:13
pabelangerthat's what I am thinking too23:14
*** kambiz has quit IRC23:14
*** zigo has quit IRC23:14
*** thor_____ has joined #openstack-infra23:14
pabelangerI'm not to confident to do SQL queries on nodepool.o.o, any other takers?23:14
*** pblaho has quit IRC23:14
*** YorikSar has quit IRC23:15
*** mikal has quit IRC23:15
*** Daviey has quit IRC23:15
*** pblaho has joined #openstack-infra23:15
*** mikal has joined #openstack-infra23:15
clarkbI can walk you through it23:15
*** Daviey has joined #openstack-infra23:15
clarkblet me get a sql connection going so I am not blind23:15
*** zigo has joined #openstack-infra23:15
*** sambetts has joined #openstack-infra23:15
*** cburgess has quit IRC23:15
*** thorst_ has joined #openstack-infra23:15
*** YorikSar has joined #openstack-infra23:15
*** cburgess has joined #openstack-infra23:16
*** thorst__ has quit IRC23:16
*** thorst___ has quit IRC23:16
*** thorst__ has joined #openstack-infra23:17
pabelangerso far I have select * from node where provider_name LIKE 'trip%';23:17
clarkbpabelanger: the trick for deleting is you can do a select * with your where first, then if that listing looks good s/select */delete/23:17
*** kambiz has joined #openstack-infra23:17
clarkbso select * from node where provider_name = 'tripleo-test-cloud-rh1';23:17
*** thors____ has quit IRC23:17
*** thorst___ has joined #openstack-infra23:18
clarkbI would avoid like just because we don't have to match multiple values23:18
pabelangerokay, have the select working23:18
*** thor_____ has quit IRC23:18
clarkbpabelanger: if those records look good just replace the select * with delete23:19
*** vhosakot has quit IRC23:19
clarkband that will remove those same rows23:19
pabelangerokay: running23:19
pabelangerdelete from node where provider_name = 'tripleo-test-cloud-rh1';23:19
*** thors____ has joined #openstack-infra23:19
pabelangerQuery OK, 54 rows affected (0.02 sec)23:19
pabelangerconfirmed removed23:19
*** thorst_ has quit IRC23:19
clarkbpabelanger: then select * from snapshot_image where provider_name = 'tripleo-test-cloud-rh1';23:20
pabelangerokay, here we go23:20
clarkband if that looks good reaplce select * with delete23:20
*** thorst_ has joined #openstack-infra23:20
pabelangerQuery OK, 4 rows affected (0.00 sec)23:20
clarkband thats it. we don't do multinode in tripleo so nothing in subnodes tables and dib images are provider independent23:21
*** thorst__ has quit IRC23:21
pabelangerso, looks like nodepool won't build new nodes if it can't delete unknown cloud nodes23:22
*** thorst__ has joined #openstack-infra23:22
*** thorst___ has quit IRC23:22
*** thorst___ has joined #openstack-infra23:23
*** thors____ has quit IRC23:23
*** thor_____ has joined #openstack-infra23:24
*** wajdi has quit IRC23:24
*** roxanaghe has quit IRC23:24
*** thorst_ has quit IRC23:24
pabelangeroh, no. it is blocked again?23:24
pabelangeror just lag for me23:24
*** thorst_ has joined #openstack-infra23:25
*** Liuqing has joined #openstack-infra23:25
pabelangerclarkb: fungi: think the problem is back :(23:25
*** thors____ has joined #openstack-infra23:26
*** thorst__ has quit IRC23:26
*** thorst__ has joined #openstack-infra23:26
*** thorst___ has quit IRC23:27
*** tho______ has joined #openstack-infra23:27
pabelangerseems to happen as soon as a new node comes online ready23:27
*** thor_____ has quit IRC23:28
*** thorst___ has joined #openstack-infra23:28
*** thorst_ has quit IRC23:29
*** IlyaG has joined #openstack-infra23:29
*** thorst_ has joined #openstack-infra23:29
*** thors____ has quit IRC23:30
*** thors____ has joined #openstack-infra23:30
*** Liuqing has quit IRC23:30
*** thorst__ has quit IRC23:31
*** Liuqing has joined #openstack-infra23:31
*** Liuqing has quit IRC23:31
*** thorst__ has joined #openstack-infra23:31
pabelangerI am unsure how to proceed23:31
*** tho______ has quit IRC23:32
*** salv-orl_ has quit IRC23:32
*** Sukhdev has joined #openstack-infra23:32
*** thorst___ has quit IRC23:32
*** thor_____ has joined #openstack-infra23:32
*** thorst___ has joined #openstack-infra23:33
clarkbI think we try the other things like we discussed, start with foreground and see if that exposes any issues23:33
clarkband if that doesn't help move on to debugging patches and or reverts23:33
*** yamahata has quit IRC23:34
*** tho______ has joined #openstack-infra23:34
pabelangerclarkb: okay, let me stop again, and launch in the forground23:34
*** thors____ has quit IRC23:34
*** watersoul_ has joined #openstack-infra23:35
*** thors____ has joined #openstack-infra23:35
*** thorst__ has quit IRC23:36
*** thorst__ has joined #openstack-infra23:36
pabelangerstarted with nodepoold in foreground23:36
*** thor_____ has quit IRC23:36
*** thorst_ has quit IRC23:37
*** watersoul has quit IRC23:37
*** thorst_ has joined #openstack-infra23:37
*** thorst___ has quit IRC23:37
*** thorst___ has joined #openstack-infra23:38
*** tho______ has quit IRC23:38
*** amitgandhinz has quit IRC23:39
*** thor_____ has joined #openstack-infra23:39
*** Qiming_ has joined #openstack-infra23:39
*** thors____ has quit IRC23:40
*** thors____ has joined #openstack-infra23:40
*** thorst__ has quit IRC23:41
*** thorst__ has joined #openstack-infra23:41
*** sridhar_ram1 has quit IRC23:41
*** thorst_ has quit IRC23:41
*** cody-somerville has joined #openstack-infra23:42
*** thorst_ has joined #openstack-infra23:42
*** thorst___ has quit IRC23:42
*** thorst___ has joined #openstack-infra23:43
*** thor_____ has quit IRC23:43
*** johnny__ has quit IRC23:44
openstackgerritPhilip Marc Schwartz proposed openstack-infra/system-config: Add zuul_launcher to openstack_project and run on jenkinsXX hosts.
*** thor_____ has joined #openstack-infra23:44
*** thors____ has quit IRC23:45
*** mriedem has joined #openstack-infra23:45
*** ajo has quit IRC23:45
*** thors____ has joined #openstack-infra23:45
cloudnullclarkb: I've been beating on the OSIC cluster a bit and I can find anything that looks lke its busted. I've been doing bulk builds and floating IP attachments using heat and all of my instances are coming online timely.23:45
clarkbpabelanger: hoping that with it in the foreground if it is main dying you will see the traceback23:45
cloudnullif you find something please let me know.23:45
cloudnulli'd be happy to dig more into it23:45
clarkbcloudnull: ok, I can get you a list of ips and uuids if thta helps but right now we are not building any new hosts due to a nodepool issue so I don't have new data for you23:46
*** thorst__ has quit IRC23:46
*** dmanchad has quit IRC23:46
cloudnulllet me know.23:46
*** thorst__ has joined #openstack-infra23:46
pabelangerclarkb: no, it just locked. Nothing usefully was displayed23:46
*** thorst_ has quit IRC23:46
pabelangerclarkb: I've downgraded paramiko to see if that is the issue23:47
pabelangerclarkb: since things go back once we SSH into a system23:47
*** thorst_ has joined #openstack-infra23:47
cloudnulli should be around this evening and more available tomorrow than i was today23:47
pabelangerclarkb: I think that is the problem, paramiko==2.0.023:47
pabelangerclarkb: as nodes are coming back online now23:47
*** thorst___ has quit IRC23:48
clarkbpabelanger: huh23:48
pabelangerclarkb: I think the issue is related when we SSH into the node to run the script23:48
*** thorst___ has joined #openstack-infra23:48
pabelangerbut downgrading to paramiko==1.16.0, things are working again23:48
clarkbcloudnull: those timestamps are all utc23:49
*** ajo has joined #openstack-infra23:49
fungiwe are getting some floating ip quota exceeded exceptions but presumably unrelated23:49
fungipabelanger: did we end up with paramiko 2.x?23:49
*** thor_____ has quit IRC23:49
pabelangerfungi: ya, I think we'll need to do some cleanups23:49
*** thors____ has quit IRC23:49
fungithat likely came along after the last successful restart23:49
pabelangerfungi: yes23:49
fungigood call23:49
pabelangerso we should likely pin to <2.0 for now23:50
pabelangeruntil we can debug why it is failing23:50
*** thors____ has joined #openstack-infra23:50
*** simondodsley has quit IRC23:50
clarkbpabelanger: ya if this continues to work for longer than before lets get that in23:50
*** thorst__ has quit IRC23:50
*** jamesmca_ has joined #openstack-infra23:50
clarkbthat should also be testable by the devstack plugin23:50
*** jamielennox|away is now known as jamielennox23:51
*** thorst__ has joined #openstack-infra23:51
*** dmanchad has joined #openstack-infra23:51
*** thorst_ has quit IRC23:52
*** aratus has quit IRC23:52
*** thor_____ has joined #openstack-infra23:52
*** thorst___ has quit IRC23:52
*** thorst_ has joined #openstack-infra23:52
openstackgerritDerek Higgins proposed openstack-infra/tripleo-ci: Update testenvs to use SSD's
*** thorst___ has joined #openstack-infra23:53
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Cap paramiko<2.0.0
pabelangerclarkb: fungi: ^23:54
clarkbpabelanger: if you push a revert of that change up on top of it that should get tested by the functional tests23:54
*** jamesmca_ has quit IRC23:54
*** thors____ has quit IRC23:54
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Revert "Cap paramiko<2.0.0"
*** thorst__ has quit IRC23:55
*** thorst__ has joined #openstack-infra23:56
pabelangerwe still need to release puppet, which should re-add triple-o23:56
*** rvasilets__ has quit IRC23:56
*** thor_____ has quit IRC23:56
*** thors____ has joined #openstack-infra23:57
*** thorst_ has quit IRC23:57
*** thor_____ has joined #openstack-infra23:58
*** thorst___ has quit IRC23:58
*** Qiming_ has quit IRC23:58
*** sflanigan has joined #openstack-infra23:58
*** sflanigan has joined #openstack-infra23:58
*** jsavak has joined #openstack-infra23:58
*** thorst_ has joined #openstack-infra23:59
*** dingyichen has joined #openstack-infra23:59

Generated by 2.14.0 by Marius Gedminas - find it at!