Friday, 2018-02-09

*** Daisy_ has quit IRC00:03
ianwcmurphy: ... it does seem stuck again00:06
*** xarses_ has quit IRC00:06
ianwwe've got two's that are orphans ->
*** hamzy_ has joined #openstack-infra00:17
*** felipemonteiro_ has joined #openstack-infra00:17
*** caphrim007_ has joined #openstack-infra00:21
*** gongysh has joined #openstack-infra00:21
*** dave-mccowan has joined #openstack-infra00:22
ianw -> [Zuul] standard output/error still open after child exited00:24
ianwi can only seem to see that in the /tmp/ file log00:24
*** caphrim007 has quit IRC00:24
*** caphrim007_ has quit IRC00:25
*** threestrands has joined #openstack-infra00:27
*** threestrands has quit IRC00:27
*** threestrands has joined #openstack-infra00:27
*** ijw has quit IRC00:32
*** efoley has quit IRC00:34
*** matbu_ has joined #openstack-infra00:36
*** zhurong has joined #openstack-infra00:46
*** xinliang has joined #openstack-infra00:48
*** xinliang has quit IRC00:48
*** xinliang has joined #openstack-infra00:48
*** ijw has joined #openstack-infra00:48
*** matbu_ has quit IRC00:51
openstackgerritJames E. Blair proposed openstack-infra/zuul master: WIP Add zuul from scratch doc
EmilienMcorvus: now I'm curious. Do you have a script that generate all the patches for "Zuul: Remove project name" with a sleep between?00:52
*** gongysh has quit IRC00:52
ianwcmurphy/mordred: started!/story/2001528 about this hang.  i feel like our but then there's a stuck swift process too?00:53
ianwi'll try another run and see if it's the same thing ....00:53
corvusEmilienM: yes, 20 minute sleep in between00:54
*** VW has quit IRC00:54
EmilienMcorvus: nice :)00:54
corvusit has 171 patches left to go00:54
EmilienMcorvus: much better than my bash script that run all in once. Next time I'll add sleeps00:54
*** VW has joined #openstack-infra00:54
corvusEmilienM: at AJaeger's suggestion, it does all master branches first, then others (so hopefully some changes will land on queens in time)00:54
corvusi also shuffiled the projects, so not all of the puppet projects get changes in sequence.  that way you don't have to worry as much about when to approve -- you can just approve them as they come in00:55
*** ijw has quit IRC00:56
corvusi have some ideas about how to make the configs more memory efficient, but i won't be able to get to that until probably late march or april at the earliest, so we deal with it the best we can for now :)00:56
*** harlowja has joined #openstack-infra00:57
EmilienMsleep driven deployment00:57
EmilienMwelcome to tripleo00:57
corvusmiraculously, i think i've uploaded 490 changes, and 378 of them are already closed00:59
*** daidv_ has quit IRC01:05
*** dhill_ has quit IRC01:05
*** slaweq has joined #openstack-infra01:06
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Default min-ready to 0
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Store build logs automatically
*** xinliang has quit IRC01:07
*** aviau has quit IRC01:08
*** aviau has joined #openstack-infra01:08
*** tosky has quit IRC01:09
*** slaweq has quit IRC01:10
*** zhurong has quit IRC01:11
*** verdurin has quit IRC01:12
*** verdurin_ has joined #openstack-infra01:12
*** zhurong has joined #openstack-infra01:14
*** xinliang has joined #openstack-infra01:20
*** xinliang has quit IRC01:20
*** xinliang has joined #openstack-infra01:20
*** xarses has joined #openstack-infra01:21
*** xinliang has quit IRC01:26
*** threestrands has quit IRC01:29
*** hongbin has joined #openstack-infra01:32
*** caphrim007 has joined #openstack-infra01:34
*** Jeffrey4l has joined #openstack-infra01:34
*** Daisy_ has joined #openstack-infra01:37
*** caphrim007 has quit IRC01:38
*** Nil_ has quit IRC01:38
openstackgerritKendall Nelson proposed openstack-infra/project-config master: Enable StoryBoard Use for Election Repo
*** Daisy_ has quit IRC01:39
openstackgerritHongbin Lu proposed openstack-infra/irc-meetings master: Zun holds team meeting at #openstack-zun
*** Daisy_ has joined #openstack-infra01:39
*** olaph has quit IRC01:41
*** olaph has joined #openstack-infra01:41
*** xinliang has joined #openstack-infra01:43
*** xinliang has quit IRC01:43
*** xinliang has joined #openstack-infra01:43
*** r-daneel has quit IRC01:45
*** Jeffrey4l_ has joined #openstack-infra01:46
*** Jeffrey4l has quit IRC01:47
dmsimardcorvus: your patch for the project name in ara made me realize about a regression I hadn't seen because no patches were sent for a while :/01:48
dmsimardSo corvus++01:48
*** caphrim007 has joined #openstack-infra01:48
*** Swami has quit IRC01:51
ianwcmurphy / AJaeger : i strongly suspect that swift-init does not daemonize correctly; should (maybe?) tell us.  this poses a range of questions; like why only on suse, why ansible doesn't clean up, etc etc01:52
*** cuongnv has joined #openstack-infra01:54
*** salv-orlando has quit IRC01:56
notmynameianw: FWIW mattoliverau is taking a look02:00
mattoliverauianw: I'll go throw up a suse swift dev env and have a poke02:01
ianwhmm, let me see if i can hold this node02:01
ianwwell maybe it will be held when it fails i hope, not too sure02:05
ianwmattoliverau want to give me a public key and i can give you login?02:05
mattoliverauI can at least test swift-init on a suse vm and see if it also has problems with container-sync.02:07
*** slaweq has joined #openstack-infra02:11
*** slaweq has quit IRC02:16
ianwohh, 542535 stuffs up and doesn't actually start the container-sync process ... but it does show that if it *isn't* running things work, so, yeah02:26
*** liujiong has joined #openstack-infra02:30
*** namnh has joined #openstack-infra02:37
*** mriedem has quit IRC02:44
*** kiennt26 has joined #openstack-infra02:46
*** slaweq has joined #openstack-infra02:49
*** hamzy_ has quit IRC02:53
*** slaweq has quit IRC02:53
*** hamzy_ has joined #openstack-infra02:58
*** annp has joined #openstack-infra03:07
*** VW has quit IRC03:08
*** VW has joined #openstack-infra03:09
*** cuongnv has quit IRC03:09
*** cuongnv has joined #openstack-infra03:09
*** jlabarre has quit IRC03:21
*** olaph1 has joined #openstack-infra03:26
*** olaph has quit IRC03:29
*** suhdood has joined #openstack-infra03:29
openstackgerritMerged openstack-infra/irc-meetings master: Zun holds team meeting at #openstack-zun
*** RuiChen has quit IRC03:37
*** Daisy_ has quit IRC03:39
*** nicolasbock has quit IRC03:39
*** Daisy_ has joined #openstack-infra03:39
prometheanfiretonyb: regarding your email, shouldn't you ask here?03:42
*** Daisy_ has quit IRC03:44
*** ykarel|away has joined #openstack-infra03:44
*** ykarel|away is now known as ykarel03:45
tonybprometheanfire: Perhaps I thought the norm now was to do it over there03:46
*** liujiong has quit IRC03:46
*** dave-mccowan has quit IRC03:48
prometheanfireeh, I'd do both, especially as the releases are more time sensitive now03:48
*** nicolasbock has joined #openstack-infra03:49
*** yee37934 has joined #openstack-infra03:51
*** yee379 has quit IRC03:51
tonybprometheanfire: this is for ocata so isn't time-sensitive03:55
prometheanfireah, k then03:55
*** nicolasbock has quit IRC03:56
*** ramishra has joined #openstack-infra03:59
*** rosmaita has quit IRC04:02
*** gongysh has joined #openstack-infra04:04
*** psachin has joined #openstack-infra04:04
*** felipemonteiro__ has joined #openstack-infra04:05
*** felipemonteiro__ has quit IRC04:05
*** felipemonteiro__ has joined #openstack-infra04:06
*** felipemonteiro_ has quit IRC04:08
*** kiennt26 has quit IRC04:11
*** namnh has quit IRC04:13
*** cuongnv has quit IRC04:13
*** cuongnv has joined #openstack-infra04:16
*** slaweq has joined #openstack-infra04:16
*** slaweq has quit IRC04:21
*** cuongnv has quit IRC04:23
*** cuongnv has joined #openstack-infra04:24
*** gongysh has quit IRC04:26
*** gongysh has joined #openstack-infra04:26
*** kiennt26 has joined #openstack-infra04:27
*** gongysh has quit IRC04:27
*** slaweq has joined #openstack-infra04:29
*** felipemonteiro__ has quit IRC04:33
*** slaweq has quit IRC04:34
*** zhurong has quit IRC04:35
*** cuongnv has quit IRC04:37
*** suhdood has quit IRC04:37
ianwtonyb: should be enqueued04:38
tonybianw: Thanks04:38
ianwcorvus: i also might have noticed something; while zuul enqueue-ref was waiting, it seems like the status page was inaccessible?  both rpc calls right; does one block the other?04:39
*** VW has quit IRC04:48
*** VW has joined #openstack-infra04:49
ianwtonyb: actually, trying again in release-post ...04:50
*** VW has quit IRC04:50
*** VW has joined #openstack-infra04:50
*** links has joined #openstack-infra04:52
*** links has quit IRC04:54
*** gongysh has joined #openstack-infra04:54
*** dhajare has joined #openstack-infra04:57
*** ijw has joined #openstack-infra05:01
*** Daisy_ has joined #openstack-infra05:02
*** Daisy_ has quit IRC05:02
*** Daisy_ has joined #openstack-infra05:02
*** links has joined #openstack-infra05:03
openstackgerritIan Wienand proposed openstack-infra/project-config master: tools/ : add some arm64 notes
*** ijw has quit IRC05:06
*** links has quit IRC05:10
*** harlowja has quit IRC05:10
*** coolsvap has joined #openstack-infra05:10
*** andreas_s has joined #openstack-infra05:11
*** links has joined #openstack-infra05:12
*** claudiub|2 has joined #openstack-infra05:13
*** hongbin has quit IRC05:13
*** gongysh has quit IRC05:15
*** andreas_s has quit IRC05:17
*** VW has quit IRC05:18
*** VW has joined #openstack-infra05:19
*** salv-orlando has joined #openstack-infra05:21
*** VW has quit IRC05:23
*** auristor has quit IRC05:23
*** jchhatbar has joined #openstack-infra05:24
*** salv-orlando has quit IRC05:26
*** auristor has joined #openstack-infra05:26
*** harlowja has joined #openstack-infra05:35
*** slaweq has joined #openstack-infra05:35
*** zhurong has joined #openstack-infra05:36
*** jchhatbar is now known as janki05:37
*** harlowja has quit IRC05:39
*** slaweq has quit IRC05:40
*** dhajare has quit IRC05:53
*** aeng has quit IRC05:55
*** salv-orlando has joined #openstack-infra05:59
*** VW has joined #openstack-infra06:01
*** pgadiya has joined #openstack-infra06:03
*** e0ne has joined #openstack-infra06:05
*** VW has quit IRC06:06
*** e0ne has quit IRC06:06
dirkAJaeger: cmurphy : sorry, I preferred sleeping. What's the conclusion so far?06:18
AJaegerdirk: best ask mattoliverau as well, looked like a swift init problem. I just glanced over scrollback06:23
*** pcichy has joined #openstack-infra06:24
*** pbourke has quit IRC06:30
*** VW has joined #openstack-infra06:30
dirkAJaeger: mattoliverau : ah, that rings a bell.. we  had issues with Swift-init already in the past06:30
*** VW has quit IRC06:35
openstackgerritKota Tsuyuzaki proposed openstack-infra/project-config master: Add release-note-job to storlets project
*** VW has joined #openstack-infra06:38
*** salv-orlando has quit IRC06:39
*** dhajare has joined #openstack-infra06:39
*** cuongnv has joined #openstack-infra06:40
*** VW has quit IRC06:42
openstackgerritMerged openstack-infra/nodepool master: Add default logging configuration
*** slaweq has joined #openstack-infra06:45
*** slaweq has quit IRC06:50
openstackgerritMerged openstack-infra/nodepool master: Exit launchers and builders immediately
openstackgerritMerged openstack-infra/nodepool master: Exercise pidfile before daemonizing
openstackgerritMerged openstack-infra/nodepool master: Resolve paths before demonization
openstackgerritMerged openstack-infra/nodepool master: Default min-ready to 0
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Move release-tools out of jenkins/scripts
*** xarses has quit IRC07:01
*** andreas_s has joined #openstack-infra07:03
*** namnh has joined #openstack-infra07:03
*** olaph has joined #openstack-infra07:04
*** olaph1 has quit IRC07:05
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Consolidate legacy scripts
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Move more slave scripts over from project-config
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove some moved slave scripts
*** olaph1 has joined #openstack-infra07:10
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Move more slave scripts over from project-config
*** olaph has quit IRC07:12
*** VW has joined #openstack-infra07:12
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove some moved slave scripts
ianwdirk: yep, see and the storyboard bug in there07:16
*** rcernin has quit IRC07:17
*** VW has quit IRC07:17
ianwcorvus: another potential issue; i'm not sure if TIMED_OUT nodes get caught by the autohold matching.  at least it didn't seem to for the suse one i've setup07:18
*** snapiri has quit IRC07:19
AJaegerianw: you've given a -1 on - but that has a depends-on, so should be fine to not merge by accident. Or do you wait for another change?07:20
ianwAJaeger: oh, what i meant was that the change actually has to be in executors, which requires restarting them all?07:21
AJaegerianw: I would +2A andreaf's
AJaegerianw: really? that's a zuul-jobs change.07:22
ianwoh, maybe i misread it07:23
AJaegerplease double check - and if you agree, let's merge 54048507:23
ianwyep, thanks07:23
ianwok, i'm about to disappear so don't want to merge zuul-jobs :)  too scary07:24
AJaegerianw: have a nice weekend!07:24
ianwcheers, ttyl07:24
*** slaweq has joined #openstack-infra07:25
*** eumel8 has joined #openstack-infra07:26
*** e0ne has joined #openstack-infra07:27
*** pcaruana has joined #openstack-infra07:35
*** florianf has joined #openstack-infra07:36
*** gongysh has joined #openstack-infra07:38
mattoliverauYeah, I gave some suggestions, try with a -w so swift-ring-builder doesn't try bad hold a pipe to the daemon it's launching and see if that helps, or use run_process or whatever it's called in devstack (on my phone so can't look it up)07:39
mattoliverauSorry, swift-init.07:39
mattoliverauSomehow swift-ring-builder autocompletes in my phone.. curse of a swift dev me thinks :p07:40
openstackgerritMerged openstack-infra/zuul-jobs master: Change the list of extensions to a dict
AJaegerandreaf: ^07:48
*** alexchadin has joined #openstack-infra07:48
*** VW has joined #openstack-infra07:49
*** VW has quit IRC07:54
openstackgerritFrank Kloeker proposed openstack-infra/project-config master: Setup translation jobs for Queens
eumel8good morning, eu-infra: ^^ Horizon has release branch and we want to setup translation jobs asap, thx :)07:59
openstackgerritSpyros Trigazis (strigazi) proposed openstack-infra/project-config master: Switch magnum to release-openstack-server
*** snapiri has joined #openstack-infra08:06
*** kjackal has joined #openstack-infra08:06
*** dtantsur|afk is now known as dtantsur08:08
*** vaidy has quit IRC08:08
*** isviridov_away has quit IRC08:08
*** jpena|off is now known as jpena08:10
openstackgerritHoang Trung Hieu proposed openstack/diskimage-builder master: Add iscsi-boot element
*** pcichy has quit IRC08:13
*** pcichy has joined #openstack-infra08:14
*** amoralej|off is now known as amoralej08:16
*** ralonsoh has joined #openstack-infra08:17
*** olaph has joined #openstack-infra08:18
*** aviau has quit IRC08:19
*** olaph1 has quit IRC08:19
*** aviau has joined #openstack-infra08:19
AJaegereumel8: - I suggest to remove neutron-lbaas-dashboard from the list08:23
*** makowals has quit IRC08:23
*** tesseract has joined #openstack-infra08:26
*** jchhatbar has joined #openstack-infra08:27
*** namnh has quit IRC08:27
*** VW has joined #openstack-infra08:28
*** janki has quit IRC08:29
*** d0ugal has quit IRC08:33
*** VW has quit IRC08:34
eumel8AJaeger: just wondering why there are a stable/queens branch in the repo08:35
*** jchhatbar has quit IRC08:41
*** jchhatbar has joined #openstack-infra08:41
*** rossella_s has quit IRC08:42
*** hashar has joined #openstack-infra08:43
*** d0ugal has joined #openstack-infra08:47
AJaegereumel8: it's not dead yet. So, translation is fine - I just would not put it into the priority list. Your call...08:49
*** jchhatbar has quit IRC08:54
*** armaan has joined #openstack-infra08:54
*** jchhatbar has joined #openstack-infra08:54
openstackgerritFrank Kloeker proposed openstack-infra/project-config master: Setup translation jobs for Queens
eumel8AJaeger: got it, thx08:55
*** gfidente has joined #openstack-infra08:56
*** pbourke has joined #openstack-infra08:57
*** jchhatbar has quit IRC09:01
*** jchhatbar has joined #openstack-infra09:01
AJaegereumel8: in the past we stopped translation of oldest releases when new one opened - do you want to keep ocata translations? There might not be releases done for those anymore09:01
*** jpich has joined #openstack-infra09:02
*** VW has joined #openstack-infra09:03
eumel8AJaeger: I want to recheck all repos later the day. Ocata EOL is 2018-02-26. Then we would remove the sync as usualy.09:07
*** alexchadin has quit IRC09:07
*** VW has quit IRC09:08
*** alexchadin has joined #openstack-infra09:08
*** ykarel is now known as ykarel|lunch09:08
*** ajo has quit IRC09:14
*** ajo has joined #openstack-infra09:14
*** zhurong has quit IRC09:16
*** Jeffrey4l_ has quit IRC09:17
*** Jeffrey4l has joined #openstack-infra09:18
openstackgerritliumengke proposed openstack-infra/system-config master: fix somes typos in doc file.
*** Daisy_ has quit IRC09:21
*** hoangcx has joined #openstack-infra09:22
strigaziAJaeger: a lot of spam in openstack-container09:22
strigaziAJaeger: a lot of spam in openstack-containers09:22
AJaegerstrigazi: sorry to hear ;( Nothing I can help with...09:23
*** dbecker has joined #openstack-infra09:23
Diabelkofungi: circling back to our discussion from yesterday - would multi-node setup or bigger flavor be possible to have?09:23
strigaziAJaeger: who should I ping? I though someone from infra team was blocking them, no?09:24
AJaegerstrigazi: irc channel or mailing list?09:25
strigaziAJaeger channel09:25
AJaegerstrigazi: we've done some temporary blocking of not authenticated users - no idea how ;(09:26
*** hjensas|afk is now known as hjensas09:26
*** jchhatbar has quit IRC09:27
*** jchhatbar has joined #openstack-infra09:27
AJaegerstrigazi, run /msg chanserv access #openstack-containers list09:27
AJaegerit shows who's op and can block people09:27
AJaegerstrigazi: and there's
strigaziAJaeger: ttx and yolanda are in the list and in this timezone, can I add myself in project-config?09:29
AJaegerstrigazi: you should ask one of them to give you the permissions for that specific channel09:30
strigaziAJaeger: thanks09:30
AJaegerI see it's silent again - but let's be prepared for next time...09:30
strigaziAJaeger: thanks again Andreas, you're very helpful as always09:31
AJaegerstrigazi: you're welcome09:31
*** jchhatbar has quit IRC09:34
*** efoley has joined #openstack-infra09:34
*** jchhatbar has joined #openstack-infra09:34
*** VW has joined #openstack-infra09:36
*** jchhatbar is now known as janki09:37
*** edmondsw has joined #openstack-infra09:40
*** VW has quit IRC09:41
Diabelkofungi: I'm asking, because right now we use 10vCPU/32GB for our unittests/systets, however we should be able to fit into something like 4-8vCPU and 16GB of RAM09:41
*** dprince has joined #openstack-infra09:41
Diabelko(and we use full deployment for those)09:42
Diabelkoand if multinode is possible, two 8vcpu/8gb nodes would be fine09:42
AJaegerDiabelko: fungi is US east coast based, he won't be around normally before 14:00 UTC09:43
DiabelkoI know ;)09:43
AJaegerDiabelko: he might read backscroll and then answer asynchronously ;)09:43
DiabelkoI'm just writing that so I won't forget09:43
AJaegerDiabelko: Sure, go ahead...09:43
*** olaph1 has joined #openstack-infra09:43
Diabelkothat's fine, I'll be available in the evening too, so we'll talk later09:43
*** olaph has quit IRC09:44
*** edmondsw has quit IRC09:45
*** shardy has joined #openstack-infra09:49
*** sambetts|afk is now known as sambetts09:50
*** tpsilva has joined #openstack-infra09:50
*** pcichy has quit IRC09:53
*** pgadiya has quit IRC09:54
*** pgadiya has joined #openstack-infra09:54
*** derekh has joined #openstack-infra10:01
*** hoangcx has quit IRC10:03
*** pcichy has joined #openstack-infra10:04
*** gongysh has quit IRC10:06
*** ihrachys has quit IRC10:11
*** ykarel|lunch is now known as ykarel10:11
cmurphyI'm so popular today10:14
*** VW has joined #openstack-infra10:16
*** VW has quit IRC10:20
*** dprince has quit IRC10:20
*** pcichy has quit IRC10:21
*** cuongnv has quit IRC10:22
*** annp has quit IRC10:22
*** kjackal has quit IRC10:23
*** kjackal has joined #openstack-infra10:24
*** zoli is now known as zoli|lunch10:28
*** zoli|lunch is now known as zoli10:28
*** ldnunes has joined #openstack-infra10:29
*** alexchadin has quit IRC10:31
*** gongysh has joined #openstack-infra10:35
openstackgerritBalazs Gibizer proposed openstack-infra/project-config master: consolidate nova job definitions
*** efoley has quit IRC10:36
*** dtantsur is now known as dtantsur|bbl10:43
*** efoley has joined #openstack-infra10:43
*** ihrachys has joined #openstack-infra10:44
*** efoley_ has joined #openstack-infra10:44
*** efoley has quit IRC10:44
andreafAJaeger \o/ thanks :)10:45
*** armaan has quit IRC10:46
AJaegerandreaf: let's get the three changes in quickly - before we branch those repos ;)10:46
*** VW has joined #openstack-infra10:47
*** kjackal has quit IRC10:47
andreafAJaeger: heh we won't branch tempest - but good yeah I'm all for getting them merged in10:48
andreafAJaeger: I think there are enough +1 / +2 on the devstack one, I'll just approve it10:49
*** VW has quit IRC10:52
*** chandankumar has quit IRC10:54
*** tosky has joined #openstack-infra10:54
*** gongysh has quit IRC10:56
AJaegerandreaf: the networking-odl one is important as well - see
*** lucas-afk is now known as lucasagomes10:57
andreafAJaeger: yeah well I have no +2 power there but I left a comment10:58
*** d0ugal has quit IRC10:59
AJaegerandreaf: thanks - mpeterson, do you want to ask the networking-odl cores? I just added them to the review...11:00
andreafAJaeger: I'm looking to add qa-release to devstack-container-plugin ACLs once merges11:05
andreafAJaeger: but I cannot find qa-release anywhere in the ACLs in git11:05
andreafAJaeger: could it be that it has been added manually to other repos and not reflected in git? is there a way to search in gerrit for that?11:06
andreafAJaeger: because I think qa-release is how QA PTLs get +2 on all qa repos so I think it is used11:07
AJaegerandreaf: indeed, qa-release does not appear in project-config - and neither in its history11:08
*** pbourke has quit IRC11:08
AJaegerandreaf: what is causing a problem for you here?11:09
andreafAJaeger: I guess it's because it's added to the projects release group,members11:10
andreafAJaeger: no I was just wondering how to make sure QA PTLs have +2 on the new QA project but I guess it will just be a matter of adding the qa-release group to their release group11:10
AJaegerwait. Why do you need release group for the repo?11:11
AJaegeras it's soon an official team, the release team can tag for you - no need to setup release ACLs anymore11:12
AJaegerandreaf: for +2:,members11:12
AJaegerIt includes devstack-core - so, all should be set, isn't it?11:13
AJaeger(and devstack-core includes qa-release)11:13
AJaegerBut you can add other groups there *already* manually11:13
AJaegerSo, looks like nothing needs to be done - agreed?11:14
andreafAJaeger: ah yeah it's already there11:14
andreafAJaeger never mind sorry to bother you for nothing :P11:14
*** d0ugal has joined #openstack-infra11:16
*** kiennt26 has quit IRC11:16
*** chandankumar has joined #openstack-infra11:17
*** VW has joined #openstack-infra11:20
*** pbourke has joined #openstack-infra11:20
robcresswell@AJaeger Hey, just an observation; the twitter bouncer for the Infra updates splits messages at the old ~140 char limit. I think 280 is standard for everyone now, so that could be adjusted to make consumption of those messages a little nicer. Not sure who / how thats managed. If its open, I can take a look.11:23
*** eroux has quit IRC11:24
*** VW has quit IRC11:24
*** alexchadin has joined #openstack-infra11:28
*** rfolco|off is now known as rfolco|rover11:30
AJaegerrobcresswell: see
AJaegerrobcresswell: that's all I know - and 280 char support would be appreciated!11:32
*** kiennt26 has joined #openstack-infra11:32
*** eroux has joined #openstack-infra11:34
*** armaan has joined #openstack-infra11:39
*** kiennt26 has quit IRC11:41
robcresswellAJaeger: Looks like its in an upstream lib:
robcresswellI'll go digging :)11:44
*** hjensas has quit IRC11:44
AJaegerthanks, robcresswell11:45
*** hjensas has joined #openstack-infra11:45
*** hjensas has quit IRC11:45
*** hjensas has joined #openstack-infra11:45
*** kjackal has joined #openstack-infra11:46
*** olaph has joined #openstack-infra11:47
*** nicolasbock has joined #openstack-infra11:47
*** olaph1 has quit IRC11:47
robcresswellAJaeger: Just to report back; feature was added in but PyPI doesn't have 3.3.1. There's an open issue ( to publish the new version. So it's supported, just waiting on a release. Thanks for the earlier link, that really helped get me started.11:51
*** VW has joined #openstack-infra11:54
*** VW has quit IRC11:59
*** alexchadin has quit IRC12:11
*** alexchadin has joined #openstack-infra12:12
*** jlabarre has joined #openstack-infra12:13
*** alexchadin has quit IRC12:16
*** alexchadin has joined #openstack-infra12:17
*** alexchadin has quit IRC12:20
*** alexchadin has joined #openstack-infra12:21
*** VW has joined #openstack-infra12:25
AJaegerrobcresswell: thanks for digging into this!12:26
robcresswellYou're welcome :)12:27
*** VW has quit IRC12:30
*** simonmcc has quit IRC12:34
*** simonmcc has joined #openstack-infra12:35
*** aviau has quit IRC12:35
*** aviau has joined #openstack-infra12:36
*** efoley_ has quit IRC12:37
*** dtantsur|bbl is now known as dtantsur12:40
*** dangers_away has quit IRC12:42
*** vabada has joined #openstack-infra12:46
*** ramishra has quit IRC12:48
*** serverascode has quit IRC12:48
*** serverascode has joined #openstack-infra12:49
*** ameade has quit IRC12:50
*** ameade has joined #openstack-infra12:51
*** olaph1 has joined #openstack-infra12:52
*** olaph has quit IRC12:53
*** armaan has quit IRC12:56
*** VW has joined #openstack-infra13:03
*** agopi_out has quit IRC13:05
*** VW has quit IRC13:07
*** jcoufal has joined #openstack-infra13:08
*** kiennt26 has joined #openstack-infra13:08
*** rfolco|rover is now known as rfolco|lunch13:12
*** tesseract has quit IRC13:12
*** kiennt26 has quit IRC13:12
*** vabada has quit IRC13:14
*** fdegir has quit IRC13:14
*** jpena is now known as jpena|lunch13:15
*** fdegir has joined #openstack-infra13:15
*** kklimonda has quit IRC13:15
*** kklimonda has joined #openstack-infra13:16
*** ramishra has joined #openstack-infra13:17
*** psachin has quit IRC13:22
*** psachin has joined #openstack-infra13:24
*** kiennt26 has joined #openstack-infra13:26
*** ameeda has quit IRC13:28
*** xyang has quit IRC13:33
*** xyang has joined #openstack-infra13:34
*** betherly has quit IRC13:34
*** betherly has joined #openstack-infra13:35
*** olaph has joined #openstack-infra13:36
*** olaph1 has quit IRC13:36
*** alexchadin has quit IRC13:39
*** gibi is now known as giblet13:43
*** tpsilva has quit IRC13:43
*** tpsilva has joined #openstack-infra13:43
*** olaph1 has joined #openstack-infra13:44
*** alexchadin has joined #openstack-infra13:44
*** bauzas is now known as bauwser13:45
openstackgerritMarc Aubry proposed openstack-infra/project-config master: Create stable/* specific ACL
*** dave-mccowan has joined #openstack-infra13:46
*** olaph has quit IRC13:46
*** alexchadin has quit IRC13:47
*** alexchadin has joined #openstack-infra13:49
*** trown|outtypewww is now known as trown13:49
*** psachin has quit IRC13:49
*** alexchadin has quit IRC13:52
*** vdrok has quit IRC13:52
*** vdrok has joined #openstack-infra13:53
*** dave-mcc_ has joined #openstack-infra13:53
*** edmondsw has joined #openstack-infra13:54
*** dsariel has quit IRC13:55
*** dave-mccowan has quit IRC13:55
*** yamamoto has quit IRC13:56
*** NobodyCam has quit IRC13:56
*** NobodyCam has joined #openstack-infra13:56
*** yamamoto has joined #openstack-infra13:56
*** links has quit IRC13:56
*** yamamoto has quit IRC13:57
*** yamamoto has joined #openstack-infra13:57
*** mriedem has joined #openstack-infra13:57
*** dhajare has quit IRC13:58
*** agopi_out has joined #openstack-infra14:00
*** alexchadin has joined #openstack-infra14:01
*** TheJulia has quit IRC14:02
*** TheJulia has joined #openstack-infra14:02
openstackgerritMerged openstack-infra/grafyaml master: Add support to change the default time filter range
*** johnsom has quit IRC14:03
*** rosmaita has joined #openstack-infra14:03
*** Goneri has joined #openstack-infra14:04
*** johnsom has joined #openstack-infra14:04
*** vaidy has joined #openstack-infra14:05
*** isviridov_away has joined #openstack-infra14:06
*** agopi_out has quit IRC14:08
*** dsariel has joined #openstack-infra14:08
AJaegerconfig-core, what do you think of these further slave_scripts cleanups? and - and then . Reviews wwelcome14:10
openstackgerritMerged openstack-infra/grafyaml master: Add yaxes format for newer versions of Grafana
*** rfolco|lunch is now known as rfolco|rover14:14
*** ethfci_ has joined #openstack-infra14:15
*** ethfci has quit IRC14:15
*** dave-mcc_ has quit IRC14:17
*** jpena|lunch is now known as jpena14:18
pabelangerWe have a snowstorm rolling through today, in case i disappear for a while14:19
*** amoralej is now known as amoralej|lunch14:20
*** ethfci_ has quit IRC14:21
*** olaph has joined #openstack-infra14:22
*** ethfci has joined #openstack-infra14:22
*** pgadiya has quit IRC14:23
*** olaph1 has quit IRC14:23
*** weshay is now known as weshay|ruck14:24
*** sshnaidm has quit IRC14:24
*** tesseract has joined #openstack-infra14:28
*** slaweq has quit IRC14:28
*** slaweq has joined #openstack-infra14:29
*** david-lyle has quit IRC14:29
*** dave-mccowan has joined #openstack-infra14:30
*** VW has joined #openstack-infra14:30
*** VW has quit IRC14:30
*** ramishra has quit IRC14:31
*** VW has joined #openstack-infra14:31
*** coolsvap has quit IRC14:33
*** slaweq has quit IRC14:33
*** dsariel has quit IRC14:35
*** eharney has joined #openstack-infra14:36
*** stephenfin is now known as finucannot14:38
*** pcichy has joined #openstack-infra14:40
*** kmalloc has quit IRC14:40
*** dave-mccowan has quit IRC14:40
*** kmalloc has joined #openstack-infra14:40
*** lucasagomes is now known as lucas-hungry14:43
*** berendt has quit IRC14:46
*** berendt has joined #openstack-infra14:46
dhellmannpabelanger : do you have a second to take a look at before that blizzard hits?14:51
*** hogepodge has quit IRC14:51
*** vabada has joined #openstack-infra14:52
*** hogepodge has joined #openstack-infra14:52
*** wolverineav has joined #openstack-infra14:52
*** vabada has quit IRC14:52
*** vabada has joined #openstack-infra14:53
*** vabada has quit IRC14:53
*** kiennt26 has quit IRC14:54
AJaegerpabelanger: could you review as well, please?14:54
*** vabada has joined #openstack-infra14:54
*** kiennt26 has joined #openstack-infra14:54
*** ykarel is now known as ykarel|away14:56
*** r-daneel has joined #openstack-infra14:57
openstackgerritMerged openstack-infra/project-config master: Switch magnum to release-openstack-server
*** salv-orlando has joined #openstack-infra14:59
*** salv-orlando has quit IRC14:59
fungiDiabelko: i'm still catching up on scrollback, but yes we have plenty of jobs which utilize multiple nodes (for a variety of reasons, but distributing work across them would certainly be a valid one). zuul now supports jobs defining arbitrarily large sets of nodes for a job15:00
*** hjensas is now known as hjensas|afk15:00
*** efoley has joined #openstack-infra15:01
*** dangers_away has joined #openstack-infra15:01
*** ykarel|away has quit IRC15:01
fungirobcresswell: AJaeger: yeah, i don't know how clearly we've documented the fact but we have been waiting for upstream support to appear in the package on pypi before we can up the message length. known issue, sorry it wasn't more readily discoverable :/15:01
*** agopi_out has joined #openstack-infra15:02
*** andreas_s has quit IRC15:03
*** dave-mccowan has joined #openstack-infra15:03
openstackgerritAlex Krzos proposed openstack-infra/grafyaml master: Allow dashboards to be tagged
*** andreas_s has joined #openstack-infra15:04
*** auristor has quit IRC15:04
AJaegerthanks, pabelanger15:05
*** hongbin has joined #openstack-infra15:06
*** agopi_out is now known as agopi15:07
*** amoralej|lunch is now known as amoralej15:07
*** alexchadin has quit IRC15:08
*** andreas_s has quit IRC15:08
*** felipemonteiro__ has joined #openstack-infra15:12
AJaegerfungi, could you review as well, please?15:13
*** caphrim007 has quit IRC15:13
*** auristor has joined #openstack-infra15:13
fungisure, may happen after the release meeting15:16
openstackgerritDoug Hellmann proposed openstack-infra/project-config master: tacker-horizon seems to need the horizon-specific release jobs
fungiAJaeger: i'll trade you for 542879 ;)15:19
fungirelease blocker, looks like15:19
* fungi checks for the failed job logs15:20
*** felipemonteiro_ has joined #openstack-infra15:21
openstackgerritMerged openstack-infra/grafyaml master: Allow dashboards to be tagged
fungidhellmann: was tacker-horizon caught through manual inspection? i don't see it on the release-job-failures ml15:22
dhellmannfungi : the auto-validation detected the potential issue, and inspecting the repo I see horizon in the tox.ini deps list15:22
fungidhellmann: thanks for confirming!15:23
dhellmannfungi : thanks for checking my math, so to speak15:23
*** felipemonteiro__ has quit IRC15:25
*** slaweq has joined #openstack-infra15:25
openstackgerritAlex Krzos proposed openstack-infra/grafyaml master: Add links to GrafYaml
*** slaweq has quit IRC15:30
*** david-lyle has joined #openstack-infra15:30
*** lucas-hungry is now known as lucasagomes15:36
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Consolidate legacy scripts
openstackgerritMerged openstack-infra/openstack-zuul-jobs master: Move more slave scripts over from project-config
*** salv-orlando has joined #openstack-infra15:47
*** slaweq has joined #openstack-infra15:48
openstackgerritJames E. Blair proposed openstack-infra/nodepool master: Store build logs automatically
*** david-lyle has quit IRC15:51
*** camunoz has joined #openstack-infra15:53
*** hashar is now known as hasharAway15:54
*** david-lyle has joined #openstack-infra15:55
*** kiennt26 has quit IRC15:56
mtreinishinfra-root: we pushed a new subunit2sql release last night, and I think the subunit gearman workers have all fallen over15:57
mtreinishI'm not seeing any events on firehose from the workers any more and the db doesn't have any recent entries15:57
mtreinishinfra-root: if someone gets a sec could they take a look15:58
*** r-daneel has quit IRC15:59
*** ijw has joined #openstack-infra16:00
*** hasharAway has quit IRC16:00
fungiinfra-root: i've confirmed the mailman versions listed at are installed on and now (unattended-upgrades for the former, i manually did the latter preemptively)16:01
*** pcaruana has quit IRC16:01
fungimtreinish: i'm freed up to take a look now16:01
mtreinishfungi: ok cool, thanks16:01
*** witek has quit IRC16:02
fungimtreinish: the log has recent entries16:02
fungi2018-02-09 16:02:15,340 Converting Subunit V2 stream: <_io.BytesIO object at 0x7fe184a00650> to SQL16:03
fungithat's on subunit-worker0116:03
mtreinishfungi: hmm, yeah I see events on the bus now16:03
mtreinishmaybe I just jumped the gun and assumed it broke when I didn't see any events for a couple minutes16:03
fungichecking 02 noqw16:03
fungi2018-02-09 16:01:37,010 Converting Subunit V2 stream: <_io.BytesIO object at 0x7fe045046770> to SQL16:04
fungiso yeah, both were doing things in the last few minutes16:04
AJaegerfungi: now on 542879 - accepted trade ;)16:04
mtreinishok, yeah that's my bad. Sorry for sounding the alarms for no reason16:04
fungimtreinish: no worries. lmk if the volume seems to be unexpectedly low16:04
mtreinishfungi: is it running the new version, 1.9.0, yet? There isn't any functional difference I'm just curious if the update triggered automagically16:05
mtreinish*any functional difference for the workers16:05
*** witek has joined #openstack-infra16:05
fungisubunit2sql (1.9.0)16:06
fungiis what's installed per pip list16:06
*** e0ne has quit IRC16:06
fungisubunit   5061 21.0  1.4 337184 122120 ?       Rl   15:42   5:05 /usr/bin/python2 /usr/local/bin/ -c /etc/subunit2sql/jenkins-subunit-worker-A.yaml -d /var/log/subunit2sql/subunit-worker-A-debug.log -p /var/run/jenkins-subunit-worker-A/jenkins-subunit-worker-A.pid16:06
fungiso seems to have restarted at 15:4216:06
fungithat's recent16:06
fungimaybe that could explain the lag16:06
*** adarazs is now known as adarazs_brb16:07
mtreinishok cool16:07
mtreinishyeah it might be working through a backlog or something16:07
*** salv-orlando has quit IRC16:07
fungi-rw-r--r-- 1 subunit subunit 5 Feb  9 15:42 /var/run/jenkins-subunit-worker-A/jenkins-subunit-worker-A.pid16:07
mtreinishbut as long as it's running that's good16:07
fungi15:42 for the pidfile on 02 as well16:08
Marx314question about branch creation: is this document still relevant?
pabelangerfungi: ack (mailman)16:08
*** salv-orlando has joined #openstack-infra16:08
fungiMarx314: it's relevant for teams whose deliverables are not under central openstack release management16:09
AJaegerMarx314: for almanach - which is not an official project - it should be relevant16:09
AJaegerMarx314: and almanach has the permissions to create branches, see "pushSignedTag = group almanach-release"16:09
AJaegerMarx314: ignore me!16:10
*** kiennt26 has joined #openstack-infra16:10
AJaegerMarx314: that is permission for tags16:10
Marx314good so how can I create a branch stable/queens I don't seem to have access to the review.openstack.org16:10
*** jamesdenton has joined #openstack-infra16:10
AJaegerMarx314: let me get you a few links...16:10
*** jaosorior has quit IRC16:10
AJaeger - ACL set up for branch creation16:11
AJaegerMarx314: and you're part of,members - which is the almanach-release group16:12
AJaegerMarx314: send a patch to add the "create " line to you ACLs and you should be set16:12
*** armaan has joined #openstack-infra16:13
openstackgerritMerged openstack-infra/project-config master: tacker-horizon seems to need the horizon-specific release jobs
AJaegerconfig-core, we can now remove some more files from jenkins/scripts, please review
openstackgerritMarc Aubry proposed openstack-infra/project-config master: Add create ACL for almanach
Marx314Thanks AJaeger!16:15
AJaegerMarx314: I can review much better if you tell me what exactly you do. I suggest to extend your commit message to include: The almanach team wants to create new stable branches, add permissions for that." That would have saved a few iterations ;)16:15
AJaegerMarx314: reviewed - time for next iteration, please16:16
Marx314reacting to -> I wanted to create a branch stable/queens16:16
AJaegerMarx314: perfectly fine ;) Just say it clearly in the commit message16:17
mnaserAJaeger: do you have a review which tests legacy-copy-jenkins-scripts ?16:17
mnaserAJaeger: i see the review it depends on added the role but doesnt run it anywhere16:17
*** yamahata has quit IRC16:17
mnaserof we already run legacy-copy-jenkins-scripts, my bad16:17
AJaegermnaser: checking...16:18
AJaegermnaser: playbooks/legacy/pre.yaml adds it16:18
fungimnaser: yeah, 542648 added it to the legacy pre phase playbook16:18
openstackgerritMarc Aubry proposed openstack-infra/project-config master: Add create ACL for almanach
mnaserok cool perfect :)16:19
AJaegermnaser: I should have tested but didn't ;(16:19
openstackgerritMarc Aubry proposed openstack-infra/project-config master: Add create ACL for almanach
mnaserAJaeger: i figure the role is already being tested in `legacy-copy-jenkins-scripts` so adding those files shouldn't break too much afiak16:20
AJaegerMarx314: +2A16:21
AJaegermnaser: just checked a running job - copying works fine...16:22
*** adarazs_brb is now known as adarazs16:25
toskyif I removed few legacy jobs from some repositories, but not from all repositories where they are used, can I submit the patch to project-config for now without the dependent patch on openstack-zuul-jobs ?16:26
AJaegerandreaf: could you resolve the merge conflict for , please?16:26
AJaegertosky: why would you do that?16:26
toskyAJaeger: to remove the duplicated jobs from the repositories which have already the new jobs16:26
AJaegertosky: why not change everything?16:27
toskyAJaeger: in progress16:27
toskyok, I will wait16:27
AJaegertosky: ah, so you're moving in-tree for several repos?16:27
AJaegertosky: In that case: send a first patch if you have a high-volume repo and it takes to get those in...16:28
toskyAJaeger: sahara jobs were shared between sahara, sahara-tests and python-saharaclient; sahara-tests and python-saharaclient have native jobs now, sahara is almost done16:28
AJaegertosky: define a template: Use that template everywhere and adopt the template ;)16:29
AJaegerstill, you need to push the template in...16:29
toskyfor two jobs? Uhm16:30
toskyit's a bit more complicated16:30
AJaegeras always ;)16:30
toskybut that's not the big issue anway16:30
toskynow, what did I miss with this backport to stable/queens? Shouldn't I see the new jobs?
AJaegertosky: you might need debugging help...16:34
AJaegertosky: I don't see anything obvious16:34
toskyAJaeger: what is the magic incantation for debugging help?16:34
*** felipemonteiro_ has quit IRC16:34
AJaegertosky: shall I edit it for you?16:34
*** felipemonteiro_ has joined #openstack-infra16:34
* tosky tries to autocomplete jebl but no results :)16:34
toskyAJaeger: oh, I didn't know that, let me try16:35
AJaegertosky: try corvus instead of jebl<tab>16:35
AJaegerSo, add "debug: true" for check pipeline16:35
toskyoh, a tactical change of nickname16:35
* corvus ducks16:35
AJaegerand once all tests finish, "toggle CI" to see the debug output in all its glory16:36
AJaegerand then check whether jebl<tab> is still hiding ;)16:36
pabelangerdmsimard: looks like ARA has non-voting jobs in gate pipeline, could you remove them?16:36
dmsimardpabelanger: yes, I can do that16:37
pabelangerdmsimard: tyty16:38
AJaegerpabelanger: could I trouble you with to remove jenkins slave scripts, please?16:38
*** caphrim007 has joined #openstack-infra16:38
corvusandreaf: why the WIP on ?16:38
AJaegerconfig-core and moving wheel scripts around: , please review16:39
*** hjensas|afk is now known as hjensas16:39
pabelangerAJaeger: left -1 with comment16:40
pabelangerAJaeger: never mind16:41
pabelangerAJaeger: just seen commit message16:41
AJaegerpabelanger: yeah, that one is completely unused - checked codesearch and also old project-copy Zuul v2 files.16:41
AJaegerpabelanger: thanks16:42
andreafcorvus: ok - dmsimard was mentioning that zuul is not for openstack only so we should make backward compatible changes so I wasn't sure we wanted this anymore16:47
corvusandreaf: yeah, that's true, but i think this is still new enough it probably hasn't grown past the openstack usage16:48
andreafcorvus: but if we still want it I'd change this a bit since by new implementation also handles the case of empty value and this doesn't16:48
corvusandreaf: ok.  i think we should nail down the api soon :)16:48
dmsimardI think we need to discuss backwards (in)compatibility in general for zuul-jobs, whether it's through tagging versions or other ways16:49
*** links has joined #openstack-infra16:49
corvusdmsimard: yep.  this warrants a formal policy.16:50
openstackgerritMatthew Treinish proposed openstack-infra/system-config master: Add subunit2sql check database server and workers
*** armaan has quit IRC16:54
mtreinishinfra-root: I'm getting a 500 from the o-h api server at:
mtreinishif someone gets a sec can you pull the traceback from the logs so I can see what's going on16:55
dmsimardmtreinish: let me get that for you16:56
*** HeOS has quit IRC16:56
dmsimardmtreinish: want the whole thing? it's about 10k lines16:57
fungia 10k line traceback?!?16:57
mnaserthat's gotta be some sort of record16:58
dmsimardwell, no, the traceback by itself isn't 10k lines.. but there's 10k lines worth of tracebacks in the error log :)16:58
fungiprobably just the most recent one16:58
*** armaan has joined #openstack-infra16:58
openstackgerritMerged openstack-infra/project-config master: Add create ACL for almanach
fungior the earliest one16:59
openstackgerritMerged openstack-infra/project-config master: Remove some moved slave scripts
*** neiljerram has joined #openstack-infra16:59
*** armaan has quit IRC16:59
fungibut i doubt we need them all16:59
* fungi needs to disappear to run some errands but will be back in an hour or so16:59
dmsimardI figured the other ones might have been worthwhile :)16:59
dmsimardmtreinish: here's the latest ones: let me know if you need more16:59
andreafcorvus, AJaeger: is it ok to host project-templates outside of openstack-zuul-job?16:59
AJaegerandreaf: yes!16:59
*** danpawlik has quit IRC16:59
*** armaan has joined #openstack-infra17:00
dmsimardmtreinish: there's a good couple of those too: ReadTimeout: HTTPConnectionPool(host='', port=80): Read timed out. (read timeout=60)17:00
AJaegerandreaf: in general ;) What is your plan?17:00
mtreinishdmsimard: thanks, that traceback should give me enough to debug the fialure17:00
andreafAJaeger: ok - I'm looking at devstack-plugin-ceph - every repo  that runs their integration jobs has the py27 job non voting on check and the py35 job on experimental17:01
mtreinishdmsimard: hmm that's probably the e-r query side of the api timing out on a cache update17:01
andreafAJaeger: so I was thinking of defining a template (along with the jobs) in devstack-plugin-ceph17:01
mtreinishas long as it eventually works it shouldn't be a problem17:01
andreafAJaeger: and then use it in project-config for all relevant repos17:01
AJaegerandreaf: and then use it *in-tree* for all repos ;)17:02
AJaegerandreaf: that's the approach I suggest - use the template in-tree with those users.17:02
andreafAJaeger: but it may become complicated to discover where a template is defined17:02
andreafAJaeger: and if you use it in-tree it's also complicated to know who uses it17:03
AJaegerlet's hear what the rest of the team thinks...17:03
AJaegerandreaf: for some of these we're still looking at best practices17:03
andreafAJaeger: yeah sure I can figure out from there but it's far from perfect17:03
AJaegerlet's see whether pabelanger, clarkb, corvus, fungi etc have any suggestions how to architect this best, andreaf ^17:04
*** myoung is now known as myoung|food17:04
openstackgerritMerged openstack-infra/nodepool master: Store build logs automatically
AJaegerandreaf: so, start with the template - and collect some more ideas here17:04
andreafAJaeger, corvus, mordred: I think we would need a tool from zuul that can resolved this i.e. find where something is defined and / or used17:04
tobiashandreaf: tristanC is working on improvements in zuul-web for that17:05
*** auristor has quit IRC17:10
tobiashandreaf: you can get an early impression there:
*** jcoufal_ has joined #openstack-infra17:13
*** jcoufal has quit IRC17:15
*** links has quit IRC17:15
mtreinishdmsimard, fungi: hmm it looks like the error is caused by a db entry missing a 'build_name' for the run17:16
mtreinishI wonder if something got lost because of the service restart with the new subunit2sql release17:16
*** ldnunes has quit IRC17:16
*** janki has quit IRC17:20
*** camunoz has quit IRC17:20
*** camunoz has joined #openstack-infra17:21
andreafAJaeger: is there a job naming convention I should follow?17:23
andreafAJaeger: for instance, if the parent job is tempest-full, and the job is defined in devstack-plugin-ceph, is devstack-plugin-ceph-tempest-full a good name or tempest-full-devstack-plugin-ceph like we used to do?17:24
mtreinishdmsimard, fungi: should fix the 50017:25
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources master: Updated get events by summit filters
openstackgerritMerged openstack-infra/openstackid-resources master: Updated get events by summit filters
*** ldnunes has joined #openstack-infra17:29
AJaegerandreaf: documents naming17:31
AJaegerandreaf: so, if the job is in devstack-plugin-ceph, calling it devstack-plugin-ceph-tempest-full is perfectly fine17:31
AJaegerandreaf: I expect a job named tempest-X to be defined in tempest repo17:32
AJaegerandreaf: that naming convention helps you to find where a job is defined...17:32
*** jpich has quit IRC17:32
*** caphrim007 has quit IRC17:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Move wheel-build scripts out of jenkins/scripts
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Move release-tools out of jenkins/scripts
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Install proposal scripts only when needed
*** VW_ has joined #openstack-infra17:34
AJaegermordred, pabelanger, dmsimard, mnaser, fungi, this is the rest of cleaning up jenkins/scripts . it leaves only in that directory.17:34
AJaegerplease carefully review the copying around17:35
*** yamamoto has quit IRC17:35
*** armaan has quit IRC17:35
*** sambetts is now known as sambetts|afk17:36
*** armaan has joined #openstack-infra17:36
andreafAJaeger ok cool17:36
*** VW__ has joined #openstack-infra17:36
*** VW_ has quit IRC17:37
*** harlowja has joined #openstack-infra17:37
*** yamamoto has joined #openstack-infra17:37
*** VW__ has quit IRC17:37
*** armaan has quit IRC17:37
*** VW has quit IRC17:37
*** VW has joined #openstack-infra17:38
*** armaan has joined #openstack-infra17:38
*** yamamoto has quit IRC17:39
*** florianf has quit IRC17:44
*** r-daneel has joined #openstack-infra17:48
*** dhill_ has joined #openstack-infra17:48
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Install proposal scripts only when needed
*** david-lyle has quit IRC17:50
*** olaph1 has joined #openstack-infra17:51
*** shardy has quit IRC17:51
*** auristor has joined #openstack-infra17:52
*** olaph has quit IRC17:52
*** gordc has joined #openstack-infra17:54
gordchi, anyone know where exists? i'm seeing an error that's its not found on stable/pike:
*** dhill_ has quit IRC17:55
*** dave-mccowan has quit IRC17:57
*** salv-orlando has quit IRC17:57
*** dave-mccowan has joined #openstack-infra17:58
*** salv-orl_ has joined #openstack-infra17:58
*** ldnunes has quit IRC17:59
*** derekh has quit IRC18:00
*** ldnunes has joined #openstack-infra18:00
AJaegergordc: we removed it - you need to backport the change to use zuul v3 native jobs18:04
AJaegergordc: sorry, wasn't aware that it was not backported ;(18:04
AJaegergordc: I'll sent you a change18:05
gordceumel8: thanks, will look18:05
gordcAJaeger: no worries. i just noticed it was failing on stable/*18:05
*** efoley has quit IRC18:05
gordcthat was quick.18:06
AJaegergordc: and
gordcAJaeger: thanks! so stable/queens should be fine? i'll just do the same if it isn't18:07
gordcthanks for quick help folks.18:07
AJaegergordc: the master change merged before christmas, so all fine18:07
*** salv-orl_ has quit IRC18:08
*** myoung|food is now known as myoung18:14
*** caphrim007 has joined #openstack-infra18:14
fungithat's one of the places codesearch won't help us18:15
fungifinding uses on non-master branches18:15
fungiafaik it only indexes content of master branches18:16
gordc:) that's what i used.18:16
*** ralonsoh has quit IRC18:16
fungii mean as far as finding uses of by non-master branches of projects with in-tree job definitions18:17
fungiso that we know when it's safe to remove18:17
pabelangerI do think there is a PR upstream to fix that in hound18:18
fungithough pattern-searching the entire body of loaded zuul configuration is probably something we can also add to the dashboard once the config browser lands18:18
fungimmm, or maybe not since that would probably not include the ansible role/playbook content18:19
*** neiljerram has quit IRC18:20
AJaegeronly a few repos had in-tree run-tox in master and I thought I checked for needed backports everywhere. Seems I missed some ;(18:20
*** kgiusti has joined #openstack-infra18:24
* AJaeger quickly double checked - didn't find any missing backports18:25
AJaegerSome repos did not yet backport zuul.yaml files18:26
pabelanger was the PR for hound I was thinking of. Must have seen it from dmsimard18:26
dmsimardpabelanger: I have a fork with that PR in it, it's what I used when building a POC deployment for RDO because we have repositories with no master branches18:26
dmsimardpabelanger: it's tricky to build forked go stuff, I really had to fork because of
dmsimardpabelanger: I don't know if it's possible to have hound index multiple branches of a single repository though18:28
dmsimardpabelanger: that PR allows you to specify a branch that is not hardcoded to master and that's it18:28
fungiany guesses why ze10 seems to run around half the available ram that the other executors do?18:28
*** yamahata has joined #openstack-infra18:28
fungioh, or is that ze0218:29
fungithe colors are very close18:29
pabelangerfungi: yah, I confused that the other day myself18:29
fungiokay, so that's ze02 with the newer kernel18:30
*** VW has quit IRC18:33
*** VW has joined #openstack-infra18:34
AJaegerfungi, pabelanger, dmsimard, could you put the stack starting at on your review list, please?18:34
*** tosky has quit IRC18:37
*** trown is now known as trown|lunch18:38
mtreinishhmm, has anyone seen gerrit do this before:
*** yamamoto has joined #openstack-infra18:40
dmsimardfungi: re: pattern searching.. I know we wrote something before to parse gerrit changes -- RDO was interested in knowing if there were changes to reqs.txt. I wonder if we could re-use that to search for zuul.yaml changes in non-master branches.18:40
mtreinishhmm, restarted firefox and it seems to have fixed itself18:40
mtreinishbut that was weird18:40
fungimtreinish: wow, no i've never seen mojibake in the interface parts of the gerrit webui18:42
fungithat's amazingly nutty18:42
*** kjackal has quit IRC18:44
*** dtantsur is now known as dtantsur|afk18:45
*** jcoufal_ has quit IRC18:45
*** slaweq has quit IRC18:45
*** yamamoto has quit IRC18:47
*** hashar has joined #openstack-infra18:48
pabelangeryah, never seen that before18:58
pabelangermust be client side, patchset looks proper here18:59
*** tosky has joined #openstack-infra19:01
*** david-lyle has joined #openstack-infra19:02
*** jpena is now known as jpena|off19:05
mtreinishpabelanger: I assumed as much when the firefox restart fixed it for me19:06
*** mriedem is now known as mriedem_afk19:06
openstackgerritAlex Krzos proposed openstack-infra/grafyaml master: Allow hidden templated vars
*** lucasagomes is now known as lucas-afk19:07
pabelangermrhillsman: indeed19:09
*** camunoz has quit IRC19:12
mrhillsmanwhat did I do?19:17
mrhillsmanoh maybe that was for matt19:17
*** lbragstad has quit IRC19:18
fungiyeah, i think pabelanger meant to be replying to mtreinish19:18
pabelangeryes, that19:20
*** camunoz has joined #openstack-infra19:29
*** ijw has quit IRC19:32
openstackgerritPaul Belanger proposed openstack-infra/project-config master: Remove intree jobs for openstack/windmill
*** lbragstad has joined #openstack-infra19:37
*** gfidente has quit IRC19:41
*** ijw has joined #openstack-infra19:48
*** tesseract has quit IRC20:00
*** rfolco|rover is now known as rfolco|off20:10
*** thorre_se has joined #openstack-infra20:13
*** ldnunes has quit IRC20:14
*** trown|lunch is now known as trown20:14
*** thorre has quit IRC20:16
*** thorre_se is now known as thorre20:16
toskycorvus: ehm... the patch now does not enter the zuul queue at all, and nothing is printed:
*** gordc has left #openstack-infra20:16
*** e0ne has joined #openstack-infra20:18
*** e0ne has quit IRC20:20
*** agopi is now known as agopi_out20:20
*** agopi_out has quit IRC20:24
*** slaweq has joined #openstack-infra20:25
*** VW has quit IRC20:26
*** VW has joined #openstack-infra20:26
*** yolanda has quit IRC20:28
*** hashar has quit IRC20:28
*** ijw has quit IRC20:30
*** amoralej is now known as amoralej|off20:30
*** pcichy has quit IRC20:33
*** jcoufal has joined #openstack-infra20:35
*** ijw has joined #openstack-infra20:38
*** dave-mccowan has quit IRC20:39
*** jcoufal has quit IRC20:41
*** ijw has quit IRC20:42
*** olaph has joined #openstack-infra20:44
*** olaph1 has quit IRC20:46
*** ijw has joined #openstack-infra20:46
*** e0ne has joined #openstack-infra20:47
*** camunoz has quit IRC20:48
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 nodeset
pabelangerconfig-core: easy review to add fedora-27 nodeset ^20:49
openstackgerritPaul Belanger proposed openstack-infra/openstack-zuul-jobs master: Add fedora-27 nodeset
*** ijw has quit IRC20:51
*** xinliang has quit IRC21:02
*** HeOS has joined #openstack-infra21:03
*** xinliang has joined #openstack-infra21:04
*** xinliang has quit IRC21:04
*** xinliang has joined #openstack-infra21:04
corvustosky: i'm not sure, but i suspect that the debug log for that change was too long... i think there's a bug in zuul where if it sits in queue too long it records multiple debug entries... let me copy it out of the logs for you.21:11
*** agopi_out has joined #openstack-infra21:13
*** e0ne has quit IRC21:13
toskycorvus: thanks!21:15
toskynow I need to understand the error message21:16
*** olaph1 has joined #openstack-infra21:17
SamYaplepabelanger: re: apache docker mirroring stuff. previously, did you actually see the cache polutlating?21:18
*** olaph has quit IRC21:19
toskyan error with matching not on the first dependency (the jobs in sahara-tests) but the job in devstack?21:19
*** armaan has quit IRC21:23
corvustosky: which job?21:23
*** armaan has joined #openstack-infra21:24
toskycorvus: - please note that the ancestors are defined only in sahara-tests/master (but that does not seem to be the problem)21:29
toskyand the logs that you pasted shows that zuul bails up on "No matching parents for 'devstack'"21:31
*** armaan has quit IRC21:33
corvustosky: sorry i mean which job are you investigating?21:34
corvustosky: i'm guessing python-saharaclient-tempest and -scenario?21:35
corvustosky: assuming so, it's because the new devstack job doesn't exist on stable/pike yet -- we need to backport it21:35
corvustosky: er rather stable/queens, not pike21:36
corvus(it *does* exist for pike, it doesn't exist for queens)21:36
fungiwell, devstack doesn't _have_ a stable/queens branch yet21:37
corvusfungi: a notable impediment21:37
corvusas soon as it does, this should all work21:37
corvus(since, when it's created, it'll be forked from master which has the job already)21:37
toskycorvus: yep, sorry, those two jobs (the failing ones)21:38
toskycorvus: but shouldn't fallback? Or does it fall back only if the branch does not exist?21:38
corvusi think we need a slight tweak for the branch creation ordering for next time21:38
tosky(fall back to master)21:39
*** trown is now known as trown|outtypewww21:39
toskyand also: does it mean that the devstack job does not exist also on stable/pike yet, and then I would hit the same issue with the next backport to pike?21:40
tosky(and why am I the first one to hit this? :)21:40
corvustosky: there isn't a fallback for job definitions as such; there are ways to work around this for other kinds of branch mismatches, but in this case, we intend to have a devstack queens branch, so the right thing is to wait for it to exist.21:40
corvustosky: no it *does* exist on stable/pike, i mis-spoke earlier21:40
corvustosky: lines 113 and 114 in tell us this21:40
corvustosky: it exists on master and stable/pike21:41
corvustosky: then line 115 is the error that tells us that there is no matching devstack job for the current change, whis is on queens21:41
toskyI see21:41
toskyI guess that the stable/queens of devstack will be created soon21:42
toskywith the other branches21:42
toskyor does it branch later?21:42
toskywell, it can't branch later, or all the jobs which use devstack as base would be broken for backports on queens21:42
fungiprobably a question for #openstack-release or #openstack-qa but i expect it branches asap now that we've discovered this catch-2221:43
toskyandreaf, mtreinish, gmann_ : ^^21:43
corvusaccording to it branches next week21:44
corvusi'd like to discuss moving that up a stage or two at the ptg21:45
toskyyeah, but...21:45
corvusaiui the reasoning was to get as many fixes needed for RCs of the services into devstack before the branch point, to reduce backports.  so i think the thing to discuss is how far we can push that.21:47
corvus*ideally* we would invert this, so that devstack branched first21:48
*** yamahata has quit IRC21:48
*** DuncanT has quit IRC21:49
toskyor add some fall back rules for such cases21:49
*** DuncanT has joined #openstack-infra21:50
*** salv-orlando has joined #openstack-infra21:50
fungiwe may be in a tight spot with release management. i think dhellmann and smcginnis are out this afternoon and ttx has likely already packed it in for the weekend. dims may be around here somewhere21:51
corvusyes, if we can't do this in the right sequence, we can alter the branch matchers on devstack/master to compensate.  it's not an ideal solution, but it's possible.  i'd like us to get to the point where we have affirmitively decided that we must branch other projects before devstack and have their jobs work before we do that though.21:54
corvusone way or the other, we'll have something better by rocky21:55
*** thorre_se has joined #openstack-infra22:01
*** dprince has joined #openstack-infra22:03
*** thorre has quit IRC22:04
*** thorre_se is now known as thorre22:04
*** salv-orlando has quit IRC22:04
fungiso from an ordering standpoint, i guess if project A defines a variant of a job defined in project B then we need to create the stable branch for project B before project A22:05
fungithis could get messy if we get a tangle of projects declaring variants of jobs in other projects22:05
*** ijw has joined #openstack-infra22:05
*** kong has quit IRC22:05
fungito the point where we end up having to branch them all simultaneously22:05
*** kong has joined #openstack-infra22:06
*** myoung is now known as myoung|off22:06
fungithough we may be able to draw a job graph which indicates which projects need to branch before which other projects?22:08
*** r-daneel has quit IRC22:09
*** kgiusti has left #openstack-infra22:10
*** slaweq has quit IRC22:10
*** ijw has quit IRC22:10
*** tpsilva has quit IRC22:13
*** ildikov has quit IRC22:13
*** ildikov has joined #openstack-infra22:14
*** jcoufal has joined #openstack-infra22:19
*** olaph1 is now known as olaph22:19
openstackgerritFrank Kloeker proposed openstack-infra/project-config master: Setup translation jobs for Queens (no priority)
*** ijw has joined #openstack-infra22:22
corvusfungi: s/defines a variant of/inherits from/22:23
corvusfungi: i suspect it's not actually going to be that tangled22:24
*** jcoufal has quit IRC22:24
corvusfungi: but if there are cases where it is, then yeah, we should basically try to branch them as close to simultaneously as possible22:24
corvusfungi: note however, that this is a pretty soft fail.  if these conditions aren't met, the jobs just don't run.  so it's not like things are broken, just not tested.  (of course, if you land a change without testing, *then* things are broken :)22:25
*** patrickeast has quit IRC22:26
corvusfungi: but it's not like we have to calculate a strict ordering to unwedge anything22:26
*** patrickeast has joined #openstack-infra22:26
eumel8actually I wanted to test the job first with some few repo, but now all projects are in:
andreafcorvus fungi dims tosky it's getting pretty late for me but if you need anything urgent from me ping me on IRC or even better via email I can help22:28
*** yamahata has joined #openstack-infra22:28
toskyandreaf: nothing that need to be done right now, just tl;dr the dependency system in zuul requires an earlier branching of devstack, and that requires some discussion; check the past logs when you can22:29
toskycorvus: but if jobs aren't run, there is no +v either22:30
toskyI think22:30
andreaftosky: so for the integrated gate on pike we still run the legacy job - I did not have time to backport stuff yet, not sure it's worth it even22:31
corvustosky: some jobs are still run, so you'll get the +v.  i think you're just not getting reports on that change because of the debug: true thing.22:31
andreaffor stable/queens one we have a stable branch on devstack everything should work22:31
andreafwe merged the extensions as dict patch this afternoon so we didn't have to backport22:32
pabelangerSamYaple: yes, required debug mode for apache to see22:33
andreafI think there is quite a bit of work to be done over time when cutting the branch also on grenade job configurations and test matrix setup etc... I haven't done it myself in the past but I guess I know most of the bits and pieces22:33
pabelangerSamYaple: but will show up in logs22:34
*** ijw has quit IRC22:34
pabelangerSamYaple: can also run tcpdump and look at HTTP traffic and see it use mirrors22:34
*** salv-orlando has joined #openstack-infra22:38
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Add zuul from scratch doc
openstackgerritJames E. Blair proposed openstack-infra/zuul master: Reclassify some debug log lines as info
*** fultonj has quit IRC22:40
*** dave-mccowan has joined #openstack-infra22:40
*** olaph1 has joined #openstack-infra22:42
*** armaan has joined #openstack-infra22:43
*** olaph has quit IRC22:43
SamYaplepabelanger: ok was just curious. so there is nothing beyond just that "validate" function that blocks it. shame22:46
SamYaplemaybe ill try to get a PR up to support it22:46
fungicorvus: oh, i missed that bit... so we're simply not selecting the jobs which inherit from the nonexistent branch on the other repo? from a release workflow standpoint i agree that's less terrible because in theory changes during rc must be tested and merged in master and only then backported22:49
corvusfungi: yep -- it's just not obvious in this case because that change was also getting hit with the debug:true problem and not reporting.22:50
corvusturns out we have have some protection against 4000-line review comments22:50
fungigo figure22:53
*** cargonza has quit IRC22:53
*** cargonza has joined #openstack-infra22:54
*** dprince has quit IRC22:54
*** edmondsw has quit IRC22:59
andreafcorvus mordred this kind of import fails miserably in zuul (with no logging again)
andreafwe run ansible 2.3.3 so it may be a version thing23:00
andreafdo you know how I could do something like that this that would keep ansible / zuul happy?23:00
corvusdmsimard: is that the include_role bug you were working on showing up again ^ ?23:01
*** VW_ has joined #openstack-infra23:02
dmsimardcorvus: sounds like it, yes23:03
corvusdmsimard: i was just looking up the logs -- f994a7dd2b064194b65876a8a278bae8 on ze0223:04
*** agopi_out has quit IRC23:04
*** VW has quit IRC23:05
*** armaan has quit IRC23:06
dmsimardcorvus: I haven't had the chance to revisit that yet. I'll put it on the top of my list. Last I looked I wasn't able to reproduce it so I've added that review to the story!/story/200132923:06
corvusdmsimard: er, i think that's the wrong bulid sorry23:06
corvusandreaf: it should be the tempest-full job, right?23:07
*** VW_ has quit IRC23:07
corvusthat's e4fdb1c004fe47478a5468bb6318ad66 on ze0323:07
andreafcorvus yeah23:07
andreafcorvus if I downgrade ansible to 2.3.3 on my laptop the task fails for me as well so I think it's an ansible thing23:08
corvusandreaf, dmsimard:
andreafcorvus: ansible seems to be doing a poor job in evaluating conditionals before other things - like with the with_dict, there is no way to make a when stop with_dict from checking the variable type23:09
*** jlabarre has quit IRC23:09
*** jtomasek has quit IRC23:11
*** zaro_ has quit IRC23:11
*** zaro_ has joined #openstack-infra23:11
*** csatari has quit IRC23:14
*** csatari has joined #openstack-infra23:15
*** mriedem_afk has quit IRC23:16
*** geekinutah has quit IRC23:16
*** geekinutah has joined #openstack-infra23:16
*** salv-orlando has quit IRC23:18
*** salv-orlando has joined #openstack-infra23:20
*** slaweq has joined #openstack-infra23:23
andreafcorvus, dmsimard: this seems to work, at least on my laptop - ansible is not happy only if the variable is not defined - if it is will happily process the when condition and not include the role when applicable23:23
*** salv-orlando has quit IRC23:24
*** slaweq has quit IRC23:27
*** salv-orlando has joined #openstack-infra23:28
pabelangerandreaf: why don't you use a filter, when: tempest_post_devstack | default(omit)23:29
pabelangerat least I think that works with when statements23:29
pabelangeror maybe default(False)23:29
*** esberglu has quit IRC23:37
notmynameare project coverage reports published anywhere beyond the job results for a patch?23:42
*** agopi_out has joined #openstack-infra23:43
*** eumel8 has quit IRC23:45
funginotmyname: not presently, though there has been discussion in the past of trying to do trending. might make sense for the health dashboard23:46
fungiyou can find them through the builds tab of the zuul status dashboard but that's not commit-to-commit trending of course23:47
* fungi disappears for dinner23:47
*** lbragstad has quit IRC23:49
*** wolverineav has quit IRC23:56
*** wolverineav has joined #openstack-infra23:57
*** ijw has joined #openstack-infra23:57

Generated by 2.15.3 by Marius Gedminas - find it at!