Tuesday, 2019-11-12

*** zbr has quit IRC00:09
*** slaweq has joined #openstack-infra00:11
*** rkukura has quit IRC00:11
*** rkukura has joined #openstack-infra00:15
*** slaweq has quit IRC00:16
*** kjackal has quit IRC00:27
*** ociuhandu has joined #openstack-infra00:28
*** ociuhandu has quit IRC00:39
*** kjackal has joined #openstack-infra00:43
*** mriedem has quit IRC00:44
*** Wryhder has joined #openstack-infra00:53
*** yoctozepto has quit IRC00:53
*** Lucas_Gray has quit IRC00:54
*** yoctozepto has joined #openstack-infra00:54
*** Wryhder is now known as Lucas_Gray00:54
*** kjackal has quit IRC01:02
*** yoctozepto has quit IRC01:05
*** yoctozepto has joined #openstack-infra01:05
*** rkukura has quit IRC01:08
*** rkukura has joined #openstack-infra01:12
*** ociuhandu has joined #openstack-infra01:14
mordredinfra-root: https://github.com/wagoodman/dive is an interesting tool to poke around and look at docker images01:18
*** ociuhandu has quit IRC01:18
*** lbragstad_ has joined #openstack-infra01:24
*** lbragstad has quit IRC01:26
*** rh-jelabarre has quit IRC01:27
*** rfolco has quit IRC01:54
*** ociuhandu has joined #openstack-infra01:57
*** ociuhandu has quit IRC02:08
*** rlandy|ruck|bbl is now known as rlandy|ruck02:11
*** slaweq has joined #openstack-infra02:11
*** slaweq has quit IRC02:16
*** Lucas_Gray has quit IRC02:18
*** roman_g has quit IRC02:34
*** rlandy|ruck has quit IRC02:34
*** ociuhandu has joined #openstack-infra02:46
*** ociuhandu has quit IRC02:51
*** rcernin_ has joined #openstack-infra03:00
*** rcernin has quit IRC03:03
*** apetrich has quit IRC03:08
*** kjackal has joined #openstack-infra03:13
*** rkukura has quit IRC03:18
*** rcernin_ has quit IRC03:26
*** rcernin has joined #openstack-infra03:26
*** yamamoto has joined #openstack-infra03:31
*** yamamoto has quit IRC03:36
*** udesale has joined #openstack-infra03:41
*** udesale has quit IRC03:45
*** udesale has joined #openstack-infra03:46
*** yamamoto has joined #openstack-infra03:48
openstackgerritColleen Murphy proposed opendev/irc-meetings master: Remove keystone team office hours  https://review.opendev.org/69376303:56
*** slaweq has joined #openstack-infra04:11
*** slaweq has quit IRC04:16
*** ykarel|away has joined #openstack-infra04:25
*** tkajinam_ has joined #openstack-infra04:29
*** ykarel|away is now known as ykarel04:31
*** tkajinam has quit IRC04:32
*** ociuhandu has joined #openstack-infra04:34
*** ociuhandu has quit IRC04:41
*** kjackal has quit IRC04:45
*** tkajinam_ has quit IRC04:58
*** tkajinam has joined #openstack-infra04:59
*** surpatil has joined #openstack-infra05:15
*** slaweq has joined #openstack-infra05:26
*** rkukura has joined #openstack-infra05:30
*** ociuhandu has joined #openstack-infra05:30
*** ociuhandu has quit IRC05:35
*** udesale has quit IRC05:43
*** udesale has joined #openstack-infra05:43
*** udesale has quit IRC05:44
*** udesale has joined #openstack-infra05:44
*** lpetrut has joined #openstack-infra06:33
*** lpetrut has quit IRC06:34
*** lpetrut has joined #openstack-infra06:35
*** dpawlik has joined #openstack-infra06:49
*** dpawlik has quit IRC06:55
*** armax has quit IRC07:01
*** udesale has quit IRC07:03
*** udesale has joined #openstack-infra07:04
*** ociuhandu has joined #openstack-infra07:10
*** ociuhandu has quit IRC07:10
*** ociuhandu has joined #openstack-infra07:11
*** dpawlik has joined #openstack-infra07:14
*** ociuhandu has quit IRC07:16
*** rcernin has quit IRC07:22
*** piotrowskim has joined #openstack-infra07:34
*** trident has quit IRC07:37
*** apetrich has joined #openstack-infra07:43
*** jpena|off is now known as jpena07:46
*** iurygregory has joined #openstack-infra07:46
*** ociuhandu has joined #openstack-infra07:48
*** trident has joined #openstack-infra07:48
*** tosky has joined #openstack-infra07:52
yoctozeptoAJaeger: yeah, ironic folks only know it is notmyname behind the bot implementation but other knowledge seems to be gone :-) I contacted notmyname and await response07:52
*** yamamoto has quit IRC07:53
*** ociuhandu has quit IRC07:54
*** yamamoto has joined #openstack-infra07:58
*** yamamoto has quit IRC08:01
*** dchen has quit IRC08:01
*** dchen has joined #openstack-infra08:03
*** yamamoto has joined #openstack-infra08:08
*** adriant has quit IRC08:08
*** iokiwi has quit IRC08:08
*** yamamoto has quit IRC08:09
*** tesseract has joined #openstack-infra08:11
*** ralonsoh has joined #openstack-infra08:14
*** dchen has quit IRC08:15
*** pgaxatte has joined #openstack-infra08:25
*** zbr has joined #openstack-infra08:30
*** dpawlik has quit IRC08:31
*** tosky has quit IRC08:32
*** dpawlik has joined #openstack-infra08:34
*** zbr has quit IRC08:35
*** zbr has joined #openstack-infra08:36
*** tosky has joined #openstack-infra08:37
*** tkajinam has quit IRC08:37
*** amoralej|off is now known as amoralej08:38
*** yamamoto has joined #openstack-infra08:41
*** yamamoto has quit IRC08:46
*** dtantsur|afk is now known as dtantsur08:48
*** priteau has joined #openstack-infra08:58
*** pkopec has joined #openstack-infra08:58
*** rpittau|afk is now known as rpittau08:59
*** ociuhandu has joined #openstack-infra09:01
openstackgerritMerged opendev/irc-meetings master: Remove keystone team office hours  https://review.opendev.org/69376309:07
*** xek_ has joined #openstack-infra09:09
*** yamamoto has joined #openstack-infra09:13
*** iurygregory has quit IRC09:17
*** iurygregory has joined #openstack-infra09:18
*** ykarel is now known as ykarel|lunch09:23
*** sshnaidm|off is now known as sshnaidm|rover09:30
*** ociuhandu has quit IRC09:30
*** ociuhandu has joined #openstack-infra09:31
*** irclogbot_0 has quit IRC09:39
*** irclogbot_0 has joined #openstack-infra09:40
*** tosky has quit IRC09:41
*** tosky has joined #openstack-infra09:42
*** ociuhandu has quit IRC09:42
*** ociuhandu has joined #openstack-infra09:44
*** yamamoto has quit IRC09:44
*** rcernin has joined #openstack-infra09:44
*** apetrich has quit IRC09:47
*** jaosorior has joined #openstack-infra09:51
*** tosky_ has joined #openstack-infra09:52
*** ociuhandu has quit IRC09:53
*** yamamoto has joined #openstack-infra09:53
*** ociuhandu has joined #openstack-infra09:54
*** tosky has quit IRC09:54
*** ociuhandu has quit IRC09:55
*** ociuhandu has joined #openstack-infra09:56
*** yamamoto_ has joined #openstack-infra09:56
*** yamamoto has quit IRC09:57
*** ociuhandu has quit IRC09:57
*** tosky has joined #openstack-infra09:58
*** ociuhandu has joined #openstack-infra09:58
*** apetrich has joined #openstack-infra09:59
*** ociuhandu has quit IRC10:00
*** ociuhandu has joined #openstack-infra10:01
*** tosky_ has quit IRC10:01
*** ociuhandu has quit IRC10:02
*** yamamoto_ has quit IRC10:02
*** ociuhandu has joined #openstack-infra10:03
*** gfidente|afk is now known as gfidente10:03
*** tosky_ has joined #openstack-infra10:05
*** tosky has quit IRC10:08
*** ociuhandu has quit IRC10:08
*** rfolco has joined #openstack-infra10:08
*** soniya29 has joined #openstack-infra10:10
*** rcernin has quit IRC10:11
*** tosky has joined #openstack-infra10:14
*** tosky_ has quit IRC10:17
*** tosky_ has joined #openstack-infra10:21
*** tosky has quit IRC10:24
*** panda has quit IRC10:24
*** yamamoto has joined #openstack-infra10:29
*** panda has joined #openstack-infra10:29
*** yamamoto has quit IRC10:29
*** vesper has quit IRC10:30
*** vesper11 has joined #openstack-infra10:31
*** tosky has joined #openstack-infra10:34
*** tosky_ has quit IRC10:36
*** soniya29 has quit IRC10:37
*** tosky_ has joined #openstack-infra10:43
*** tosky has quit IRC10:45
*** ykarel|lunch has quit IRC10:46
*** tosky_ has quit IRC10:46
*** tosky has joined #openstack-infra10:46
*** yamamoto has joined #openstack-infra10:47
*** dpawlik has quit IRC10:47
*** ociuhandu has joined #openstack-infra10:48
*** rcernin has joined #openstack-infra10:49
*** hashar has joined #openstack-infra10:51
*** ociuhandu has quit IRC10:54
*** yamamoto has quit IRC11:01
*** tosky has quit IRC11:01
*** tosky has joined #openstack-infra11:01
*** tosky_ has joined #openstack-infra11:05
*** tosky has quit IRC11:06
*** tosky_ has quit IRC11:09
*** tosky has joined #openstack-infra11:09
*** jaosorior has quit IRC11:10
*** xek_ has quit IRC11:12
*** dpawlik has joined #openstack-infra11:15
*** ykarel|lunch has joined #openstack-infra11:15
*** udesale has quit IRC11:17
*** tosky has quit IRC11:17
*** ykarel|lunch is now known as ykarel11:18
*** dpawlik has quit IRC11:19
*** kjackal has joined #openstack-infra11:20
*** tosky has joined #openstack-infra11:23
*** roman_g has joined #openstack-infra11:26
*** rcernin has quit IRC11:26
*** dpawlik has joined #openstack-infra11:27
zbrclarkb: let me know when you are back, i want to explain bindep *role* issue.11:31
*** ramishra has quit IRC11:35
*** hashar has quit IRC11:38
*** surpatil has quit IRC11:40
*** aedc_ has joined #openstack-infra11:46
*** aedc has quit IRC11:47
*** tosky_ has joined #openstack-infra11:47
*** ykarel_ has joined #openstack-infra11:49
*** tosky has quit IRC11:49
*** tosky has joined #openstack-infra11:50
*** ykarel has quit IRC11:51
*** tosky_ has quit IRC11:53
*** aedc_ has quit IRC11:53
*** surpatil has joined #openstack-infra12:00
*** yamamoto has joined #openstack-infra12:09
*** Lucas_Gray has joined #openstack-infra12:11
*** dpawlik has quit IRC12:12
*** jpena is now known as jpena|lunch12:26
*** amoralej is now known as amoralej|lunch12:27
*** ykarel_ is now known as ykarel|afk12:28
*** ssbarnea has quit IRC12:38
*** dpawlik has joined #openstack-infra12:39
*** openstackgerrit has quit IRC12:41
*** dpawlik has quit IRC12:44
*** yamamoto has quit IRC12:44
*** ociuhandu has joined #openstack-infra12:46
*** dpawlik has joined #openstack-infra12:47
*** ykarel_ has joined #openstack-infra12:49
*** rlandy has joined #openstack-infra12:50
*** ociuhandu has quit IRC12:50
*** ykarel|afk has quit IRC12:52
*** ykarel_ is now known as ykarel|afk12:53
*** rlandy is now known as rlandy|ruck12:54
*** weshay|rover is now known as weshay12:54
*** sshnaidm|rover is now known as sshnaidm|ruck12:55
*** rlandy|ruck is now known as rlandy|over12:55
*** rlandy|over is now known as rlandy|rover12:56
*** jaosorior has joined #openstack-infra12:57
*** surpatil has quit IRC13:00
*** liuyulong has joined #openstack-infra13:02
*** soniya29 has joined #openstack-infra13:03
*** eharney has quit IRC13:04
*** rh-jelabarre has joined #openstack-infra13:07
noonedeadpunkhey:) it looks like gerrit bot died :(13:10
*** eharney has joined #openstack-infra13:12
funginoonedeadpunk: indeed, i'll prod it now13:13
fungi#status log restarted gerritbot after it hit a freenode ping timeout at 12:41:1513:15
openstackstatusfungi: finished logging13:15
*** ociuhandu has joined #openstack-infra13:16
*** carl_cai has joined #openstack-infra13:17
*** ssbarnea has joined #openstack-infra13:18
*** soniya29 has quit IRC13:22
*** jpena|lunch is now known as jpena13:27
*** mriedem has joined #openstack-infra13:37
*** ociuhandu has quit IRC13:38
*** ociuhandu has joined #openstack-infra13:39
*** amoralej|lunch is now known as amoralej13:39
*** ociuhandu has quit IRC13:44
*** rlandy|rover is now known as rlandy|rover|mtg13:45
*** Goneri has joined #openstack-infra13:48
*** dpawlik has quit IRC13:57
*** dpawlik has joined #openstack-infra13:58
*** Lucas_Gray has quit IRC13:59
*** rlandy|rover|mtg is now known as rlandy|rover13:59
*** Lucas_Gray has joined #openstack-infra14:00
*** Lucas_Gray has quit IRC14:01
*** ykarel|afk is now known as ykarel14:02
*** ociuhandu has joined #openstack-infra14:02
*** priteau has quit IRC14:04
*** dpawlik has quit IRC14:06
*** xek_ has joined #openstack-infra14:07
*** dpawlik has joined #openstack-infra14:09
*** ociuhandu has quit IRC14:13
*** ociuhandu has joined #openstack-infra14:14
*** ociuhandu has quit IRC14:19
*** ykarel_ has joined #openstack-infra14:19
*** dpawlik has quit IRC14:20
*** dpawlik has joined #openstack-infra14:21
*** ykarel has quit IRC14:22
*** eharney has quit IRC14:23
*** goldyfruit has joined #openstack-infra14:44
*** openstackgerrit has joined #openstack-infra14:47
openstackgerritTobias Henkel proposed zuul/nodepool master: Delete images by id  https://review.opendev.org/69367214:47
*** michael-beaver has joined #openstack-infra14:51
*** ociuhandu has joined #openstack-infra14:56
*** ykarel_ is now known as ykarel|afk14:59
*** tosky_ has joined #openstack-infra15:00
openstackgerritCorey Bryant proposed openstack/openstack-zuul-jobs master: Add non-voting tox py38 jobs to Ussuri templates  https://review.opendev.org/69340115:00
*** tosky has quit IRC15:00
*** ociuhandu has quit IRC15:00
*** tosky_ is now known as tosky15:01
*** raschid has joined #openstack-infra15:03
*** eharney has joined #openstack-infra15:05
*** dpawlik has quit IRC15:06
*** eharney has quit IRC15:17
*** lbragstad_ is now known as lbragstad15:20
*** kjackal has quit IRC15:21
*** rfolco is now known as rfolco|brbr15:22
*** rfolco|brbr is now known as rfolco|brb15:22
*** carl_cai has quit IRC15:26
*** armax has joined #openstack-infra15:27
*** pcaruana has joined #openstack-infra15:28
*** eharney has joined #openstack-infra15:30
*** sreejithp has joined #openstack-infra15:30
*** tosky has quit IRC15:31
*** tosky has joined #openstack-infra15:32
*** aedc has joined #openstack-infra15:39
*** ociuhandu has joined #openstack-infra15:44
*** ociuhandu has quit IRC15:44
*** ociuhandu has joined #openstack-infra15:46
*** ociuhandu has quit IRC15:53
*** ykarel|afk is now known as ykarel16:05
*** jhesketh has quit IRC16:05
*** lpetrut has quit IRC16:10
*** sreejithp_ has joined #openstack-infra16:15
*** gyee has joined #openstack-infra16:15
*** sreejithp has quit IRC16:17
*** jaosorior has quit IRC16:18
*** jhesketh has joined #openstack-infra16:21
*** tesseract has quit IRC16:32
*** ociuhandu has joined #openstack-infra16:33
*** ociuhandu has quit IRC16:33
*** ociuhandu has joined #openstack-infra16:34
*** ociuhandu has quit IRC16:40
clarkbzbr: reading scrollback in #zuul I think you and pabelanger may have sorted out that a bootstrap playbook is appropriate?16:44
zbrclarkb: not really, the virtualenv issue was more of a red herring in that case.16:45
zbrthe real issue is that "bindep role" fails to install deps even if the bindep.txt is valid.16:46
*** jpena is now known as jpena|off16:46
clarkbzbr: because bindep doesn't understand how to bootstrap additional repos16:46
clarkbthis is true of PPAs or epel16:46
clarkbI think it is outside of the scope of bindeps current design to have it try and solve that, instead you should configure needed repos prior16:46
zbrclarkb: its not bindep job to do this, but installing first package does activate the repository16:46
*** ociuhandu has joined #openstack-infra16:47
zbrtrue, outside the scope of bindep, but not outside the scope of bindep-role16:47
zbrif I use "yum install epel-release && yum install re2-devel" it works.16:47
clarkbexcept that will get you epel not using our mirrors16:47
clarkband that is unreliable16:47
zbrmirror support was never in bindep role spec. one could question why not using http proxies, still lets try to find a solution.16:49
clarkbright something other than bindep should ensure that epel is configured prior to running the bindep role16:49
clarkbthis is my point16:49
clarkbthen you can properly configure mirrors or an http proxy or whatever else is appropriate for your CI system16:49
zbrfrom the UX point of view, a developer would list packages under bindep.txt -- especatatio is to get CI to install them16:49
zbrwhy? installing "epel-release" package is enabling this repository, this default system behavior. is not some new feature.16:50
zbr(the disabled epel is something else which needs to be sorted outside bindep, i do agree on that aspect)16:51
AJaegerclarkb: do you have time to discuss next step for the opensuse-15 nodes - or should I ask you later this week?16:51
clarkbbecause in at least one case (mine) I don't want to just enable that repo. I want to consume it from our mirrors because we know the upstream is unreliable16:51
*** artom has joined #openstack-infra16:51
clarkbenabling epel in the manner you describe is bad for my system16:51
clarkbAJaeger: I was going to ask you about that today too so today is fine16:51
*** ociuhandu has quit IRC16:52
zbri hope i do not annoy you too much but my impression was that our CI is aiming to test supported configuration of various distros. the default scenario is installing the package.16:53
zbrso your personal use case may be convenient for you but is not an official supported channel16:53
clarkbzbr: we try to strike a balance between reliability and defaults16:53
AJaegerclarkb: https://review.opendev.org/#/q/status:open+topic:use-opensuse-15 is the list, we need to force-merge IMHO the two glance changes. I stacked Dirk's changes and suggest you abandon your two open ones in favor of dirk's.16:53
zbr(i know well why you want caching, not debating this)16:53
zbrclarkb: well said. is a trade-off16:53
clarkbzbr: and I don't think there is any problem with simply having something configure the needed repos prior to running bindep16:54
AJaegerclarkb: once the glance are in, it's devstack and tempest - and then infra ;)16:54
clarkbzbr: in my case that means configuring the mirrors. In your case it may mean using the defaults16:54
clarkbzbr: both should be fine for bindep16:54
clarkbthere is no conflict16:54
clarkbAJaeger: ok I can abandon mine then look at merging the glance changes16:54
zbrstill the average developer knows to populate bindep.txt file, do we expect them to alter lots of CI jobs to make it work?16:55
*** mgutehall has joined #openstack-infra16:55
AJaegerclarkb: we should check with glance folks but their PTL is not around right now16:55
AJaegerclarkb: so, that's all ;)16:55
zbri was trying to find a solution where the developer is not forced to update zuul job definitions to make it work. the expectation is that zuul will load the bindep.txt file and install what is mentioned there.]16:55
artomo/ Possibly the wrong place to ask, but I have a question about the best place to define a nodeset that's going to be shared between different projects. So for example Nova and a tempest plugin (https://review.opendev.org/#/c/691062/56/.zuul.yaml@3 specifically). Do we just just define the nodeset in one of those, and have the other use it? Or is there like a "central repository" for shared nodeset definitions?16:56
clarkbAJaeger: abandon https://review.opendev.org/#/c/690438/ and https://review.opendev.org/#/c/690435/ ?16:56
zbrcan we do it? (me trying to avoid future work on other repos)16:56
clarkbzbr: if a developer runs bindep locally they won't get the behavior your are trying to bake into the bindep role16:56
AJaegerclarkb: yes, those two16:56
clarkbzbr: I don't think your proposal addresses that concern16:56
clarkbzbr: so you've still only solved it for the ci case16:57
clarkband if we are going to solve a ci specific case imo we should do that for the ci case then which means mirrors etc16:57
zbrfunny part is that we already have epel installed on the images, but disabled16:57
clarkbartom: there is a central repo that defines nodesets: openstack/openstack-zuul-jobs16:57
AJaegerartom: define it together with the job that needs it.16:57
AJaegerclarkb: in this specific case, I don't think we need it in openstack-zuul-jobs, the nodeset is very special IMHO16:58
zbri could do "if epel-release inside bindep output, activate repo", but I guess you may not like it.16:58
clarkbAJaeger: ya probably better in the job specific location16:58
artomclarkb, AJaeger, it is pretty niche, yeah. But two proposed patches have it: https://review.opendev.org/#/c/679656/13/.zuul.yaml and https://review.opendev.org/#/c/691062/56/.zuul.yaml16:58
clarkbzbr: what is the problem with having a "enable-epel" role run in jobs if you need epel?16:58
artomSo just... stick in one of them, and whichever merges first wins?16:58
clarkbartom: I would only put it in one. Use depends-on17:00
zbrclarkb: extra maintainance, that's all. if we need that for very few jobs not a problem, but if that explodes, it may need altering lots of job definitions, that's all.17:00
*** rpittau is now known as rpittau|afk17:00
zbrprobably we want a generic option to enable/add extra repositories, not a special one for epel?17:01
artomclarkb, understood, thanks! Also, do you have any idea what Sean's talking about when he mentions the new multi numa labels, that are presumably replacing the ubuntu-bionic-expanded one?17:01
clarkbzbr: ya and possible it falls under the existing rewrite of mirror configuration for jobs17:01
AJaegerartom: since nova job is inherited in whitebox put the nodeset to the *parent* which isi nova17:02
zbrlikely I would create an ensure-repo that gets a list of repos to enable. 1st version would support only epel.17:02
clarkbmostly I want to avoid making a simple thing that doesn't actually solve what users need (and in my case that means configuring epel to use a mirror and not simply relying on the upstream)17:02
*** pgaxatte has quit IRC17:02
zbrbut at least we could build on top of it later.17:02
artomAJaeger, wait, nova job is inherited in whitebox? I... don't think so? :) At least not intentionally17:02
*** dtantsur is now known as dtantsur|afk17:02
artomI'll probably put it in whitebox - the Nova patch I have no idea when/if it'll ever merge17:02
AJaegerartom: nova is parent job, whitebox is child and thus inherits.17:02
artomWhereas whitebox it's just me and Sean, so we should get it in fairly quickly17:03
AJaegerartom: sorry, not parent job. Misread line17:03
artomAJaeger, but the whitebox job inherits from tempest-multinode-py3-full...17:03
*** Tengu has quit IRC17:03
*** pcaruana has quit IRC17:03
AJaegerartom: or name them differently...17:04
zbrclarkb: where is the mirror config for jobs? maybe i can alter this to enable epel optionally.17:04
artomAJaeger, yeah eh? It's not like we're consuming any additional resources by naming them differently. I just have this idea in my head that if 2 things are the same they should be called the same :)17:05
*** ociuhandu has joined #openstack-infra17:05
*** Tengu has joined #openstack-infra17:05
clarkbzbr: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors via https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L3517:05
*** Tengu has quit IRC17:06
*** iurygregory has quit IRC17:06
AJaegerartom: you can also: Add to whitebox with different name, merge, once nova merged: remove nodeset from whitebox and use common definition ;)17:06
clarkbAJaeger: looking at the 150 changes we need the glance ones in before the depends on for tempest and devstack are satisfied. I agree would be good to get someone to ack those glance changes first and that they likely need force merging because things have bitrotted17:07
*** Tengu has joined #openstack-infra17:07
artomAJaeger, ah, I like that, thanks!17:08
clarkbrosmaita: can you take a look at https://review.opendev.org/#/c/692526/ and https://review.opendev.org/692035 ? you are on glance stable maint17:09
rosmaitaclarkb: ack17:09
artom... and I think I found the labels I was looking for in project-config/nodepool/nl02.openstack.org.yaml17:09
artomThank you folks!17:10
AJaegerrosmaita: I propose that we force-merge them but we'd like to have your blessing ;)17:10
*** ociuhandu has quit IRC17:11
*** rkukura has quit IRC17:11
*** ykarel is now known as ykarel|away17:11
*** ociuhandu has joined #openstack-infra17:15
*** Tengu has quit IRC17:16
*** Tengu has joined #openstack-infra17:18
openstackgerritSorin Sbarnea proposed zuul/zuul-jobs master: POC: allow configure-mirrors to enable extra repos  https://review.opendev.org/69388717:19
rosmaitaAJaeger: ok, i hope abhishekk doesn't kill me, though17:23
AJaegerrosmaita: one is a backport, the other needs merging with master. So, happy to join the discussion ;) thanks!17:25
rosmaitaactually, you have https://review.opendev.org/#/c/692526/ going into a feature branch -- i think this change is already at HEAD of master17:26
rosmaitaso it shouldn't be a big deal17:26
AJaegeryes, it is at HEAD of master.17:27
clarkbAJaeger: I guess we should be good to merge those two now? Then we can bug qa team to get tempest change in17:28
AJaegerclarkb: agreed17:29
*** chandankumar is now known as rrrrrrrrrrrr17:29
AJaegerclarkb: I'll ask qa team...17:30
*** rfolco|brb is now known as rfolco17:30
clarkbok done17:31
AJaegerthanks, clarkb17:31
* AJaeger asked on #openstack-qa17:31
*** ociuhandu has quit IRC17:32
*** ociuhandu has joined #openstack-infra17:33
*** ociuhandu has quit IRC17:33
*** ociuhandu has joined #openstack-infra17:34
*** rrrrrrrrrrrr is now known as chandankumar17:34
clarkbAJaeger: do we still need the openstack/project-config side change to remove that image too?17:35
clarkbfrom nodepool and grafana and all that17:36
AJaegerclarkb: yes, that is needed as well. Want to push it?17:36
AJaegerNot sure whether anything else is missing for cleanup...17:36
*** ociuhandu has quit IRC17:36
clarkbya I can work on that change though might not get it up until after today's meeting17:38
*** ociuhandu has joined #openstack-infra17:38
*** chandankumar is now known as raukadah17:38
*** david-lyle is now known as dklyle17:41
*** ociuhandu has quit IRC17:42
*** ykarel|away has quit IRC17:44
*** ociuhandu has joined #openstack-infra17:45
*** rkukura has joined #openstack-infra17:50
*** ociuhandu has quit IRC17:54
*** igordc has joined #openstack-infra17:59
*** sreejithp has joined #openstack-infra18:07
artomAnother ansible question if I may - when writing a role for one project, is it possible to include_role from another project?18:08
*** sreejithp_ has quit IRC18:09
artomOh actually, 'copy-build-sshkey' is a Zuul role, so it should be available magically. Sorry for the noise.18:09
clarkbartom: yes, but you'll need to list that project as roles: entry in the job18:09
clarkboh ya the zuul jobs roles should already be there18:10
* artom is new to this whole world. What's not easy is telling the difference between OpenDev-specific stuff and Zuul stuff.18:11
*** gfidente is now known as gfidente|afk18:12
*** aedc has quit IRC18:25
*** amoralej is now known as amoralej|off18:33
*** ralonsoh has quit IRC18:34
*** eernst has joined #openstack-infra18:39
*** raschid has quit IRC18:53
*** kjackal has joined #openstack-infra18:58
*** pkopec has quit IRC19:05
*** ociuhandu has joined #openstack-infra19:08
*** michael-beaver has quit IRC19:10
*** ociuhandu has quit IRC19:12
*** kjackal has quit IRC19:17
*** mgutehall has quit IRC19:30
*** mgutehall has joined #openstack-infra19:32
*** jaosorior has joined #openstack-infra19:39
*** eernst has quit IRC19:40
*** sshnaidm|ruck is now known as sshnaidm|afk19:42
*** lbragstad has quit IRC19:50
*** lbragstad has joined #openstack-infra19:51
*** Goneri has quit IRC19:51
eharneythis cinder job looks like it's been stuck for some time:  http://zuul.openstack.org/stream/625f41d621ed4d1c8438d0994faa947a?logfile=console.log19:58
Shrewsoh, ooops. looks like i need to adjust my calendar for time change19:58
eharneyi'm not sure what "TASK [get df disk usage]" does exactly, but maybe it got wedged on a broken NFS share?19:59
clarkbeharney: its a task that only runs against failed jobs to determine disk usage as we've found some jobs fail due to lack of disk and that is otherwise unlogged19:59
clarkbeharney: it uses the ansible raw module to run df which should be fairly resilient itself, but if df is failing due to broken filesystems that may explain it20:00
clarkbit should timeout, though I would've expected the timeout to be hit already20:00
eharneyright, looks like it's been there since sunday?20:03
openstackgerritClark Boylan proposed openstack/project-config master: Remove opensuse-150  https://review.opendev.org/69391520:08
clarkbAJaeger: ^ that look right?20:09
clarkbeharney: ya20:09
AJaegerclarkb: do we first need to go to 0 before we can remove?20:10
AJaegerclarkb: otherwise looks fine - but will download and double check once the rest is in. thanks!20:11
clarkbAJaeger: I thought we did with the 423 removal but pretty sure we didn't20:12
clarkbI think nodepool will see the 150 nodes as leaked once we remove the config for it then clean them up automatically20:12
clarkbwe only need to set things to zero when removing provider configs as the provider config is necessary to clean up resources. In this case it is just resource config not provider config we are cleaning up20:12
clarkbeharney: I am able to ssh onto the test node but manual run of df does seem to not be doing much20:14
clarkbeharney: stat("/opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57" is what strace says it is hanging on20:16
eharneyclarkb: ok, so it is a stuck nfs share20:16
*** jaosorior has quit IRC20:16
clarkblocalhost:/srv/nfs1 on /opt/stack/data/cinder/mnt/896fb15da6036b68a917322e72ebfe57 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp6,timeo=600,retrans=2,sec=sys,clientaddr=::1,local_lock=none,addr=::1)20:17
clarkbmount seems to confirm that20:17
clarkbeharney: I can kill the ansible process which will force the job to report. However, if I do that the job node will get deleted20:18
clarkbeharney: is it worth putting a hold on that job first?20:18
*** panda has quit IRC20:18
eharneyclarkb: a hold for further examination you mean?20:18
clarkbyes20:19
clarkbI am not sure if "nfs is very broken" is the sort of thing the cinder team wants to investigate20:19
*** lbragstad has quit IRC20:19
eharneyclarkb: not sure, it seems to be completing everywhere else, so i'm not sure that this particular failure is too interesting20:19
clarkbok, if it happens again I guess we hold it there. For now I'll just kill the ansible process and force this job to complete20:19
eharneywe could do something like change the mount options to "soft, intr" so that it would fail more gracefully if this continues to happen20:20
*** lbragstad has joined #openstack-infra20:20
*** munimeha1 has joined #openstack-infra20:20
*** armax has quit IRC20:20
*** panda has joined #openstack-infra20:21
AJaegerclarkb: do we have other jobs in this state? Could there be a larger problem ?20:21
clarkbAJaeger: no that was the only one20:22
AJaegerclarkb: looking at zuul.opendev.org, this is the only one I see - so we should be good20:22
AJaegerclarkb: great20:22
*** armax has joined #openstack-infra20:22
eharneythanks for taking a look20:23
clarkbwe can also add timeouts directly to the tasks we run in that cleanup playbook to make them more resilient to this20:25
openstackgerritClark Boylan proposed opendev/base-jobs master: Timeout cleanup playbook tasks  https://review.opendev.org/69392420:29
clarkbeharney: AJaeger ^ something like that20:29
*** sreejithp has quit IRC20:34
*** sreejithp_ has joined #openstack-infra20:34
ianwclarkb: how come the job timeout doesn't apply?20:37
AJaegerclarkb: https://review.opendev.org/#/c/691966/ is the next opensuse-150 change. Dependencies are approved and in gate now...20:40
AJaegerdependencies are the tempest and devstack changes20:40
*** sreejithp_ has quit IRC20:42
*** slaweq has quit IRC20:47
smcginnisThere is a group called cinder-release that I am assuming was used before we had release automation. I don't see it referenced anywhere now.20:48
smcginnishttps://review.opendev.org/#/admin/groups/144,members20:48
smcginnisAnyone know if that is correct, and if so, how we can get the group deleted?20:48
fungiianw: post phase uses a separate timeout from pre/run, doesn't it?20:48
fungismcginnis: we can't delete gerrit groups, but we can empty, rename and hide them20:49
smcginnisfungi: Do you think it's worth the effort?20:49
smcginnisI can just continue to ignore it. :)20:50
fungithat's mostly what we do20:50
fungibut you were asking if you could get the group deleted20:50
smcginnisWorks for me.20:50
ianwfungi: yes, but it still should timeout eventually?20:53
fungii don't know20:53
fungimaybe?20:53
fungii guess we have a default timeout for it, yeah20:54
fungiinfra-root: rackspace just opened an incident for a problem with the host ze01 is running on20:54
ianwfrom my reading of self.runAnsiblePlaybook i think we always have a timeout setup20:55
fungiwe may see some job impact from that vm suddenly getting rebooted, but more likely it'll just have a bunch of cruft which needs to get cleaned up20:55
*** eharney has quit IRC20:57
paladoxfungi the deletion of group is being worked on at the hackathon this week at least that was what the google doc said :)21:00
fungipaladox: ooh, neat!21:00
paladoxhttps://gerrit-review.googlesource.com/c/gerrit/+/12913021:01
*** rfolco has quit IRC21:01
*** rfolco has joined #openstack-infra21:01
*** Goneri has joined #openstack-infra21:02
paladoxfungi we've also set a tentative date for our upgrade! NoteDB made it easier to test mediawiki/core operations/puppet since all i had to do was git mirror and then reindex!21:02
paladox(almost like i have db access)21:03
fungithat does indeed sound convenient21:03
fungieasier than mysqldump/source21:03
paladoxyup21:03
clarkbianw: my guess is bug in cleanup phase or with raw module breaking the watchdog maybe21:04
*** exsdev has quit IRC21:05
*** rfolco has quit IRC21:06
*** aluria has quit IRC21:07
*** dosaboy has quit IRC21:07
*** exsdev has joined #openstack-infra21:09
*** ociuhandu has joined #openstack-infra21:09
*** dosaboy has joined #openstack-infra21:13
*** ociuhandu has quit IRC21:13
*** ociuhandu has joined #openstack-infra21:15
*** rcernin has joined #openstack-infra21:17
*** ociuhandu has quit IRC21:19
*** ociuhandu has joined #openstack-infra21:23
*** dpawlik has joined #openstack-infra21:24
*** ociuhandu has quit IRC21:28
*** dpawlik has quit IRC21:28
*** gfidente|afk has quit IRC21:30
*** xek_ has quit IRC21:31
*** Goneri has quit IRC21:43
*** ociuhandu has joined #openstack-infra21:44
*** Goneri has joined #openstack-infra21:46
*** ociuhandu has quit IRC21:49
*** Goneri has quit IRC21:57
*** Goneri has joined #openstack-infra21:57
*** eharney has joined #openstack-infra21:58
*** Goneri has quit IRC22:01
*** Goneri has joined #openstack-infra22:01
*** slaweq has joined #openstack-infra22:11
*** mriedem has quit IRC22:12
*** aluria has joined #openstack-infra22:14
*** slaweq has quit IRC22:15
*** ociuhandu has joined #openstack-infra22:17
*** ociuhandu has quit IRC22:22
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Introduce manual setting of DIB_INIT_SYSTEM  https://review.opendev.org/41417922:27
*** eharney has quit IRC22:31
*** adriant has joined #openstack-infra22:41
*** slaweq has joined #openstack-infra22:46
*** ociuhandu has joined #openstack-infra22:51
*** ociuhandu has quit IRC22:58
*** igordc has quit IRC23:06
*** tkajinam has joined #openstack-infra23:08
openstackgerritAlex Schultz proposed openstack/project-config master: Add tripleo-operators-ansible  https://review.opendev.org/69394523:14
*** dchen has joined #openstack-infra23:26
*** goldyfruit has quit IRC23:28
openstackgerritAlex Schultz proposed openstack/project-config master: Add tripleo-operators-ansible  https://review.opendev.org/69394523:28
*** ociuhandu has joined #openstack-infra23:29
*** ociuhandu has quit IRC23:30
*** ociuhandu has joined #openstack-infra23:30
*** igordc has joined #openstack-infra23:31
mordredclarkb, AJaeger, ianw: followup from infra meeting - the reason to not switch to podman across the board isn't anything to do with podman - it's that we need to add podman support to the container jobs. right now we don't have speculative container image support for podman23:31
mordredpodman *itself* is fine23:31
mordredit's ok right now for review(-dev) because we're not really changing those images super frequently and can live with the need to land a change before taking the next step23:32
ianwmordred: do you have time to run me through how those speculative jobs might work for dib?23:35
*** ociuhandu has quit IRC23:36
mordredianw: speculative image jobs would really only come in to play for dib if we wind up testing building dib images based on container images that we also build. ... so other than that, I'm not sure I've given it much thought.23:40
mordredunless you mean something different ...23:40
openstackgerritMonty Taylor proposed zuul/nodepool master: [wip] move openstack testing to use containerised daemon  https://review.opendev.org/69346423:41
ianwmordred: what i mean is, how to get a dib change into a nodepool-builder container for testing in the functional jobs23:41
mordredAH23:42
ianwi'm looking at "assemble"23:42
ianwhttps://opendev.org/opendev/system-config/src/branch/master/docker/python-builder/scripts/assemble23:42
ianwIIRC, the Dockerfile in the nodepool source copies the nodepool source into /tmp/src in the image23:43
ianwand then "assemble" installs it?23:43
mordredyes, that's right.23:43
mordred(this is going to be a 'fun' problem)23:43
mordredianw: this is so that we can test dib changes right?23:44
ianwyep, in the devstack functional test that starts up nodepool, builds the images and boots them23:45
mordredI think what we might want to do is make a testing specific image that is FROM opendevorg/nodepool-builder23:45
mordredand installs dib from source into it23:45
ianwthe other trick is that that job pulls in glean & openstacksdk from source23:45
mordredthen have the dib job build that image as step one23:45
ianwbut one thing at a time :)23:46
mordredyeah23:46
mordredwell23:46
mordredmaybe we just make it an image that installs all three from source - since it's not a produciton image, we don't really need to worry about making it as optimized size-wise as we do for nodepool-builder itself23:46
ianwi guess nodepool-builder+openstack sdk need to be installed23:47
ianwfor glean, we need the source tree, as the dib element installs from it23:47
ianwi think the glean case is just a matter of mapping in the zuul directory to the right place for the builder23:47
*** munimeha1 has quit IRC23:49
ianwhrm, ok, so i think i got more an idea, we can just build an image in the functional test; we don't need to have a whole buildset repository setup to pull from23:53
mordredyeah. that said - we probably still want one - because we'd want to be able to depends-on nodepool changes and so we'd want speculative nodepool-builder images23:54
mordredbut- as you said - one step at a time :)23:55
ianwyeah, right now i'd like to test 414179 which fixes running in containers without init systems (yes, we've known about the problem for a long time :)23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!