Saturday, 2017-03-18

*** dimtruck is now known as zz_dimtruck00:00
*** wolverineav has joined #openstack-infra00:07
*** gyee has quit IRC00:08
openstackgerritK Jonathan Harker proposed openstack-infra/zuul feature/zuulv3: Reenable test_build_configuration_conflict  https://review.openstack.org/44627500:14
*** wolverineav has quit IRC00:14
*** ZZelle has quit IRC00:15
*** baoli has joined #openstack-infra00:21
*** esikachev has joined #openstack-infra00:21
kfox1111seeing a bunch of POST_FAILURE's. this a known thing?00:23
kfox1111job 446274,500:23
*** esikachev has quit IRC00:25
clarkbkfox1111: there are no logs at all that would imply something fairly catastrophic00:26
clarkblooks like zuul status confirms its widespread00:26
clarkb/dev/mapper/main-logs          12T   12T     0 100% /srv/static/logs00:28
* clarkb wonders who filled the disk this time00:29
jeblairi propose we hit this pretty hard with the zuulv3 hammer.00:30
clarkbI'm running du to try and track it down (and hopefully find things to delete)00:30
clarkblooks like we run the cron to delete old things once a week? am I reading that crontab entry correctly?00:31
clarkbwe are 7 ish hours away from it starting00:32
jeblairclarkb: i agree00:32
clarkb2.0G 442219 is biggest I have found so far00:32
bkeroNot tripleo-quickstart this time? :)00:33
clarkb1.4G 442519 is next00:33
clarkbbkero: kolla and neutron so far00:33
jeblairlast week was flat; this week started a growth trend on monday00:33
mnaserhm00:34
mnaseri can have a look at the kolla one00:34
clarkb2.1G 442619 is puppet-tripleo00:34
clarkbheh 442719 is also large00:35
clarkbthey are all offset by 10000:35
jeblairi'll work on a change to 30 days00:36
mnaserclarkb if its not too much to ask, the breakdown of 442519 on which job is using a lot of space so we can prevent that from happening agian00:36
clarkbmnaser: ya working on that next00:36
clarkbmnaser: it looks like its the centos and oracle deploy jobs. They use ~75MB each00:37
clarkbubuntu is only 26MB ish00:37
clarkbthen they all add up to 350MB per commit which added up to 1.4GB after 4 patchsets00:38
*** mriedem has quit IRC00:38
clarkbjeblair: cool, and maybe we want to trigger it by hand early today?00:38
*** vhosakot has quit IRC00:38
clarkbbkero: gate-tripleo-ci-centos-7-ovb-containers-oooq for 19/442619/3/check-tripleo was half a gig00:39
openstackgerritJames E. Blair proposed openstack-infra/puppet-openstackci master: Reduce log archive to 30 days  https://review.openstack.org/44717100:39
jeblairclarkb: ^00:39
clarkbbkero: and gate-tripleo-ci-centos-7-multinode-upgrades-nv and gate-tripleo-ci-centos-7-scenario003-multinode-upgrades-nv are 200 something mB for same patchset in check00:40
*** esikachev has joined #openstack-infra00:41
jeblairclarkb: would you like me to start it now?00:41
clarkbjeblair: sure00:41
*** wolverineav has joined #openstack-infra00:41
jeblairstarted screen as root on static00:41
clarkboh on the oooq job there are 4 runs of it all adding up00:41
jeblairthere's a copy of the log archive maint script in root's dir which adds the "-f" option to gzip00:43
jeblairclarkb: do you know that that's about?00:43
clarkbbkero: so on puppet-triplo its mostly lots of job runs of not so light jobs00:43
clarkbjeblair: I don't00:43
jeblairit's from feb23 so i assume it's one of the locally modified copies fungi ran00:43
*** dave-mccowan has joined #openstack-infra00:44
clarkbbkero: looks like those are from before cleaning up the quickstart stuff00:45
jeblairit's running00:45
*** wolverineav has quit IRC00:46
jeblairand throwing a fit because there's no space available for it to gzip00:46
*** esikachev has quit IRC00:46
jeblairoh i found fungi's rm-only script00:46
jeblairi'll modify and go with that00:46
clarkb2.0G 444337 looks like lots of ironic jobs with lots of patchsets00:47
jeblairokay that's running now00:47
*** Apoorva has quit IRC00:48
clarkbso I'm not finding any crazy smoking guns like we have in the past. Though getting old oooq logs cycled out will definitely help00:48
*** dave-mccowan has quit IRC00:48
*** esikachev has joined #openstack-infra00:48
clarkbbkero: sshnaidm|off EmilienM 446337 is recent and generated 390MB in oooq containers and ran that twice00:50
* bkero looks00:50
bkeroclarkb: is 446337's gate-tripleo-ci-centos-7-ovb-containers-oooq-nv job the one you're talking about?00:51
jeblairlogs has 500M avail now00:51
clarkbbkero: ya let me get a link buts its 200MB of containers00:51
clarkbbkero: http://logs.openstack.org/37/446337/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/d8dbca6/logs/oooq/overcloud-controller-0/var/log/extra/docker/00:51
clarkbthen do that a few times, it adds up00:51
jeblair2.5G00:52
bkeroclarkb: It's this: http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/collect-logs/tasks/collect.yml#n15100:53
clarkbbkero: it also looks like it collected the undercloud wtice for 41MB each http://logs.openstack.org/37/446337/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/d8dbca6/logs/oooq/ (undercloud and 127.0.0.2)00:53
EmilienMDamn it00:53
EmilienMI'm on my phone now, I can't look now00:54
bkeroEmilienM: I'm looking00:54
clarkbbkero: so is that running on every job now?00:54
EmilienMOk thanks00:54
*** esikachev has quit IRC00:54
EmilienMWhat introduced it?00:55
bkeroclarkb: Don't know, would have to check project-config00:55
clarkbkk, but ya just high level looking not collecting all the docker things would be nice00:55
clarkbas would not grabbing the same host's logs twice00:55
bkeroThe docker things should be plaintext like 'ps' or 'rpm -qa'00:55
bkeroI'm snarfing down this example now so I can see where the space is being spent00:55
clarkbbkero: its the actual containers00:55
clarkboh wait I lied00:56
bkeroYeah...00:56
clarkbits the entire /etc from every container00:56
bkeroIt's just /etc and some /var/log00:56
clarkbwhich is probably not useful at all00:56
clarkb(woo more DIR_COLORS)00:57
bkerolol00:57
bkeroSo part of this is the virtualenv tarball00:57
bkeroWhich is the brunt of that 40MB x200:57
clarkbbut also http://logs.openstack.org/37/446337/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/d8dbca6/logs/oooq/127.0.0.2/ and http://logs.openstack.org/37/446337/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/d8dbca6/logs/oooq/undercloud/ appear to be the same thing00:57
bkeroI had a patch but I thought it was unnecessary because of the new toci runner script, but apaprently ovb is still making it00:58
clarkband we've also copied the entirety of /etc there (twice)00:58
bkeroYes, there's a fix for that in the other runner scripts as well00:58
bkeroThat's the ansible playbook running on 'all' hosts including '127.0.0.1' and 'undercloud'00:58
bkeroThere's an inline sed in the runner that exempts 127.0.0.1 from collection, but apparently it's not in that runner script00:58
*** funzo has joined #openstack-infra00:59
bkeroLet me give you some patches00:59
clarkbwell I can't approve the patches so I might just go find a guiness00:59
clarkbjeblairs script run seems to have jobs apssing again00:59
clarkb(but we really need to stop adding so much data to the tripleo jobs)00:59
bkeroYeah, I'm going to head to a movie at 1930, but want to get these in before I leave00:59
clarkb(please be specific if what you grab rather than grabbing everything)00:59
clarkbwe've also got stackviz eating 17MB on every devstack-gate run01:01
clarkbhttp://logs.openstack.org/93/440693/11/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/8d09523/logs/stackviz/data/ is most of it01:02
clarkbit looks like it is duplicating a bunch of our log data but putting it in json01:02
clarkbmtreinish: ^ any idea of what it is doing there?01:02
*** wolverineav has joined #openstack-infra01:03
bkeroHuh, 127.0.0.1 dir shouldn't exist. This sed isn't working. http://logs.openstack.org/37/446337/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/d8dbca6/console.html#_2017-03-16_14_25_56_39575801:03
bkeroBut when I run it by hand it works01:03
clarkbbkero: its 127.0.0.201:03
clarkbnot .101:03
*** funzo has quit IRC01:03
*** tosky has quit IRC01:04
bkeroRight01:04
clarkbok gotta go cook dinner01:04
bkero127.0.0.201:04
bkeroSame thing, sorry01:04
clarkbI think we are recovering now. Next week we should push again on cleaning up what we collect01:04
bkeroFrom TFS: sed -i 's/hosts: all:!localhost/hosts: all:!localhost:!127.0.0.2/' /home/jenkins/workspace/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/.quickstart/playbooks/collect-logs.yml01:04
bkeroand yet from TFPB:   hosts: all:!localhost01:05
*** wolverineav has quit IRC01:07
*** dave-mccowan has joined #openstack-infra01:07
*** dave-mccowan has quit IRC01:09
openstackgerritBen Kero proposed openstack-infra/tripleo-ci master: ovb: Remove bin/ and usr/ from working_dir tarball  https://review.openstack.org/44717401:10
bkeroclarkb, EmilienM: ^ There's one part of it01:10
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Move alembic into sql driver  https://review.openstack.org/44212401:12
bkeroclarkb: looking at actual gzipped space, it looks like quickstar.tar.gz is the largest portion at 24mb, and the 127.0.0.2 and undercloud directories are only 6.4MB each01:13
bkeroThe majority of that is small files in etc/ and /var/log/messages.txt.gz01:13
clarkbright but also /etc on priciple01:14
bkerosure01:15
bkeroThe 6.4MB shouldn't be causing an outage right now01:15
bkeroI have it marked as something to fix though01:15
openstackgerritMerged openstack-infra/zuul feature/zuulv3: Remove more swift configurations  https://review.openstack.org/44605601:17
bkeroclarkb: There's the removal of docker's etc/ ^01:17
bkero... as soon as openstackgerrit gets around to posting it ... https://review.openstack.org/44717601:18
*** hongbin has quit IRC01:19
*** neiljerram has quit IRC01:22
*** dave-mccowan has joined #openstack-infra01:26
*** markvoelker has joined #openstack-infra01:26
*** panda is now known as panda|Zz01:28
*** markvoelker has quit IRC01:31
*** aviau has quit IRC01:32
*** aviau has joined #openstack-infra01:32
*** Julien-zte has quit IRC01:33
*** jamesden_ has quit IRC01:37
*** baoli has quit IRC01:37
*** baoli has joined #openstack-infra01:38
*** kaisers1 has joined #openstack-infra01:39
*** kaisers has quit IRC01:40
*** dave-mccowan has quit IRC01:40
mtreinishclarkb: you'll have to ask timothyb89 about the internal details of how stackviz works01:42
mtreinishclarkb: but my guess is that it's being formated in json so the js can read it for: http://logs.openstack.org/93/440693/11/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/8d09523/logs/stackviz/#/stdin/timeline01:42
*** mriedem has joined #openstack-infra01:46
*** EricGonczer_ has quit IRC01:47
*** rfolco has quit IRC01:52
*** funzo has joined #openstack-infra01:59
*** mat128 has joined #openstack-infra02:02
*** markvoelker has joined #openstack-infra02:03
*** funzo has quit IRC02:04
openstackgerritIan Y. Choi proposed openstack-infra/project-config master: [I18n] accept header change support with PEP8  https://review.openstack.org/44717802:04
*** esikachev has joined #openstack-infra02:09
*** esikachev has quit IRC02:13
*** ijw has quit IRC02:16
*** esikachev has joined #openstack-infra02:17
*** Julien-zte has joined #openstack-infra02:19
*** ijw has joined #openstack-infra02:20
*** mat128 has quit IRC02:21
*** esikachev has quit IRC02:22
*** ijw has quit IRC02:24
*** Julien-zte has quit IRC02:24
*** Julien-zte has joined #openstack-infra02:28
*** _Cyclone_ has quit IRC02:28
EmilienMbkero: looking02:33
EmilienMbkero: don't we have an exclude list somewhere else?02:34
*** Julien-zte has quit IRC02:35
EmilienMbkero: see artcl_exclude_list in oooq-extras02:35
*** esikachev has joined #openstack-infra02:39
*** esikachev has quit IRC02:43
*** yuvalb has quit IRC02:48
*** yuvalb has joined #openstack-infra02:48
*** nicolasbock has joined #openstack-infra02:52
*** esikachev has joined #openstack-infra02:52
*** katyafervent_awa has quit IRC02:53
*** arif-ali has quit IRC02:54
*** arif-ali has joined #openstack-infra02:55
*** katyafervent_awa has joined #openstack-infra02:55
*** ijw has joined #openstack-infra02:55
*** esikachev has quit IRC02:56
*** armax has quit IRC02:59
*** baoli has quit IRC03:00
*** rwsu has quit IRC03:00
*** funzo has joined #openstack-infra03:00
*** rwsu has joined #openstack-infra03:00
*** gongysh has quit IRC03:01
*** lihi has quit IRC03:01
*** lihi has joined #openstack-infra03:02
*** esikachev has joined #openstack-infra03:02
*** Sukhdev has quit IRC03:04
*** funzo has quit IRC03:05
*** dimak has quit IRC03:07
*** esikachev has quit IRC03:08
*** dimak has joined #openstack-infra03:08
*** esikachev has joined #openstack-infra03:12
*** esikachev has quit IRC03:17
*** esikachev has joined #openstack-infra03:20
*** esikachev has quit IRC03:27
*** gouthamr has quit IRC03:30
*** rwsu has quit IRC03:41
bkeroEmilienM: yep, we could use that to create an exclusion list for /etcs03:46
bkeroassuming the globbing would work properly03:46
bkerothat dorsnt work for the virtuslenv though03:46
*** kaisers1 has quit IRC03:47
*** furlongm has quit IRC03:47
*** Julien-zte has joined #openstack-infra03:54
*** mriedem has quit IRC03:55
*** ijw_ has joined #openstack-infra03:56
*** ijw_ has quit IRC03:57
*** nicolasbock has quit IRC03:58
*** ijw has quit IRC03:59
*** funzo has joined #openstack-infra04:01
*** kaisers has joined #openstack-infra04:02
*** ijw has joined #openstack-infra04:03
*** funzo has quit IRC04:05
*** sree has joined #openstack-infra04:17
*** kzaitsev_mb has joined #openstack-infra04:20
*** sree has quit IRC04:21
*** ijw has quit IRC04:34
*** ijw has joined #openstack-infra04:38
*** funzo has joined #openstack-infra05:02
*** ijw has quit IRC05:06
*** funzo has quit IRC05:06
*** ijw has joined #openstack-infra05:07
*** psachin has joined #openstack-infra05:08
*** ijw has quit IRC05:11
*** gongysh has joined #openstack-infra05:18
*** rfolco has joined #openstack-infra05:31
*** rfolco has quit IRC05:31
*** MasterOfBugs has quit IRC05:33
*** nadya has joined #openstack-infra05:44
*** funzo has joined #openstack-infra06:02
*** ijw has joined #openstack-infra06:07
*** funzo has quit IRC06:07
*** rcernin has joined #openstack-infra06:47
*** rcernin has quit IRC06:48
*** yamamoto has joined #openstack-infra06:58
*** esikachev has joined #openstack-infra06:59
*** funzo has joined #openstack-infra07:03
*** funzo has quit IRC07:07
*** yamamoto has quit IRC07:11
*** yamahata has joined #openstack-infra07:16
*** rcernin has joined #openstack-infra07:17
*** yamamoto has joined #openstack-infra07:20
*** nadya has quit IRC07:35
*** yamamoto has quit IRC07:38
*** rcernin has quit IRC07:42
*** yamamoto has joined #openstack-infra07:43
*** yamamoto has quit IRC07:49
*** vsaienko has joined #openstack-infra07:57
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** funzo has joined #openstack-infra08:04
*** Julien-zte has quit IRC08:04
*** vsaienko has quit IRC08:08
*** funzo has quit IRC08:08
*** sree has joined #openstack-infra08:23
openstackgerritAndreas Jaeger proposed openstack-infra/devstack-gate master: DNM: Testing py3-run-tests  https://review.openstack.org/44719508:28
*** sree has quit IRC08:28
openstackgerritMerged openstack-infra/project-config master: Run devstack-gate python3 unittests  https://review.openstack.org/41755508:35
*** kzaitsev_mb has quit IRC08:41
*** yamamoto has joined #openstack-infra08:48
*** yamamoto has quit IRC08:53
openstackgerritMerged openstack-infra/project-config master: layout/tripleo: improve scenarios coverage for glance  https://review.openstack.org/44525008:54
openstackgerritVitaly Gridnev proposed openstack-infra/devstack-gate master: Add sahara to features.yaml in subnode  https://review.openstack.org/44715808:55
*** markvoelker has quit IRC08:56
*** funzo has joined #openstack-infra09:05
*** funzo has quit IRC09:09
openstackgerritAndreas Jaeger proposed openstack-infra/devstack-gate master: DNM: Testing py3-run-tests  https://review.openstack.org/44719509:14
*** Julien-zte has joined #openstack-infra09:15
*** Julien-zte has quit IRC09:24
AJaegerclarkb, https://review.openstack.org/417555  but having two tox environments run breaks the freeze command, see http://logs.openstack.org/95/447195/2/check/gate-devstack-gate-unittests/0ba2a39/console.html#_2017-03-18_09_16_38_28811209:25
*** Julien-z_ has joined #openstack-infra09:26
AJaegerclarkb: so, what should we do? Update run-tox.sh to handle more than one venv or run two tests?09:26
*** yamahata has quit IRC09:26
*** tosky has joined #openstack-infra09:27
*** pepijn has left #openstack-infra09:47
*** cody-somerville has quit IRC09:47
*** cody-somerville has joined #openstack-infra09:47
*** cody-somerville has joined #openstack-infra09:47
*** markvoelker has joined #openstack-infra09:56
openstackgerriton sun proposed openstack/python-jenkins master: Jenkins-2.x will return an error such as 403 Forbidden without authorization, and break. At this moment, the jenkins version can be found in HTTPError, so we need to get version from the http response header.  https://review.openstack.org/44720010:00
*** Jeffrey4l_ has joined #openstack-infra10:00
*** markvoelker has quit IRC10:02
openstackgerritVitaly Gridnev proposed openstack-infra/devstack-gate master: Add sahara to features.yaml in subnode  https://review.openstack.org/44715810:03
*** Jeffrey4l has quit IRC10:04
*** funzo has joined #openstack-infra10:05
*** pbourke has quit IRC10:05
*** pbourke has joined #openstack-infra10:06
*** funzo has quit IRC10:10
*** gongysh has quit IRC10:14
*** sree has joined #openstack-infra10:24
*** sree has quit IRC10:29
*** nicolasbock has joined #openstack-infra10:33
*** jamielennox is now known as jamielennox|away10:40
*** kzaitsev_mb has joined #openstack-infra10:45
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: New ovb playbook and settings  https://review.openstack.org/43156710:50
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test new ovb with images building  https://review.openstack.org/44522810:55
*** dizquierdo has joined #openstack-infra11:04
openstackgerritamrith proposed openstack-infra/project-config master: Hot Fix: enable DEVSTACK_GATE_TROVE for trove's jobs  https://review.openstack.org/44721111:05
*** funzo has joined #openstack-infra11:06
*** funzo has quit IRC11:10
amrithg'morning infra, a hot-fix for trove to enable DEVSTACK_GATE_TROVE for the trove jobs. https://review.openstack.org/#/c/447211/11:12
amrithclarkb, fungi, jeblair this is what we discussed around 6pm (eastern) yesterday11:13
amrithany other infra cores around, please take a look this should be a quick +311:13
*** yamamoto has joined #openstack-infra11:20
*** claudiub has joined #openstack-infra11:21
*** yamamoto has quit IRC11:28
fungiamrith: making breakfast but i'll take a look11:41
amriththx fungi what's for breakfast?11:42
amriththanks for asking, but I already ate11:42
amrith:)11:42
fungiamrith: congee with umeboshi11:48
amrithmmm, I should've waited.11:48
amrithreminds me of summit in tokyo11:49
fungioh, yep the hotel breakfast was exquisite11:55
openstackgerritMerged openstack-infra/project-config master: Hot Fix: enable DEVSTACK_GATE_TROVE for trove's jobs  https://review.openstack.org/44721111:56
*** dizquierdo has quit IRC11:57
*** markvoelker has joined #openstack-infra11:57
*** jkilpatr has joined #openstack-infra12:01
*** markvoelker has quit IRC12:02
amrithfungi, in tokyo I didn't eat in the hotel other than on the first day. after that I just went across the street to the big all day restaurant called "Family Restaurant" https://goo.gl/maps/arDnuBHbZJ52 the fare there was way better than the Marmolada (https://goo.gl/maps/LHqjL6EoC572) in the hotel.12:05
amriththanks for +3'ing the trove job12:05
amrithlet me see if it works :)12:05
fungiooh, somehow i never even noticed "family restaurant" but looks great12:06
*** EricGonczer_ has joined #openstack-infra12:07
*** esikachev has quit IRC12:18
*** sree has joined #openstack-infra12:25
*** csmart has quit IRC12:25
*** masayukig[m] has quit IRC12:25
*** yamamoto has joined #openstack-infra12:28
*** sree has quit IRC12:29
*** baoli has joined #openstack-infra12:30
*** csmart has joined #openstack-infra12:30
*** yamamoto has quit IRC12:34
*** psachin has quit IRC12:35
*** esikachev has joined #openstack-infra12:35
*** esikachev has quit IRC12:40
*** yamamoto has joined #openstack-infra12:40
*** flepied has quit IRC12:42
*** sshnaidm|off has quit IRC12:43
*** esikachev has joined #openstack-infra12:50
*** esikachev has quit IRC12:54
*** EricGonczer_ has quit IRC12:56
*** flepied has joined #openstack-infra12:58
mnasersay we had a job that needed credentials for part of it, is there something we can reuse that exists today?13:01
mnaser(i assume there is if there are jobs to push to pypi)13:01
*** quite has quit IRC13:07
*** funzo has joined #openstack-infra13:08
Jeffrey4l_mnaser, one issue for this is: we have to run "docker login" in ci scripts.  then it is possible to get by anyone.13:11
*** funzo has quit IRC13:12
*** ym200 has joined #openstack-infra13:13
*** quite has joined #openstack-infra13:15
mnaserJeffrey4l_ im pretty sure you can store credentials in a file or import from stdin or something along those lines13:16
*** quite has quit IRC13:16
Jeffrey4l_if it is stored in a file, i can print plain password to ci's console.log.13:18
Jeffrey4l_mancdaz, ^13:18
mnaserJeffrey4l_ id be interested to know how pypi uploads handle this in that case13:18
*** quite has joined #openstack-infra13:20
*** mriedem has joined #openstack-infra13:27
fungimnaser: for low-volume/low-data jobs which can run without root access and entirely from statically-defined scripts and preinstalled applications, yes we have some persistent slaves which can act as credential safes13:29
mnaserfungi unfortunately this one would be a bit more on the high volume high data (i.e. publishing docker images to dockerhub for latest master and branches)13:30
fungifor example, pypi releases are built on single-use nodes, the results are published to a holding area (tarballs.o.o), then retrieved by a persistent signing node which generates and publishes a signature and also by a release node which retrieves it, analyzes it, and then uploads it to pypi13:31
mnaseri see, in a kolla case i guess it would be: pull down what is published to tarballs.o.o and publish that back out13:31
mnaserwhich probably adds a bit of overhead13:31
fungimnaser: yeah, we discussed that possibility with the kolla team previously, and the data needed to publish to dockerhub is many orders of magnitude greater than things we publish to pypi, npm, maven central, et cetera so probably need an entirely new solution13:32
mnaseri think a persistent building slave would be best.  because it would also take much less time rebuilding things on new commits because of docker cache13:33
fungii have a suspicion you would be unable to accomplish that without root access and with only preinstalled software and static scripts (not running anything contained within the software repository)13:35
fungithose limitations ensure that a job can't backdoor the node in such a way as to alter future jobs run on the same machine13:36
mnaserfungi i mean we could build kolla as a non root user, the push can happen as non root too13:37
mnaserthe only thing needed is that $user would have to be added to the docker group13:37
mnaserand then for example the sudo /usr/local/bin/push_images.sh is the only thing allowed for $user to sudo into13:37
mnaserand that does the pushing of the images with the credentials stored inside root13:38
mnaserwhich separates the build process 100%, then the push process independently (without the ability to try and sneak something in) -- relying on sudo to stop you from doing bad things13:38
*** ym200 has quit IRC13:39
fungithat script would need some very careful vetting to make sure that environment variables or parameters passed by the calling user can't be leveraged to make it do anything it's not expected to do13:39
mnaserof course13:39
mnaserim thinking /usr/local/bin/push_images.sh $TAG (which is where to tag the images remotely) -- and all that would do is a local loop to discover all images tagged with $TAG and docker push $IMAGE:$TAG13:40
fungiand for safety the job would also only be able to run predefied scripts and applications which are not installed at job runtime and are not contained within the repo13:40
mnaserof course, i agree with that, so the system would have that installed say: part of system-config13:41
mnaserand so that way that script cant be touched without infra-core ok-ing it13:41
fungibasically have to prevent the job from running something defined by the repo being released such that it might leave a process running or change files in the user's homedir that could be used to alter behavior for subsequent jobs13:42
mnaseryup.  now a small unrelated question.  could the slaves be configured to only do 1 job at a time13:43
fungithey can only ever run one job at a time anyway13:43
mnaserwhich is even better13:43
mnaserthat way we dont have to worry about two jobs happening at once13:44
mnaserthanks for all the info fungi -- i'll give this a bit more thought, share with kolla folks and then we can have a small proposal to run by infra13:46
*** esikachev has joined #openstack-infra13:51
*** esikachev has quit IRC13:56
*** funzo has joined #openstack-infra14:09
*** sshnaidm|off has joined #openstack-infra14:09
*** esikachev has joined #openstack-infra14:11
*** funzo has quit IRC14:13
*** esikachev has quit IRC14:15
*** esikachev has joined #openstack-infra14:19
*** yamamoto has quit IRC14:22
*** esikachev has quit IRC14:23
openstackgerritVitaly Gridnev proposed openstack-infra/project-config master: Specify empty config for subnode  https://review.openstack.org/44721514:23
openstackgerritVitaly Gridnev proposed openstack-infra/project-config master: [sahara multinode] Specify empty config for subnode  https://review.openstack.org/44721514:24
*** esikachev has joined #openstack-infra14:25
*** masayukig[m] has joined #openstack-infra14:26
*** ijw has quit IRC14:27
*** ijw has joined #openstack-infra14:29
*** esikachev has quit IRC14:29
*** yamamoto has joined #openstack-infra14:31
*** esikachev has joined #openstack-infra14:43
*** hongbin has joined #openstack-infra14:50
*** Wei_Liu has quit IRC14:53
*** Wei_Liu has joined #openstack-infra14:54
*** jamesdenton has joined #openstack-infra14:54
*** armax has joined #openstack-infra14:58
openstackgerritSteven Dake proposed openstack-infra/project-config master: Make Kolla check nv jobs gate voting  https://review.openstack.org/44721614:58
mnaserclarkb mordred job failed on rax with no ip on eth1.. it looks like the network data was supplied correctly - http://logs.openstack.org/45/446645/4/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/83ec423/console.html#_2017-03-18_14_43_47_28907115:03
*** funzo has joined #openstack-infra15:09
*** hongbin_ has joined #openstack-infra15:12
*** hongbin has quit IRC15:14
*** funzo has quit IRC15:14
*** jaypipes has joined #openstack-infra15:14
*** armax has quit IRC15:15
sdakemnaser fwiw fungi has indicated rax provider is inconsistent in the past and he has been working to make it consistent15:20
*** sshnaidm|off has quit IRC15:20
*** Goneri has joined #openstack-infra15:24
*** sshnaidm|off has joined #openstack-infra15:32
*** yamahata has joined #openstack-infra15:33
*** _Cyclone_ has joined #openstack-infra15:34
*** sshnaidm|off has quit IRC15:38
*** jkilpatr has quit IRC15:44
*** esikachev has quit IRC15:45
*** nicolasbock has quit IRC15:45
*** hongbin has joined #openstack-infra15:49
*** hongbin_ has quit IRC15:51
*** esikachev has joined #openstack-infra15:59
*** mdrabe has joined #openstack-infra16:01
*** mdrabe_ has joined #openstack-infra16:02
*** mdrabe_ has quit IRC16:05
*** mdrabe has quit IRC16:05
*** mdrabe has joined #openstack-infra16:06
*** funzo has joined #openstack-infra16:10
*** knangia has quit IRC16:11
*** esikache1 has joined #openstack-infra16:11
*** larainema has quit IRC16:12
*** esikachev has quit IRC16:13
*** funzo has quit IRC16:14
*** armax has joined #openstack-infra16:15
*** rcernin has joined #openstack-infra16:17
*** david-lyle has quit IRC16:18
*** yamamoto has quit IRC16:20
SpamapSmnaser: it's worth noting that zuulv3 adds the ability for sharing secrets with zuul that can reach out to 3rd party services. So your docker image push will get simpler with v3, I think.16:20
*** yamamoto has joined #openstack-infra16:21
*** sree has joined #openstack-infra16:24
*** yamamoto has quit IRC16:26
*** sree has quit IRC16:28
*** yamamoto has joined #openstack-infra16:31
clarkbmnaser: so its a glean bug, yay16:34
*** tphummel has joined #openstack-infra16:35
*** yamamoto has quit IRC16:36
openstackgerritPaul Belanger proposed openstack-infra/zuul feature/zuulv3: Fix testtools.run syntax for README.rst  https://review.openstack.org/44721916:41
*** rcernin has quit IRC16:45
*** Apoorva has joined #openstack-infra16:45
*** rcernin has joined #openstack-infra16:45
*** yamahata has quit IRC16:46
mnaserclarkb yeah at least we made progress there16:49
mnaseri think its a bug somewhere because it seems to be providing the right info16:49
*** Apoorva has quit IRC16:50
*** armax has quit IRC16:51
*** hongbin has quit IRC16:54
*** armax has joined #openstack-infra17:00
openstackgerritColleen Murphy proposed openstack-infra/puppet-iptables master: Add support for openSUSE  https://review.openstack.org/44722017:00
*** sshnaidm|off has joined #openstack-infra17:05
*** hongbin has joined #openstack-infra17:10
*** psachin has joined #openstack-infra17:11
*** megm has joined #openstack-infra17:13
clarkbmnaser: most likely glean. We'll probably need to set up a check for this case and dump more info when we hit it17:15
*** kzaitsev_mb has quit IRC17:19
*** ssbarnea has joined #openstack-infra17:26
*** yamamoto has joined #openstack-infra17:33
*** mriedem1 has joined #openstack-infra17:35
*** armax has quit IRC17:36
*** mriedem has quit IRC17:37
*** yamamoto has quit IRC17:38
mnaserhttps://review.openstack.org/#/c/446645/17:46
mnaserwould anyone know why gate isnt running for that?17:46
mnasersomeone fired a recheck after the dependent change merged and it rechecked but didnt merge17:47
*** hongbin has quit IRC17:50
*** tphummel has quit IRC17:51
*** rcernin has quit IRC17:53
*** jamesdenton has quit IRC17:58
*** dave-mccowan has joined #openstack-infra17:59
pabelangermnaser: according to gerrit it needs Needs Verified still18:01
mnaserpabelanger: but zuul status shows that its not in queue anywhere18:01
*** sshnaidm|off has quit IRC18:02
pabelangerright, +A won't work, while it has Needs Verified18:02
pabelangerso, another recheck?18:02
pabelangerwill have to check zuul logs to see why that is18:02
*** sshnaidm|off has joined #openstack-infra18:03
mnaserit was passing already and there was one recheck already done (i dont mind doing another one)18:04
pabelangermnaser: try rebasing against master18:06
pabelangerfor some reason, gerrit is giving a black dot on related changes18:06
pabelangerI cannot remember what that dot means18:07
pabelangerOh18:07
pabelangeryou are dependant on a abandon change18:07
pabelangeratleast that is what gerrit thinks18:07
mnaseruh conflicting names maybe?18:08
mnaserhttps://review.openstack.org/#/c/447053/ -> https://review.openstack.org/#/c/446645/ is what the chain supposed to be18:08
pabelanger446648 has the same change-id, I wonder if that is the issue18:08
mnaseri think so18:08
pabelangerwhich is abandoned18:08
*** rcernin has joined #openstack-infra18:08
mnaseri think thats the reason18:09
mnaseris there a way to just delete that whole thing18:10
pabelangerrebase 446645 to master (tip) that should get things going again18:11
*** funzo has joined #openstack-infra18:12
mnaser"base revision is missing: master"18:12
mnasershould i do it locally18:12
pabelangeryes, don't use gerrit ui18:12
*** jaypipes has quit IRC18:13
mnaserpabelanger ok i think i got it right, the parent is changed18:14
mnaserill try avoiding that stuff and make backports after things merge18:14
pabelanger++18:15
mnasermerci pabelanger !18:15
*** Julien-z_ has quit IRC18:16
*** funzo has quit IRC18:17
*** Julien-zte has joined #openstack-infra18:20
*** rcernin has quit IRC18:23
*** bhavik1 has joined #openstack-infra18:24
*** dave-mccowan has quit IRC18:27
*** esikache1 has quit IRC18:29
*** sdague_ has joined #openstack-infra18:29
*** sdague_ has quit IRC18:29
*** nadya has joined #openstack-infra18:34
*** psachin has quit IRC18:36
*** vsaienko has joined #openstack-infra18:36
*** bhavik1 has quit IRC18:37
*** wolverineav has joined #openstack-infra18:40
*** wolverineav has quit IRC18:41
*** wolverineav has joined #openstack-infra18:42
*** ihrachys has joined #openstack-infra18:44
*** ihrachys_ has joined #openstack-infra18:46
*** ihrachys has quit IRC18:48
*** bhavik1 has joined #openstack-infra18:50
*** esikache1 has joined #openstack-infra18:50
*** zz_dimtruck is now known as dimtruck19:02
*** ihrachys_ has quit IRC19:06
*** nadya has quit IRC19:06
*** ihrachys has joined #openstack-infra19:11
*** funzo has joined #openstack-infra19:12
*** esikache1 has quit IRC19:14
*** Julien-zte has quit IRC19:15
*** funzo has quit IRC19:17
*** Julien-zte has joined #openstack-infra19:17
*** ssbarnea has quit IRC19:20
*** ssbarnea has joined #openstack-infra19:21
*** esikache1 has joined #openstack-infra19:30
*** wolverineav has quit IRC19:31
*** esikache1 has quit IRC19:35
*** wolverineav has joined #openstack-infra19:41
*** esikache1 has joined #openstack-infra19:42
*** flepied has quit IRC19:43
*** sc68cal has joined #openstack-infra19:43
*** Sukhdev has joined #openstack-infra19:43
*** wolverineav has quit IRC19:45
*** sc68cal has quit IRC19:50
*** esikache1 has quit IRC19:52
*** sc68cal has joined #openstack-infra19:54
mrmartinfungi: groups.o.o is using oauth2 for authentication, and stores the full oauth2 username in users table's data field's oauth2_fullname field. the users.data field is json encoded.19:56
fungimrmartin: okay, so it's possible those didn't change when the openid strings got normalized in openstackid? or i guess i'll wait for someone to report an issue in that case if it's unlikely to have been affected20:01
*** wolverineav has joined #openstack-infra20:01
mrmartinfungi: i think so20:01
*** flepied has joined #openstack-infra20:01
fungiawesome, thanks!20:02
*** ihrachys has quit IRC20:02
*** bhavik1 has quit IRC20:02
*** wolverineav has quit IRC20:05
*** sshnaidm|off has quit IRC20:06
*** neiljerram has joined #openstack-infra20:07
*** ihrachys has joined #openstack-infra20:08
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Fix beaker-rspec tests  https://review.openstack.org/44724020:08
*** vsaienko has quit IRC20:08
*** mdrabe has quit IRC20:09
*** funzo has joined #openstack-infra20:13
*** ihrachys has quit IRC20:13
*** vsaienko has joined #openstack-infra20:14
*** markvoelker has joined #openstack-infra20:14
*** sirushti has quit IRC20:15
*** funzo has quit IRC20:18
*** sirushti has joined #openstack-infra20:18
*** EricGonczer_ has joined #openstack-infra20:19
*** esikache1 has joined #openstack-infra20:22
*** neiljerram has quit IRC20:26
*** esikache1 has quit IRC20:26
*** nadya has joined #openstack-infra20:27
*** nadya has quit IRC20:28
*** andreww has quit IRC20:29
*** wolverineav has joined #openstack-infra20:30
*** Jeffrey4l__ has joined #openstack-infra20:31
*** Jeffrey4l__ has quit IRC20:33
*** vsaienko has quit IRC20:33
*** esikache1 has joined #openstack-infra20:33
*** Jeffrey4l__ has joined #openstack-infra20:34
*** Jeffrey4l_ has quit IRC20:34
*** wolverineav has quit IRC20:35
*** vsaienko has joined #openstack-infra20:36
*** esikache1 has quit IRC20:38
*** wolverineav has joined #openstack-infra20:40
*** esikache1 has joined #openstack-infra20:41
*** Julien-zte has quit IRC20:41
*** Julien-zte has joined #openstack-infra20:43
*** wolverineav has quit IRC20:44
*** esikache1 has quit IRC20:47
*** EricGonczer_ has quit IRC20:48
*** Julien-zte has quit IRC21:00
*** Julien-zte has joined #openstack-infra21:03
*** Sukhdev has quit IRC21:04
*** nicolasbock has joined #openstack-infra21:04
*** Sukhdev has joined #openstack-infra21:05
*** vsaienko has quit IRC21:07
*** wolverineav has joined #openstack-infra21:09
*** vsaienko has joined #openstack-infra21:10
*** wolverineav has quit IRC21:14
*** funzo has joined #openstack-infra21:14
*** andreww has joined #openstack-infra21:14
*** funzo has quit IRC21:19
*** esikache1 has joined #openstack-infra21:19
*** wolverineav has joined #openstack-infra21:19
*** esikache1 has quit IRC21:23
*** wolverineav has quit IRC21:24
*** esikache1 has joined #openstack-infra21:29
*** Jeffrey4l__ has quit IRC21:35
*** esikache1 has quit IRC21:37
*** Sukhdev has quit IRC21:38
*** sshnaidm|off has joined #openstack-infra21:38
*** mriedem1 is now known as mriedem21:39
*** wolverineav has joined #openstack-infra21:39
*** dimtruck is now known as zz_dimtruck21:42
*** wolverineav has quit IRC21:43
*** Jeffrey4l__ has joined #openstack-infra21:47
*** Julien-zte has quit IRC21:49
*** Julien-z_ has joined #openstack-infra21:50
*** nicolasbock has quit IRC21:50
*** vsaienko has quit IRC21:51
*** esikache1 has joined #openstack-infra21:51
*** Julien-z_ has quit IRC21:55
*** esikache1 has quit IRC21:55
*** EricGonczer_ has joined #openstack-infra21:58
*** wolverineav has joined #openstack-infra21:58
*** Julien-zte has joined #openstack-infra21:59
*** hongbin has joined #openstack-infra22:00
*** wolverineav has quit IRC22:03
*** esikache1 has joined #openstack-infra22:11
*** funzo has joined #openstack-infra22:15
*** esikache1 has quit IRC22:15
*** dave-mccowan has joined #openstack-infra22:17
*** wolverineav has joined #openstack-infra22:18
*** funzo has quit IRC22:19
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Fix beaker-rspec tests  https://review.openstack.org/44724022:22
*** wolverineav has quit IRC22:23
*** Julien-zte has quit IRC22:25
*** funzo has joined #openstack-infra22:25
*** esikache1 has joined #openstack-infra22:26
*** dave-mccowan has quit IRC22:27
*** Julien-zte has joined #openstack-infra22:28
*** dave-mccowan has joined #openstack-infra22:28
*** wolverineav has joined #openstack-infra22:28
*** esikache1 has quit IRC22:31
*** wolverineav has quit IRC22:36
*** dave-mccowan has quit IRC22:36
*** EricGonczer_ has quit IRC22:37
*** IRCFrEAK has joined #openstack-infra22:40
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Fix beaker-rspec tests  https://review.openstack.org/44724022:42
*** Julien-zte has quit IRC22:43
*** Julien-zte has joined #openstack-infra22:45
*** zz_dimtruck is now known as dimtruck22:49
*** asselin has quit IRC22:50
*** IRCFrEAK has quit IRC22:51
*** esikache1 has joined #openstack-infra22:51
*** ssbarnea has quit IRC22:55
*** esikache1 has quit IRC22:57
*** andreww has quit IRC23:00
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Run puppet 4 in beaker tests  https://review.openstack.org/44724323:03
*** dimtruck is now known as zz_dimtruck23:04
*** zz_dimtruck is now known as dimtruck23:04
*** david-lyle has joined #openstack-infra23:05
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Fix beaker-rspec tests  https://review.openstack.org/44724023:10
openstackgerritColleen Murphy proposed openstack-infra/puppet-openstackci master: Run puppet 4 in beaker tests  https://review.openstack.org/44724323:10
*** baoli_ has joined #openstack-infra23:17
*** markvoelker has quit IRC23:17
*** baoli has quit IRC23:18
*** ijw has quit IRC23:29
*** ijw has joined #openstack-infra23:32
*** iOx has joined #openstack-infra23:45
*** iOx has quit IRC23:56
*** andreww has joined #openstack-infra23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!