Saturday, 2018-01-06

« Friday, 2018-01-05 Index Sunday, 2018-01-07 »
*** jascott1 has joined #openstack-infra00:00
clarkbthe way the script works is it checks if the device exists and makes swap on it if so. If no device its a noop00:00
clarkbwhich is important on non rax clouds without extra disks, unless we want to use swap files as fallback00:00
*** bobh has joined #openstack-infra00:03
pabelangeryah, the last time we had no swap on boot, it was because something in deviceblock changed, maybe that is the case again.00:03
pabelangerbut, this has been a few times where we boot server only to find out later swap is missing. So we'll need a better way to confirm swap is created when needed00:04
clarkboh wow its 4pm on friday. where has the day gone. Re ubuntu kernels I'll probably check in over the weekend in hopes that ubuntu has kernels out before the 9th and start rolling those out if so. I do have a eyeball examination appointment on monday morning. Slightly concerned they will dilate my pupils making usage of computer hard so I may not be around for early monday fun00:04
*** bobh_ has joined #openstack-infra00:04
*** jascott1 has quit IRC00:04
pabelanger++ I can also help with that too00:04
jlvillalAnyone know why the Gerrit page doesn't show the failed job in the job list? https://review.openstack.org/#/c/527743/00:05
jlvillalThe Zuul comment message says the grenade job failed, but I don't see it in the list of jobs up top00:05
clarkbpabelanger: in eavesdrop case I don't see an epehemeral device, there is / and the data volume00:05
clarkbpabelanger: so maybe just falling back on swapfile is sane?00:05
dmsimardlooks like openafs is failing to build on ze09 rebuild00:05
fungipabelanger: yeah, i have a suspicion 1gb flavors might not get ephemeral disk, but i haven't tested that theory00:05
dmsimardoverlapping processses: /bin/bash /usr/sbin/dkms build -m openafs -v 1.6.15 -k 4.4.0-104-generic00:06
pabelangerfungi: clarkb: okay, I can poke into that next week00:06
dmsimardor maybe it's just taking a long time, I dunno00:06
pabelangerdmsimard: will take some time00:06
clarkbjlvillal: you need to toggleci. its because the tableification of zuul comments doesn't understand retry limit00:06
*** sree has quit IRC00:06
jlvillalclarkb: Okay. Yeah I saw the retrylimit in the zuul comments. I was just unsure why not in the main list up top. So sounds like known issue.00:07
jlvillalclarkb: Thanks00:07
clarkbjlvillal: the general known issue is the table genreation doesn't account for all cases where zuul might -1 a change. Requirements job fails come with lots of verbage that breaks that too for example. But also if you make zuulv3 job changes that don't prase you get lots of words that it doesn't understand as well00:08
jlvillalclarkb: Ah okay. Good to know.00:08
*** bobh has quit IRC00:08
* jlvillal is seeing a fair number of retry_limit failures at the moment.00:08
*** jkilpatr has quit IRC00:08
clarkbjlvillal: is it specific to that job? retry limits happen when failures occur in the pre steps of a job. Zuul will attempt to run the job again in that case up to a retry limit then fail with that message00:09
jlvillalclarkb: Looking at Zuul right now with a filter of 'ironic'00:09
clarkbif its for a specific job then the jobs pre steps might be broken, if its all overthen more likely something like networking00:09
jlvillalclarkb: The two ironic jobs both have retry_limit failures00:09
jlvillalclarkb: Though the exact two same jobs in both...00:10
jlvillalThe links seem to go nowhere at the moment. So I can't read the logs00:10
*** jkilpatr has joined #openstack-infra00:11
clarkbya I thought we still tried to collect logs which makes me think maybe the jobs aren't ever really starting.00:11
*** sree has joined #openstack-infra00:11
clarkbdmsimard: ^ is ze09 running a zuul-executor that might be breaking things?00:11
jlvillalOne patch has 2 retry_limit failures, the other patch now has 3 retry_limit failures00:11
* clarkb greps for retry_limit in logs00:13
jlvillalclarkb: Yeah, seems to be happening all over the place. I see it in neutron and cinder jobs too.00:13
clarkbhrm this isn't the easiest thing ot track in the logs as the executors are unaware of the global number of retries and the scheduler doesn't seem to know why things are failing00:15
clarkbits ze10 I think00:18
clarkbdmsimard: ^ fyi, but looking into it00:19
*** jkilpatr has quit IRC00:19
clarkbthe git repo for tempest is corrupt on 10 I think00:20
clarkbI'm stopping the executor there00:20
corvusthere is no ze09 connected00:21
clarkbcorvus: http://paste.openstack.org/show/639292/00:21
clarkbya its ze1000:21
corvusyep.  just thought i'd confirm in case there was >1 problem00:22
corvusclarkb: i think it would be okay to delete everything under executor-git to be safe00:22
clarkbI think we can delete the contents of the executor-git dir and have zuul rebuild it right?00:22
clarkbcool00:22
corvus:)00:22
clarkbjust waiting on zuul-executor to be stopped then we can do that and start it again00:23
clarkbcorvus: might also want to have an executor startup step for running git status on repos to quickly canary problems like this?00:23
pabelangermaybe we can consider adding a zuul-executor field to  http://zuulv3.openstack.org/builds.html to help sort by failures00:23
corvusclarkb: yeah... git status is probably fast enough for an executor startup00:24
clarkbok zuul is mostly stopped00:25
openstackgerritJames E. Blair proposed openstack-infra/system-config master: Add note on how to talk to zuul's gearman  https://review.openstack.org/53152200:25
clarkbprocesses like 1 S zuul       499     1  0  80   0 - 11262 -      00:17 ?        00:00:00 ssh: /var/lib/zuul/builds/9dcb31c409e2400fabd2d4c8a8c8be14/.ansible/cp/f9805f2cac [mux] are all that are left00:26
corvusclarkb: ^ first time i've connected to gearman manually with ssl; i documented it00:26
clarkbcorvus: ^ any idea what those are before I clear them out and then delete the git repos?00:26
corvusclarkb: they will die on their own in 60s00:27
corvusclarkb: that's the controlpersist processes00:27
clarkbok so can be ignored /me clears out git repos and build dirs00:28
*** Swanson has quit IRC00:30
clarkbexecutor is started and cloning things, watching it by tailing the log file00:30
dmsimardstepped away momentarily00:31
clarkbjlvillal: I think things should be better now. Sorry for the noise00:31
dmsimard"service zuul-executor stop" isn't working, should it ?00:34
dmsimardthe processes stay alive00:34
clarkbdmsimard: it does a graceful ish stop00:34
dmsimardthere's nothing running though00:35
dmsimard(new ze09)00:35
clarkboh this may be a corner case on the graceful stopping? eg if there is nothing to wait for stopping then it spins indefinitely?00:35
clarkbor its blocking on some thread that is trying t oconnect ot zk etc00:35
pabelangerwhat is in debug.log? is it trying to connect to zookeeper and failing before of firewall DNS?00:35
clarkbbut since that is blocking it ya that00:36
dmsimardhttp://paste.openstack.org/raw/639293/00:36
*** markvoelker has quit IRC00:37
pabelangerit is blocked on gearman I think00:37
*** sree has quit IRC00:37
pabelangerlikely firewall rules need refreshed00:37
pabelangerfor wait until puppet kicks them00:37
pabelanger2018-01-06 00:11:26,518 DEBUG zuul.ExecutorServer: Waiting for server00:37
dmsimardpabelanger: I don't want executor to run00:37
dmsimardwe just rebuilt it and I need to remount /var/lib/zuul into a partition on the additional ephemeral disk00:38
dmsimardand stopping zuul-executor doesn't really stop it00:38
pabelangerdmsimard: right, it cannot be stopped because socket hasn't been open. Guess we have a bug00:38
pabelangeruntil it connects to gearman00:38
clarkb3108 doesn't seem to be running though00:39
clarkbso that one did stop?00:39
clarkbthe two processes in the cgroup are the ones running00:39
pabelangerdmsimard: kill -9 would work00:39
dmsimardof course it would :p00:39
dmsimardkilled00:40
dmsimardI'll do the legwork and start it00:40
pabelangerdmsimard: http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?h=feature/zuulv3#n167900:40
pabelangerwe need to move that up before merger starts00:40
*** claudiub has quit IRC00:41
*** sree has joined #openstack-infra00:41
pabelangersomething to test for sure, if gearman is blocking, not sure stop would work anyways00:42
clarkbiirc we do something like while not connected sleep(0.1) we could probably check for a state change there too00:42
*** jkilpatr has joined #openstack-infra00:47
dmsimardthe service is starting but the process isn't staying alive, no hint in the logs00:47
fungilook for a stale pidfile00:48
dmsimardthere was one, let's see now00:48
*** apevec has quit IRC00:49
dmsimardyeah starting it, there's a few lines in the logs (starting streamer, stopping streamer), it leaves a pidfile behind and doesn't stay alive00:50
pabelangerI wrote a systemd file for zuul / nodepool today for zuulv3 docs. I'm going to push it up to puppet-zuul next week, should help with stale pidfiles issues00:51
fungidmsimard: lack of write perms?00:51
fungi(to, say, /var/lib/zuul)00:51
dmsimardthere's this stack trace that occurs when, I believe, I issue a stop on the service http://paste.openstack.org/raw/639295/00:51
dmsimardfungi: I'd love the logs to tell me that :D00:52
pabelangerdmsimard: yah, I'm working on a patch to address that. will push it up monday00:52
fungidmsimard: agreed, though startup issues can often race logging setup which makes it harder to be sur you catch some of that00:53
dmsimard /var/lib/zuul looks fine ... is there a way to run puppet manually ? maybe it needs to do something after remounting /var/lib/zuul (although from looking at history of another ze it didn't seem like there was anything)00:54
*** sree has quit IRC00:54
pabelangerdmsimard: /var/run/zuul/zuul-executor.pid exists00:55
pabelangertry deleting that00:55
dmsimardpabelanger: been there done that00:55
dmsimardjust did it now, tried starting, nothing00:56
pabelangerdmsimard: might if I try something?00:58
dmsimardpabelanger: go ahead00:58
dmsimardpabelanger: mind you, I haven,t done any firewall business on other nodes00:59
dmsimardin case there's something related to that01:00
dmsimardcould probably check tcpdump to see if it attempts connecting anywhere01:00
pabelangerPermissionError: [Errno 13] Permission denied: '/var/lib/zuul/ansible'01:00
pabelangernow, why would that be01:01
pabelangeroh01:01
pabelangerdmsimard: /var/lib/zuul has wrong permissions01:01
*** jascott1 has joined #openstack-infra01:01
pabelangershould be zuul:zuul01:01
pabelangerdmsimard: if you kick.sh ze09.o.o from puppetmaster.o.o, should fix it01:02
dmsimardhmm, it was zuul:zuul before.. drwxr-xr-x  8 zuul    zuul    4096 Jan  6 00:11 zuul01:04
dmsimardI guess I must have screwed up something at some point when doing the remount01:04
dmsimardpabelanger: where is kick.sh ?01:04
fungisystem-config tools01:05
fungiand yeah, i had a feeling it was ownership on that dir since you needed to recreate it01:05
pabelanger/opt/system-config/production/tools/kick.sh01:05
*** bobh_ has quit IRC01:06
*** bobh has joined #openstack-infra01:06
dmsimardfungi: I didn't re-create it :( I moved its contents elsewhere01:07
fungidmsimard: when you mount a filesystem at a mount point, the top-level directory's perms and ownership from that fs shadow those of the mount point01:07
*** pramodrj07 has quit IRC01:07
dmsimardI went from the history of ze0801:07
*** caphrim007 has joined #openstack-infra01:07
fungiso you generally need to chmod/chown the fs to match after mounting01:08
dmsimardyeah I figured it had to be from the mount01:08
dmsimardthanks for being patient with me as I learn through this :)01:09
fungiyou bet!01:09
*** ijw has joined #openstack-infra01:09
dmsimardso to make a long story short, 09 and 10 being out since yesterday probably did not help our OOM issues01:10
fungiagreed01:10
*** xarses_ has quit IRC01:10
fungiput more pressure on the rest01:10
pabelangerIIRC, launch-node.py does take a mount-point option, maybe we should see if we can get that working with /var/lib/zuul01:10
dmsimardzuul-executor process is running /o/01:11
*** caphrim007 has quit IRC01:11
dmsimardnow what firewall holes needs to be poked ? zookeeper ?01:11
pabelangerdidn't we fix that with new puppet firewall stuff?01:11
pabelangerI haven't actually seen how that works now01:12
pabelangerbut zookeeper is running on nodepool.o.o01:12
dmsimardI thought so too but clarkb mentioned something, let me read backlog01:12
pabelangerkk01:12
clarkbpuppet resolves the names then writes IP addresses to the config on host01:13
dmsimardoh that's how it works ?01:13
dmsimardso puppet needs to re-run after dns has been updated01:14
clarkbya01:14
pabelangercool, in that case, wait until puppet wheel or kick.sh nodepool.o.o01:14
pabelangerbut, that is pretty cool now :)01:15
dmsimardif I check iptables on nodepool.o.o01:15
dmsimardI don't see the IP of, say, ze0801:15
dmsimardso I'm not sure this is the right place01:15
dmsimardthere's a few IPs for port 2181 though01:16
pabelangeryah, so sounds like you need to reload firewall01:16
dmsimardpabelanger: looking at /etc/iptables/rules.v4, I see this: http://paste.openstack.org/show/639299/01:17
pabelangeryah, I don't think we have new iptables_allowed_hosts on nodepool.o.o01:17
pabelangerlooking at site.pp01:17
dmsimard(amongst other things)01:17
dmsimardthere's no executors in there01:17
pabelangeroh, right01:17
pabelangerexecutors just need gearman01:18
pabelangernot zookeeper01:18
pabelangerso, that is on zuulv3.o.o01:18
dmsimardoh so on zuulv3.o.o01:18
dmsimardyup I see the rules01:18
dmsimardi.e, -A openstack-INPUT -m state --state NEW -m tcp -p tcp --dport 4730 -s ze09.openstack.org -j ACCEPT01:18
dmsimardso restart iptables-persistent I guess01:19
dmsimard(or reload perhaps?)01:19
pabelangerreload01:19
pabelangeris what I've used01:19
pabelangerthere it goes01:19
dmsimardok the new ip is there01:19
pabelangerand see cloning now01:19
dmsimardyay01:19
pabelangerwe can push up a patch next week and use new firewall stuff :)01:20
dmsimard#status log (dmsimard) ze09.o.o was rebuilt from scratch after what seems to be a failed live migration which thrashed the root partition disk01:20
openstackstatusdmsimard: finished logging01:20
dmsimardpabelanger: I'd like to be hinted at the permission issue somehow too01:20
*** bobh has quit IRC01:20
dmsimardfor /var/lib/zuul01:20
dmsimardalso maybe, "hey, I can't connect to my gearman server"01:20
pabelangerdmsimard: yah, leif and I had some issues today about that too.  I think it would be sane to output more verbose messages on those failures01:21
dmsimardany messages at all would be better than the current nothing, regardless of verbosity :)01:21
pabelangeragree01:21
dmsimardI can create something in storyboard I guess01:21
pabelangerstepping away for a bit01:21
*** sree has joined #openstack-infra01:23
*** sree has quit IRC01:27
dmsimardcreated 2 storyboard issues in zuul regarding improvement opportunities found in this learning experience :D01:28
* dmsimard steps away too01:28
dmsimardbefore stepping away, I realized I also needed to kick graphite.openstack.org to update the iptables rules01:30
* dmsimard steps away01:30
*** bobh has joined #openstack-infra01:33
*** markvoelker has joined #openstack-infra01:35
*** sree has joined #openstack-infra01:36
*** wolverineav has joined #openstack-infra01:39
*** sree has quit IRC01:41
*** tosky has quit IRC01:43
*** wolverineav has quit IRC01:44
*** caphrim007 has joined #openstack-infra01:47
*** xarses_ has joined #openstack-infra01:47
*** abelur_ has joined #openstack-infra01:47
*** sree has joined #openstack-infra01:49
*** andreww has joined #openstack-infra01:50
*** edmondsw has joined #openstack-infra01:50
*** xarses_ has quit IRC01:52
*** sree has quit IRC01:54
*** edmondsw has quit IRC01:55
*** andreww has quit IRC02:02
*** mriedem has quit IRC02:11
*** bobh has quit IRC02:16
*** chrisyang_0660_ has joined #openstack-infra02:19
*** kklimonda_ has joined #openstack-infra02:19
*** ntpttr_ has joined #openstack-infra02:20
*** yamahata__ has joined #openstack-infra02:20
*** rajinir_ has joined #openstack-infra02:20
*** kuromagi has joined #openstack-infra02:20
*** hogepodge_ has joined #openstack-infra02:21
*** johnsom_ has joined #openstack-infra02:21
*** kzaitsev1pi has joined #openstack-infra02:21
*** HenryG_ has joined #openstack-infra02:21
*** tinwood_ has joined #openstack-infra02:21
*** SamYaple_ has joined #openstack-infra02:22
*** persia__ has joined #openstack-infra02:22
*** zerick_ has joined #openstack-infra02:22
*** Roamer`_ has joined #openstack-infra02:22
*** trown has joined #openstack-infra02:23
*** alex_xu_ has joined #openstack-infra02:23
*** jmccrory_ has joined #openstack-infra02:25
*** uberjay_ has joined #openstack-infra02:25
*** masayukig_ has joined #openstack-infra02:25
*** dims_ has joined #openstack-infra02:25
*** Hunner- has joined #openstack-infra02:25
*** lastmikoi_ has joined #openstack-infra02:25
*** Gorian- has joined #openstack-infra02:26
*** flwang has quit IRC02:26
*** AJaeger has quit IRC02:26
*** larainema has quit IRC02:26
*** RuiChen has quit IRC02:26
*** lastmikoi has quit IRC02:26
*** dims has quit IRC02:26
*** zerick has quit IRC02:26
*** chrisyang_0660 has quit IRC02:26
*** ajo has quit IRC02:26
*** ericyoung has quit IRC02:26
*** rajinir has quit IRC02:26
*** kzaitsev_pi has quit IRC02:26
*** uberjay has quit IRC02:26
*** persia has quit IRC02:26
*** masayukig has quit IRC02:26
*** hogepodge has quit IRC02:26
*** kklimonda has quit IRC02:26
*** johnsom has quit IRC02:26
*** Nil_ has quit IRC02:26
*** isviridov_away has quit IRC02:26
*** vaidy has quit IRC02:26
*** HenryG has quit IRC02:26
*** Roamer` has quit IRC02:26
*** jmccrory has quit IRC02:26
*** rm_work has quit IRC02:26
*** comstud has quit IRC02:26
*** Gorian has quit IRC02:26
*** kuromagi^ has quit IRC02:26
*** Hunner_ has quit IRC02:26
*** tinwood has quit IRC02:26
*** robled has quit IRC02:26
*** trown|outtypewww has quit IRC02:26
*** alex_xu has quit IRC02:26
*** mordred has quit IRC02:26
*** dteselkin has quit IRC02:26
*** asilenkov has quit IRC02:26
*** ntpttr__ has quit IRC02:26
*** isq_ has quit IRC02:26
*** SamYaple has quit IRC02:26
*** chrisyang_0660_ is now known as chrisyang_066002:26
*** hogepodge_ is now known as hogepodge02:27
*** rajinir_ is now known as rajinir02:27
*** kklimonda_ is now known as kklimonda02:27
*** lastmikoi_ is now known as lastmikoi02:27
*** jmccrory_ is now known as jmccrory02:27
*** comstud has joined #openstack-infra02:27
*** robled has joined #openstack-infra02:27
*** robled has joined #openstack-infra02:27
*** dteselkin has joined #openstack-infra02:28
*** rm_work has joined #openstack-infra02:28
*** asilenkov has joined #openstack-infra02:28
*** toabctl has quit IRC02:29
*** ericyoung has joined #openstack-infra02:29
*** johnsom_ is now known as johnsom02:29
*** bandini has quit IRC02:30
*** isviridov_away has joined #openstack-infra02:31
*** bandini has joined #openstack-infra02:31
*** toabctl has joined #openstack-infra02:32
*** Nil_ has joined #openstack-infra02:33
*** AJaeger has joined #openstack-infra02:33
*** mordred has joined #openstack-infra02:33
*** flwang1 has joined #openstack-infra02:33
*** bobh has joined #openstack-infra02:34
*** vaidy has joined #openstack-infra02:35
*** iyamahat has quit IRC02:37
*** harlowja has quit IRC02:38
*** sree has joined #openstack-infra02:40
*** Hunner- has quit IRC02:43
*** Hunner- has joined #openstack-infra02:43
*** Hunner- is now known as Hunner_02:43
*** sree has quit IRC02:45
*** sree has joined #openstack-infra02:51
*** d0ugal has quit IRC02:53
*** ramishra has joined #openstack-infra02:53
*** sree has quit IRC03:03
*** d0ugal has joined #openstack-infra03:10
*** harlowja has joined #openstack-infra03:10
*** sree has joined #openstack-infra03:11
*** abelur_ has quit IRC03:12
*** sree has quit IRC03:15
*** harlowja has quit IRC03:19
*** M4g1c5t0rM has joined #openstack-infra03:22
*** sree has joined #openstack-infra03:27
*** sree has quit IRC03:32
*** edmondsw has joined #openstack-infra03:38
*** bobh has quit IRC03:42
*** dave-mccowan has joined #openstack-infra03:43
*** edmondsw has quit IRC03:43
*** sree has joined #openstack-infra03:51
*** sree has quit IRC04:00
*** sree has joined #openstack-infra04:02
*** dave-mccowan has quit IRC04:08
*** Douhet has quit IRC04:10
*** RuiChen has joined #openstack-infra04:14
*** ajo has joined #openstack-infra04:16
*** larainema has joined #openstack-infra04:16
*** ramishra has quit IRC04:17
*** ijw has quit IRC04:20
*** armax has quit IRC04:23
*** sree has quit IRC04:24
*** armaan has joined #openstack-infra04:27
*** bhavik1 has joined #openstack-infra04:30
*** bhavik1 has quit IRC04:33
*** M4g1c5t0rM has quit IRC04:40
*** david-lyle has quit IRC04:58
*** david-lyle has joined #openstack-infra04:59
*** david-lyle has quit IRC05:00
*** david-lyle has joined #openstack-infra05:01
*** armaan has quit IRC05:18
*** spzala has quit IRC05:25
*** edmondsw has joined #openstack-infra05:27
*** spzala has joined #openstack-infra05:28
*** edmondsw has quit IRC05:31
*** spzala has quit IRC05:33
*** slaweq has joined #openstack-infra05:42
*** slaweq has quit IRC05:47
*** andreww has joined #openstack-infra06:09
*** xarses_ has joined #openstack-infra06:12
*** andreww has quit IRC06:13
*** harlowja has joined #openstack-infra06:26
*** sree has joined #openstack-infra06:30
*** sree has quit IRC06:34
*** yamamoto has joined #openstack-infra06:41
*** slaweq has joined #openstack-infra06:42
*** yamamoto has quit IRC06:46
*** slaweq has quit IRC06:47
*** kmalloc has quit IRC06:59
*** sree has joined #openstack-infra07:02
*** ijw has joined #openstack-infra07:03
*** sree has quit IRC07:07
*** ijw has quit IRC07:07
*** edmondsw has joined #openstack-infra07:15
*** greghayn1 has joined #openstack-infra07:19
*** edmondsw has quit IRC07:19
*** greghaynes has quit IRC07:21
*** kiennt26 has joined #openstack-infra07:29
*** greghayn1 has quit IRC07:32
*** kiennt26 has quit IRC07:36
*** {archer} is now known as chandankumar07:36
*** wolverineav has joined #openstack-infra07:42
*** spzala has joined #openstack-infra07:45
*** greghayn1 has joined #openstack-infra07:46
*** wolverineav has quit IRC07:47
*** frickler has quit IRC07:48
*** frickler has joined #openstack-infra07:48
*** frickler has quit IRC07:48
*** spzala has quit IRC07:49
*** frickler has joined #openstack-infra07:49
*** harlowja has quit IRC08:00
*** sree has joined #openstack-infra08:02
*** sree has quit IRC08:11
*** sree has joined #openstack-infra08:24
*** pcichy_ has joined #openstack-infra08:28
*** sree has quit IRC08:29
*** sree has joined #openstack-infra08:36
*** sree has quit IRC08:41
*** shardy_afk has quit IRC08:43
*** ramishra has joined #openstack-infra08:49
*** sree has joined #openstack-infra08:51
*** sree has quit IRC08:55
*** edmondsw has joined #openstack-infra09:03
*** jascott1 has quit IRC09:06
*** edmondsw has quit IRC09:07
*** numans has quit IRC09:15
*** numans has joined #openstack-infra09:16
*** ramishra has quit IRC09:16
*** dteselkin has quit IRC09:19
*** aviau has quit IRC09:20
*** numans has quit IRC09:20
*** benj_ has quit IRC09:20
*** aviau has joined #openstack-infra09:21
*** swest has quit IRC09:22
*** dteselkin has joined #openstack-infra09:22
*** benj_ has joined #openstack-infra09:22
*** sree has joined #openstack-infra09:25
*** swest has joined #openstack-infra09:25
*** numans has joined #openstack-infra09:25
*** jtomasek has joined #openstack-infra09:26
*** sree has quit IRC09:30
*** jtomasek has quit IRC09:38
*** sree has joined #openstack-infra09:51
*** slaweq has joined #openstack-infra09:56
*** sree has quit IRC09:56
openstackgerritAndreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147009:57
*** sree has joined #openstack-infra10:03
*** sree has quit IRC10:08
openstackgerritAndreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147010:08
* AJaeger figured out the bindep/virtualenv/ansible problem - an absolute path for virtualenv fixed it10:09
openstackgerritAndreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147010:09
*** pbourke has quit IRC10:11
*** pbourke has joined #openstack-infra10:12
*** sree has joined #openstack-infra10:17
*** sree has quit IRC10:22
openstackgerritAndreas Jaeger proposed openstack-infra/bindep master: Add native Zuul v3 bindep-fallback test  https://review.openstack.org/53147010:26
*** slaweq has quit IRC10:28
openstackgerritAndreas Jaeger proposed openstack-infra/project-config master: Remove bindep-fallback from bindep  https://review.openstack.org/53153610:29
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy bindep-ballback jobs  https://review.openstack.org/53153710:29
openstackgerritAndreas Jaeger proposed openstack-infra/openstack-zuul-jobs master: Remove legacy-puppet-murano-puppet-beaker-rspec  https://review.openstack.org/53153810:33
*** slaweq has joined #openstack-infra10:46
*** eumel8 has quit IRC10:49
*** eumel8 has joined #openstack-infra10:49
*** edmondsw has joined #openstack-infra10:51
*** edmondsw has quit IRC10:56
*** sdague has joined #openstack-infra11:09
*** pcichy_ has quit IRC11:17
*** slaweq has quit IRC11:24
*** slaweq has joined #openstack-infra11:25
*** slaweq has quit IRC11:25
*** slaweq has joined #openstack-infra11:25
*** slaweq has quit IRC11:28
*** slaweq has joined #openstack-infra11:35
*** yamamoto has joined #openstack-infra11:37
*** slaweq has quit IRC11:39
*** cmurphy has quit IRC11:43
*** spzala has joined #openstack-infra11:45
*** spzala has quit IRC11:49
*** yamamoto has quit IRC11:52
*** cmurphy has joined #openstack-infra11:53
*** sree has joined #openstack-infra11:57
*** numans has quit IRC12:00
*** numans has joined #openstack-infra12:00
*** sree has quit IRC12:02
*** numans has quit IRC12:04
*** sree has joined #openstack-infra12:07
*** numans has joined #openstack-infra12:11
*** sree has quit IRC12:12
openstackgerritliusheng proposed openstack-infra/zuul feature/zuulv3: Use hotlink instead log url in github job report  https://review.openstack.org/53154512:12
*** sree has joined #openstack-infra12:23
*** spzala has joined #openstack-infra12:27
*** sree has quit IRC12:30
*** claudiub has joined #openstack-infra12:32
*** spzala has quit IRC12:32
*** sree has joined #openstack-infra12:37
*** edmondsw has joined #openstack-infra12:39
*** sree has quit IRC12:41
*** sree has joined #openstack-infra12:43
*** edmondsw has quit IRC12:43
*** sree has quit IRC12:52
*** yamamoto has joined #openstack-infra12:52
*** yamamoto has quit IRC13:02
*** sdague has quit IRC13:11
*** dave-mccowan has joined #openstack-infra13:14
*** yamamoto has joined #openstack-infra13:15
*** sree has joined #openstack-infra13:23
*** dave-mcc_ has joined #openstack-infra13:23
*** dave-mccowan has quit IRC13:26
*** yamamoto has quit IRC13:30
*** afred312 has joined #openstack-infra13:33
*** slaweq has joined #openstack-infra13:35
*** sree has quit IRC13:40
*** slaweq has quit IRC13:41
*** sree has joined #openstack-infra13:46
*** linkmark has joined #openstack-infra13:47
*** sree has quit IRC13:51
*** sree has joined #openstack-infra13:54
*** yamamoto has joined #openstack-infra13:57
*** yamamoto has quit IRC13:57
*** dave-mcc_ has quit IRC14:02
*** sree has quit IRC14:08
*** sree has joined #openstack-infra14:12
*** yamamoto has joined #openstack-infra14:14
*** sree has quit IRC14:17
*** mriedem has joined #openstack-infra14:17
*** tosky has joined #openstack-infra14:23
*** edmondsw has joined #openstack-infra14:27
*** edmondsw has quit IRC14:32
*** hongbin has joined #openstack-infra14:36
*** hongbin_ has joined #openstack-infra14:42
*** hongbin has quit IRC14:44
*** hongbin_ has quit IRC14:44
*** hongbin has joined #openstack-infra14:44
*** mriedem has quit IRC14:46
*** rhallisey has quit IRC14:47
*** fungi has quit IRC14:56
*** sree has joined #openstack-infra15:01
*** jascott1 has joined #openstack-infra15:03
*** sree has quit IRC15:05
*** jascott1 has quit IRC15:08
*** fungi has joined #openstack-infra15:11
*** tristanC has quit IRC15:18
*** tristanC has joined #openstack-infra15:19
*** ianychoi has joined #openstack-infra15:36
*** iyamahat has joined #openstack-infra15:40
*** dansmith has joined #openstack-infra15:40
*** spzala has joined #openstack-infra15:46
*** spzala has quit IRC15:47
*** sree has joined #openstack-infra16:01
*** jkilpatr has quit IRC16:03
*** sree has quit IRC16:06
*** dave-mccowan has joined #openstack-infra16:08
*** spzala has joined #openstack-infra16:13
*** edmondsw has joined #openstack-infra16:16
*** dave-mccowan has quit IRC16:17
*** dave-mccowan has joined #openstack-infra16:18
*** edmondsw has quit IRC16:20
*** dave-mcc_ has joined #openstack-infra16:22
*** dave-mccowan has quit IRC16:24
*** linkmark has quit IRC16:25
*** jkilpatr has joined #openstack-infra16:32
*** spzala has quit IRC16:33
dmsimardinfra-root: while the rebuilt ze09 is properly allowed in the graphite.o.o firewall (and there is traffic for that rule) grafana is flatlining ze09 which means there's a disconnect somewhere but I don't know where16:37
*** spzala has joined #openstack-infra16:41
*** dave-mcc_ has quit IRC16:43
*** spzala has quit IRC16:46
*** zzzeek has quit IRC16:50
*** sree has joined #openstack-infra16:50
*** zzzeek has joined #openstack-infra16:51
*** spzala has joined #openstack-infra16:53
*** zzzeek has quit IRC16:53
*** edmondsw has joined #openstack-infra16:54
*** zzzeek has joined #openstack-infra16:55
*** sree has quit IRC16:55
*** iyamahat has quit IRC16:56
*** spzala_ has joined #openstack-infra16:57
*** spzala has quit IRC16:58
*** edmondsw has quit IRC16:59
*** jkilpatr has quit IRC16:59
*** sree has joined #openstack-infra17:00
*** larainema has quit IRC17:03
*** jkilpatr has joined #openstack-infra17:08
*** slaweq has joined #openstack-infra17:12
*** sree has quit IRC17:13
*** slaweq has quit IRC17:17
*** Douhet has joined #openstack-infra17:19
*** slaweq has joined #openstack-infra17:30
*** slaweq has quit IRC17:35
*** bhavik1 has joined #openstack-infra17:37
*** sree has joined #openstack-infra17:37
*** bhavik1 has quit IRC17:40
*** sree has quit IRC17:42
*** mriedem has joined #openstack-infra17:45
*** sree has joined #openstack-infra17:47
*** sree has quit IRC17:51
corvusdmsimard: ze09 has a hostname of 'ze09'.  all the others have a hostname of 'ze0X.openstack.org'17:57
corvusdmsimard: you can see what's being reported by looking in graphite17:57
dmsimardAh, didn't think about checking there. So hostname issue then. I won't be able to look anytime soon so if anyone else wants to pick that up that's ok17:58
*** sree has joined #openstack-infra18:05
*** sree has quit IRC18:09
*** slaweq has joined #openstack-infra18:14
*** EmilienM_ has joined #openstack-infra18:22
*** hongbin has quit IRC18:23
*** yamamoto has quit IRC18:25
*** yamamoto has joined #openstack-infra18:27
*** yamamoto has quit IRC18:27
*** eumel8 has quit IRC18:28
*** EmilienM has quit IRC18:28
*** EmilienM_ is now known as EmilienM18:28
*** EmilienM has joined #openstack-infra18:28
*** persia__ is now known as persia18:32
*** sree has joined #openstack-infra18:37
*** jkilpatr has quit IRC18:39
*** sree has quit IRC18:42
*** slaweq has quit IRC18:42
*** edmondsw has joined #openstack-infra18:43
*** edmondsw has quit IRC18:47
*** jkilpatr has joined #openstack-infra18:57
*** e0ne has joined #openstack-infra19:00
*** jkilpatr has quit IRC19:01
*** e0ne_ has joined #openstack-infra19:01
*** jkilpatr has joined #openstack-infra19:02
*** e0ne has quit IRC19:05
*** jkilpatr has quit IRC19:08
*** makowals has joined #openstack-infra19:15
*** makowals has quit IRC19:16
*** Apoorva has joined #openstack-infra19:19
*** jbernard has quit IRC19:21
*** sdague has joined #openstack-infra19:21
*** Roamer`_ is now known as Roamer`19:23
*** jbernard has joined #openstack-infra19:26
*** Apoorva_ has joined #openstack-infra19:26
*** Apoorva has quit IRC19:26
*** jkilpatr has joined #openstack-infra19:27
*** yamamoto has joined #openstack-infra19:28
*** e0ne_ has quit IRC19:34
*** numans has quit IRC19:34
*** yamamoto has quit IRC19:37
*** numans has joined #openstack-infra19:39
*** wolverineav has joined #openstack-infra19:46
*** EmilienM has quit IRC20:10
*** EmilienM has joined #openstack-infra20:13
*** slaweq has joined #openstack-infra20:28
*** edmondsw has joined #openstack-infra20:31
*** edmondsw has quit IRC20:35
*** sree has joined #openstack-infra20:36
*** sree has quit IRC20:45
*** spzala_ has quit IRC20:59
*** jascott1 has joined #openstack-infra21:04
*** jascott1 has quit IRC21:08
*** markvoelker has quit IRC21:27
*** slaweq has quit IRC21:28
*** kzaitsev1pi has quit IRC21:28
*** kzaitsev_pi has joined #openstack-infra21:28
*** sree has joined #openstack-infra21:33
*** sree has quit IRC21:37
*** ijw has joined #openstack-infra21:45
*** xarses_ has quit IRC21:45
*** sree has joined #openstack-infra21:56
*** sree has quit IRC22:00
*** wolverineav has quit IRC22:07
*** wolverineav has joined #openstack-infra22:08
*** wolverineav has quit IRC22:08
*** wolverineav has joined #openstack-infra22:08
*** Apoorva_ has quit IRC22:09
*** wolverineav has quit IRC22:13
*** xarses_ has joined #openstack-infra22:17
*** edmondsw has joined #openstack-infra22:19
*** puiterwijk has quit IRC22:22
*** markvoelker has joined #openstack-infra22:22
*** edmondsw has quit IRC22:24
*** puiterwijk has joined #openstack-infra22:33
*** puiterwijk has joined #openstack-infra22:33
*** puiterwijk has joined #openstack-infra22:36
*** puiterwijk has joined #openstack-infra22:36
*** xarses_ has quit IRC22:39
*** igordc has quit IRC22:42
*** sree has joined #openstack-infra22:46
*** sree has quit IRC22:50
*** Apoorva has joined #openstack-infra23:09
*** slaweq has joined #openstack-infra23:10
*** sree has joined #openstack-infra23:23
*** Apoorva has quit IRC23:25
*** sree has quit IRC23:27
*** bobh has joined #openstack-infra23:28
*** slaweq has quit IRC23:33
*** slaweq has joined #openstack-infra23:34
*** dsariel has joined #openstack-infra23:35
*** slaweq has quit IRC23:38
*** sree has joined #openstack-infra23:45
*** sree has quit IRC23:50
*** jesusaur has quit IRC23:57
« Friday, 2018-01-05 Index Sunday, 2018-01-07 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!