Saturday, 2018-02-24

*** xarses_ has quit IRC00:01
clarkbof course now that I've done that it won't actually start :/00:03
sc`sounds like beer o'clock is rapidly approaching00:03
clarkbfirst problem was the pid file doesn't get claened up when you hit the service that hard00:05
clarkbbut after cleaning that up its still not starting and I'm not quire sure why00:06
clarkbthere is a bug in our init script where we rm * in the pid file dir which I think will delete the other service's pid files :/00:10
clarkbbut that isn't the problem here00:10
clarkbinfra-root did we change anything about the init script here? I'm not seeing where we pass the config file to zuul-scheduler00:14
*** yamamoto has joined #openstack-infra00:15
clarkbI guess maybe that is an optional argument00:16
clarkband of course it starts in the foreground00:16
clarkbugh00:16
mnaser:(00:21
clarkbI'm not seeing anything logged that would indicate why its exiting. The exit status is 000:21
mnaserclarkb: log files?00:21
clarkbthere is nothing in the zuul logs, there is nothing in journalctl00:21
*** yamamoto has quit IRC00:21
clarkbit runs fine if I su to zuul and run it in the foreground with the -d (no daemon flag)00:21
mnaserjournalctl -xe has nothing useful either?00:21
clarkbmnaser: nope00:21
sc`the answer is near the bottom of a beer bottle. i know, not helping.00:21
mnaserclarkb: so if i understand the issue, zuul wont start with init scripts00:22
mnaserbut will start fine in foreground00:22
clarkbjournalctl -xe http://paste.openstack.org/show/683502/00:22
clarkbmnaser: correct, I think it must be somethign to do with the daemonization code as a result00:22
clarkbeither in zuul itself or in the init script00:22
sc`is it still using sysvinit and not a unit?00:22
clarkbyes00:23
mnaserclarkb: well if zuul isn't starting up at all and not logging anything i think its a sysvinit thing00:23
sc`ooo... fun :(00:23
mnaserclarkb: i can write a quick systemd unit file if you have the init script?00:23
mnaserand we can give that a shot.. for now?00:23
*** tosky has quit IRC00:24
sc`that's what zuul-scheduler.service is, i think00:24
mnasersc`: im pretty sure thats an lsb compat service, not a real unit00:24
sc`yeah. not a real unit00:24
mnaserhence the 'Started LSB: Zuul scheduler.'00:24
clarkbyes00:24
clarkbI'd really rather understand why its broken rather than just replaceing what we think is broken00:25
mnaserclarkb: what if you call the init file directly00:25
mnaserset -x in the init script and /etc/init.d/zuul-scheduler start ?00:25
clarkbya I guess its worth a try00:26
sc`is the ExecStart incantation correct?00:26
mnaserclarkb: hey um00:26
mnaserhttps://github.com/openstack-infra/puppet-zuul/blob/master/files/zuul-scheduler.init and https://github.com/openstack-infra/puppet-zuul/blob/master/files/zuul-schedulerv3.init00:26
clarkbsc`: there is no execstart its an init script00:26
mnaseris it possible you should use zuul-schedulerv3?00:26
mnaserclarkb: https://github.com/openstack-infra/puppet-zuul/blob/eb2a5f81b1f71f6552ce9adabf9919d4bf728fb7/manifests/init.pp#L284-L292 nvm its not00:27
mnaserit looks like it gives a new exit codes that might be useful00:27
mnaserso echo $? might be of use too00:28
clarkbya exit codes are 000:28
clarkbI checked those with systemctl status00:28
*** eernst has quit IRC00:29
mnaserstart-stop-daemon --start --quiet --pidfile /var/run/zuul-scheduler/zuul-scheduler.pid -c zuul --exec /usr/local/bin/zuul-scheduler --test00:29
sc`clarkb: to the extent i've worked with systemd and sysvinit, i needed ExecStart for launching the init script itself00:30
*** ihrachys has quit IRC00:30
mnaserstart-stop-daemon --start --quiet --pidfile /var/run/zuul-scheduler/zuul-scheduler.pid -c zuul --exec /usr/local/bin/zuul-scheduler00:30
mnaserclarkb: ^ those are a recreation of what the init script runs.. maybe they might give useful output running them manually?00:30
clarkbsc`: I think that means you aren't using the sysv compat correctly00:30
*** ihrachys has joined #openstack-infra00:30
clarkbsc`: you should let systemd figure that out for you00:30
clarkbmnaser: I'm currently trying to understand the set -x output after running it, there is a lot of indirection in the systemd compat stuff00:30
mnaserclarkb: oh okay i didn't know you had some useful output from that, let me know if you need an extra pair of eyes00:31
clarkbit looks like it gets sent off to run `/bin/systemctl --no-pager start zuul-scheduler.service`00:31
*** lbragstad has quit IRC00:32
mnaserclarkb: i once had that very strange issue where it looked like a loop00:32
mnaserwhere init.d calls systemd and systemd calls initd00:33
clarkbya thats what it kind of appears is happening00:33
clarkbbut my set -x only goes as far as the first execution because i haven't daemon reloaded00:33
clarkbI guess I try that now00:33
mnasermaybe check if the system had any packages recently updated00:34
clarkbI'm beginning to wonder if maybe there was memory/swapping fallout in systemd?00:35
mnaserclarkb: if there isn't much other services running, maybe worth a reboot?  also, doing some reading, apparently calling /etc/init.d/foo redirects to systemd which creates a 'virtual' unit file00:36
mnaserexport _SYSTEMCTL_SKIP_REDIRECT=100:36
mnaserthat will remove the redirection if you want to do the set -x troubleshooting (src: https://www.turnkeylinux.org/blog/debugging-systemd-sysv-init-compat)00:36
clarkbexport that in the calling shell?00:37
mnaserclarkb: correct00:37
mnaserso .. export _SYSTEMCTL_SKIP_REDIRECT=1; /etc/init.d/zuul-scheduler start;00:37
mnaserthat should technically not go through systemd00:37
mnaseraccording to that article anyways00:37
mnasersome other articles references SYSTEMCTL_SKIP_REDIRECT=1 without the preceeding _, if it doesn't work00:38
mnasermost articles reference SYSTEMCTL_SKIP_REDIRECT=100:38
clarkbwell that got zuul to start :/00:38
clarkbI'm tempted to just leave it like this for now :)00:39
mnaserclarkb: its your call but it looks like something went haywire in the redirection00:39
clarkbya I wonder if systemd didn't take too kindly to swapping like that00:39
*** wolverineav has quit IRC00:39
clarkbwe might need to reboot to sort that out?00:40
mnasermaybe try doing a stop with the redirection off, then try doing a start again with systemd redirection back on00:40
*** wolverineav has joined #openstack-infra00:40
mnaseri was avoiding the reboot in case some other fun activities come up post reboot :)00:40
mnaserlike maybe "hey i haven't been fsck'd in a while!"00:40
clarkbya I don't want to reboot it either :)00:40
clarkbok here goes a clean stop00:40
mnaser🤞00:41
clarkbof course it probably won't actually stop until its finished configuring itself :(00:42
clarkbclean stop worked but will not start after a clean stop using systemctl00:44
clarkbI think what I'm going to do is just run the init script directly with redirection off00:44
clarkband we'll leave it at athat00:44
clarkbfrom what I can tell there are a hadnful of bugs in the init script regardless that we need to fix :(00:44
clarkbwe don't handle pid files correctly at all00:44
*** wolverineav has quit IRC00:44
pabelangerclarkb: I'm at YYZ for next few hours, anything I can help with00:45
clarkbpabelanger: zuul-scheduler init script is not working under systemd00:45
clarkbpabelanger: it runs and works fine if I disable the systemd magic unit redirection00:46
clarkband basically just run it directly00:46
clarkbI've also noticed that our init scripts don't delete the correct pid files00:46
clarkbpabelanger: I wonder if systemd is in a bad state after the ooming00:46
pabelangerclarkb: odd, it was working last time I reset, wednesday this week I think00:46
clarkbya I did one yesterday too00:47
clarkband I think mordred did one earlier today that was fine00:47
pabelangerOh, we OOM?00:47
clarkbwhich makes me wonder if its more the memory problem00:47
clarkbpabelanger: after mordreds restart we went straight back to having no memory and using all swap00:47
clarkbwhcih is how I ended up doing this zuul restart00:47
clarkbplan was to not requeue any changes to see if it used all memory on start up00:47
*** gyee has quit IRC00:47
pabelangerk00:47
clarkbbecause that would imply something in our merged configs was causing us to use all the memory00:47
clarkbbut then it has refused to start up00:48
clarkbpabelanger: we can try a reboot but really I just kidn of want to get to packing and sleep so maybe I'll just start it using the init script directly and we can pick this up later when we aren't traveling00:48
pabelangerclarkb: yah, reboot seems fine to me00:48
clarkbI mean00:49
clarkbfine for when I'm not trying to get off the computer :)00:49
*** eernst has joined #openstack-infra00:50
clarkbok I'm going to start zuul again now using the init script directly00:50
clarkband I'm going to leave it running00:50
pabelangeris it possible something with the zuul-merger are still loading old branches?00:50
pabelangerok00:51
clarkbI don't know that its actually a zuul issue00:51
clarkbthere is basically nothing logged by the startup process when using systemd00:51
clarkball it says is exit code 000:51
clarkbprocess exited00:51
clarkboh you mean for the memory use? that is possible I guess. Not sure how they deal with branch deletion00:52
pabelangeryah, I am not sure myself.00:53
pabelangerokay, I see cat jobs00:53
clarkbya I'm going to see now if web or fingergw will start00:53
clarkbfingergw and web both started00:54
clarkbthey use basically the same init script so I'm really confused00:54
clarkbok we are using ~1GB of memory after starting00:55
clarkbthats pretty stable. I guess I am going to enqueue gate now00:55
pabelanger2018-02-24 00:54:36,823 INFO zuul.Scheduler: Full reconfiguration complete00:56
clarkbok gate enqueued without major memory jump00:56
clarkbdoing check now00:56
pabelangerclarkb: did we enqueue last time too?00:56
pabelangeror did memory spike right on start00:57
clarkbpabelanger: yes I think so00:57
*** xarses has quit IRC00:57
clarkbpabelanger: we don't know if it was right at start or due to the enqueue00:57
clarkbpabelanger: which is why I wanted to start it in a controlled fashion and see if it started with massive emory use or if enqueing things did it00:57
clarkbso far neither has done it00:57
*** xarses has joined #openstack-infra00:57
clarkbup to 1.6GB or so now00:57
clarkbnow 2GB00:58
clarkbwhich is still quite low00:58
clarkbnow 3GB00:59
clarkband check is enqueued00:59
clarkbso I don't think it happened on start or due to the enqueue01:00
pabelangeryah, things look good currently01:00
*** wolverineav has joined #openstack-infra01:00
clarkb#status notice Zuul was restarted to workaround a memory issue. If your jobs are not running they may need to be rechecked01:01
openstackstatusclarkb: sending notice01:01
clarkband iirc I have to wait for that to finish before write a second status log01:01
-openstackstatus- NOTICE: Zuul was restarted to workaround a memory issue. If your jobs are not running they may need to be rechecked01:02
openstackstatusclarkb: finished sending notice01:04
*** wolverineav has quit IRC01:05
clarkb#status log The zuul-scheduler init script on zuul01.o.o appeared to stop working when attempting to start zuul after stopping it for running out of memory. Systemctl would report the process started successfully then exited 0. There were no zuul logs and no additional info in journalctl to further debug. Had to start zuul-scheduler using the init script directly after running `export01:05
openstackstatusclarkb: finished logging01:05
clarkb_SYSTEMCTL_SKIP_REDIRECT=1` to avoid the systemd sysv redirection.01:05
clarkbthats not gonna log properly01:05
clarkb#status log Had to start zuul-scheduler using the init script directly after running `export _SYSTEMCTL_SKIP_REDIRECT=1` to avoid the systemd sysv redirection. One theory is that systemd was unhappy after operating in the low memory environment. We need to get this working again as well as fix bugs iwth pid file handling in the init script. Might consider a server reboot.01:07
openstackstatusclarkb: finished logging01:07
clarkbstill holding steady at 3GB01:07
pabelangerconfirmed01:08
clarkbinfra-root as an FYI zuul-scheduler and the init system on zuul01 are currently unhappy with each other. I do not understand it. See above for details. We might consider a server reboot when it is convenient to see if it corrects the problem (assuming its systemd issue). We also have some bugs in our init scripts that we should fix around cleaning up pid files.01:08
clarkbpabelanger: ok I need to go finish packing I have an early flight tomorrow morning. Maybe you can keep an eye on it until you  board your plane?01:09
pabelangerclarkb: ack01:09
pabelangerhere for another 90mins01:09
clarkbI'm really stumped by this systemctl/systemd/sysv init script behavior01:10
clarkbits almost like it isn't running zuul at all01:10
clarkband just getting lost within the init system01:10
mnaserseems like a good time to add some systemd unit files01:10
mnaserand avoid the headache :P01:10
clarkbya that may be the best next step after ar eboot01:11
clarkbok thanks everyone for the help. I'm really afk now01:12
pabelangermnaser: yah, I've write some for testing in windmill, don't think it would be much effort to update puppet-zuul to use them01:13
mnaserpabelanger: for all the flack systemd gets, units are waaay easier and simpler than sysvinit01:14
*** yamamoto has joined #openstack-infra01:17
*** markvoelker has quit IRC01:19
*** markvoelker has joined #openstack-infra01:20
pabelangerk, zuul is still at 3GB01:20
*** eernst has quit IRC01:20
*** yamamoto has quit IRC01:22
*** xarses has quit IRC01:23
*** markvoelker has quit IRC01:25
*** eernst has joined #openstack-infra01:33
sc`from my perspective, it didn't seem like the start action was being fed to the init script. i alluded to this earlier, but was told i was talking about doing it wrong01:34
clarkbsc`: the way sys v compat is supposed to work in systemd is you put sys v init scripts in /etc/init.d/foo, then daemon-reload and you are done01:35
clarkbsc`: so you never write an excplicit execstart01:35
clarkbif you are writing units with execstarts that call sysv init scripts then youa ren't using the compat layer in systemd you are making your own01:35
sc`i try to avoid doing compatibility shims when i have to wade off in systemd01:36
clarkband fwiw the two other sys v init scripts which are basicaly identical to this one worked fine :(01:37
sc`happy friday :(01:37
pabelangerwell, zuul still running with about 3GB RAM, so we have that going for us01:38
*** salv-orlando has joined #openstack-infra01:41
*** gcb has joined #openstack-infra01:42
clarkbthinking about it more the next thing we should probably try is actually stopping the service via systemctl01:42
clarkboh except01:42
clarkbsystemd might be mad simply because it didn't start it in the first place?01:43
clarkbanyways try stopping, reboot, then look at unit files sounds good to me01:44
clarkbalso maybe the more forceful kill of the daemon is another reason for it being confused?01:44
*** salv-orlando has quit IRC01:45
pabelangeroh, it is possible systemd thought zuul-scheduler was still running, and service zuul-scheduler start just nooped01:47
pabelangerI've see that before01:48
pabelangerI had to service zuul-scheduler stop, delete pid, then service zuul-scheduler start01:48
pabelangerbefore01:48
clarkbpabelanger: ya I mean thats roughly what I did01:49
clarkbstop, it didn't stop after 10 minutes, so killed it, deleted pid file, then started it01:49
clarkband systemctl status seemed to show it knew about things exiting so I dunno01:50
pabelangerright, service zuul-scheduler status would have been interesting to see the output01:50
*** slaweq has joined #openstack-infra01:50
clarkbya I looked at that process started and exited with status code 001:50
pabelangerbefore the start attempt01:50
clarkboh before the start attempt01:50
pabelangerodd indeed01:50
pabelangerokay, gate change. Relocating, but zuul looks stable still01:52
*** slaweq has quit IRC01:54
openstackgerritSean McGinnis proposed openstack-infra/release-tools master: Minor cleanup of presentation  https://review.openstack.org/54769501:56
smcginnisI have a patch out there against bugdaystats for over a year now.02:03
smcginnisCould someone take a look and let me know if this is wanted or not? https://review.openstack.org/40916602:03
smcginnisTrying to clean up some of my outstanding patches.02:03
*** VW has joined #openstack-infra02:07
*** hongbin has joined #openstack-infra02:10
openstackgerritMerged openstack-infra/system-config master: Mirror tumbleweed for opensuse  https://review.openstack.org/54764602:10
*** xarses_ has joined #openstack-infra02:12
*** yamahata has quit IRC02:12
*** xarses_ has quit IRC02:14
*** ihrachys has quit IRC02:14
*** ihrachys has joined #openstack-infra02:15
*** xarses has joined #openstack-infra02:15
*** VW has quit IRC02:18
*** yamamoto has joined #openstack-infra02:19
*** markvoelker has joined #openstack-infra02:21
*** hongbin has quit IRC02:24
*** yamamoto has quit IRC02:24
*** hongbin has joined #openstack-infra02:24
openstackgerritMerged openstack-infra/release-tools master: Minor cleanup of presentation  https://review.openstack.org/54769502:33
*** salv-orlando has joined #openstack-infra02:41
*** olaph1 has joined #openstack-infra02:42
*** olaph has quit IRC02:43
*** armaan has quit IRC02:44
*** xarses_ has joined #openstack-infra02:46
*** salv-orlando has quit IRC02:46
*** hongbin has quit IRC02:48
*** markvoelker has quit IRC02:55
*** yamahata has joined #openstack-infra03:02
*** yuchangc_ has joined #openstack-infra03:03
*** masuberu has quit IRC03:11
*** claudiub|2 has quit IRC03:17
*** yamamoto has joined #openstack-infra03:21
*** yamamoto has quit IRC03:26
*** masuberu has joined #openstack-infra03:33
*** masuberu has quit IRC03:38
*** olaph has joined #openstack-infra03:39
*** eernst has quit IRC03:39
*** olaph1 has quit IRC03:40
*** armaan has joined #openstack-infra03:41
*** salv-orlando has joined #openstack-infra03:42
*** eernst has joined #openstack-infra03:46
*** salv-orlando has quit IRC03:47
*** markvoelker has joined #openstack-infra03:52
*** olaph1 has joined #openstack-infra03:55
*** olaph has quit IRC03:56
*** eernst has quit IRC04:05
*** baoli has quit IRC04:10
*** masuberu has joined #openstack-infra04:13
*** eharney has quit IRC04:14
*** lbragstad has joined #openstack-infra04:15
*** masber has joined #openstack-infra04:15
*** masuberu has quit IRC04:18
*** yamamoto has joined #openstack-infra04:21
*** Goneri has quit IRC04:24
*** markvoelker has quit IRC04:26
*** yamamoto has quit IRC04:27
*** auristor has quit IRC04:38
*** salv-orlando has joined #openstack-infra04:43
*** gongysh has joined #openstack-infra04:44
*** auristor has joined #openstack-infra04:44
*** salv-orlando has quit IRC04:47
*** olaph has joined #openstack-infra04:57
*** olaph1 has quit IRC04:58
*** gongysh has quit IRC04:59
*** lbragstad has quit IRC05:05
*** d0ugal has quit IRC05:08
*** aviau has quit IRC05:10
*** aviau has joined #openstack-infra05:10
*** markvoelker has joined #openstack-infra05:23
*** d0ugal has joined #openstack-infra05:23
*** yamamoto has joined #openstack-infra05:23
*** yamamoto has quit IRC05:29
*** gongysh has joined #openstack-infra05:32
*** salv-orlando has joined #openstack-infra05:44
*** salv-orlando has quit IRC05:48
*** slaweq has joined #openstack-infra05:52
*** markvoelker has quit IRC05:57
*** slaweq has quit IRC05:57
*** kjackal has quit IRC06:06
openstackgerritDong Ma proposed openstack-infra/project-config master: Add doc job for qinling  https://review.openstack.org/54771906:17
*** olaph has quit IRC06:17
*** olaph has joined #openstack-infra06:17
*** sree_ has joined #openstack-infra06:18
*** sree_ is now known as Guest1871606:18
*** Guest18716 has quit IRC06:23
*** sree_ has joined #openstack-infra06:24
*** sree_ is now known as Guest2243506:25
*** yamamoto has joined #openstack-infra06:25
*** Guest22435 has quit IRC06:26
*** yamamoto has quit IRC06:30
*** salv-orlando has joined #openstack-infra06:44
*** salv-orlando has quit IRC06:49
*** markvoelker has joined #openstack-infra06:54
*** lpetrut has joined #openstack-infra06:57
*** armaan has quit IRC07:03
*** armaan has joined #openstack-infra07:04
*** armaan has quit IRC07:05
*** armaan has joined #openstack-infra07:05
*** yamamoto has joined #openstack-infra07:27
*** markvoelker has quit IRC07:27
*** e0ne has joined #openstack-infra07:29
*** sree_ has joined #openstack-infra07:30
*** sree_ is now known as Guest3659607:30
openstackgerritDirk Mueller proposed openstack-infra/zuul-jobs master: Setup repositories for openSUSE Tumbleweed  https://review.openstack.org/54766007:32
*** yamamoto has quit IRC07:33
*** Guest36596 has quit IRC07:34
*** masber has quit IRC07:39
*** armaan has quit IRC07:44
*** e0ne has quit IRC07:45
*** armaan has joined #openstack-infra07:54
*** pgadiya has joined #openstack-infra07:55
*** armaan has quit IRC07:55
*** armaan has joined #openstack-infra07:56
*** olaph1 has joined #openstack-infra08:08
openstackgerritDirk Mueller proposed openstack-infra/zuul-jobs master: Setup repositories for openSUSE Tumbleweed  https://review.openstack.org/54766008:09
*** olaph has quit IRC08:09
*** lathiat has quit IRC08:13
*** armaan has quit IRC08:21
*** armaan has joined #openstack-infra08:21
*** markvoelker has joined #openstack-infra08:24
*** pgadiya has quit IRC08:27
*** armaan has quit IRC08:28
fricklermnaser: thanks for fixing up my devstack patch, nice coincidence with the gate issue08:28
*** armaan has joined #openstack-infra08:29
*** yamamoto has joined #openstack-infra08:29
*** yuchangc_ has quit IRC08:33
*** yamamoto has quit IRC08:34
*** yamahata has quit IRC08:35
AJaegerdmsimard: do you want to abandon https://review.openstack.org/547691 ?08:44
*** gongysh has quit IRC08:46
*** salv-orlando has joined #openstack-infra08:46
*** e0ne has joined #openstack-infra08:49
*** salv-orlando has quit IRC08:51
*** e0ne has quit IRC08:55
*** e0ne has joined #openstack-infra08:55
*** lathiat has joined #openstack-infra08:55
*** lpetrut has quit IRC08:57
*** markvoelker has quit IRC08:58
*** gongysh has joined #openstack-infra08:58
*** pcichy has quit IRC09:00
mnaserfrickler: no worries09:10
*** lpetrut has joined #openstack-infra09:28
*** yamamoto has joined #openstack-infra09:30
*** yamamoto has quit IRC09:35
*** armaan has quit IRC09:43
*** armaan has joined #openstack-infra09:43
*** salv-orlando has joined #openstack-infra09:47
*** yamamoto has joined #openstack-infra09:48
*** e0ne has quit IRC09:51
*** salv-orlando has quit IRC09:51
*** gongysh has quit IRC09:53
*** markvoelker has joined #openstack-infra09:54
*** gcb has quit IRC09:55
*** pgadiya has joined #openstack-infra09:56
*** armaan has quit IRC10:00
*** armaan has joined #openstack-infra10:00
*** pgadiya has quit IRC10:01
*** pcichy has joined #openstack-infra10:10
*** armaan has quit IRC10:11
*** armaan has joined #openstack-infra10:12
*** sree_ has joined #openstack-infra10:13
*** xarses_ has quit IRC10:13
*** sree_ is now known as Guest3053710:13
*** yamamoto has quit IRC10:17
*** Guest30537 has quit IRC10:17
*** agopi has quit IRC10:19
*** rossella_s has quit IRC10:24
*** rossella_s has joined #openstack-infra10:25
*** markvoelker has quit IRC10:28
*** claudiub|2 has joined #openstack-infra10:28
*** rossella_s has quit IRC10:40
*** e0ne has joined #openstack-infra10:41
*** armaan has quit IRC10:43
*** armaan has joined #openstack-infra10:43
*** slaweq has joined #openstack-infra10:44
*** salv-orlando has joined #openstack-infra10:48
*** slaweq has quit IRC10:48
ssbarneawhat should i do to assure that zuul runs on a change? I got a minor change on https://review.openstack.org/#/c/484905/ but it did not run, even after 14+ hours.10:51
*** salv-orlando has quit IRC10:52
odyssey4messbarnea it's very likely that got caught in-between a zuul restart, try commenting 'recheck' in the review to ask zuul to run tests again10:54
ssbarneaodyssey4me: thanks. just did.10:55
*** slaweq has joined #openstack-infra11:11
*** pcichy has quit IRC11:14
*** vivsoni has quit IRC11:14
*** slaweq has quit IRC11:16
*** yamamoto has joined #openstack-infra11:17
*** pcichy has joined #openstack-infra11:21
*** yamamoto has quit IRC11:23
*** markvoelker has joined #openstack-infra11:25
*** HeOS has quit IRC11:31
*** e0ne has quit IRC11:31
*** vivsoni has joined #openstack-infra11:42
*** slaweq has joined #openstack-infra11:43
*** rossella_s has joined #openstack-infra11:44
*** salv-orlando has joined #openstack-infra11:48
*** yuchangcai has joined #openstack-infra11:49
*** e0ne has joined #openstack-infra11:51
*** olaph has joined #openstack-infra11:53
*** salv-orlando has quit IRC11:53
*** olaph1 has quit IRC11:54
*** markvoelker has quit IRC11:58
*** numans has quit IRC11:59
*** vivsoni has quit IRC11:59
*** vivsoni has joined #openstack-infra12:00
*** rossella_s has quit IRC12:02
*** rossella_s has joined #openstack-infra12:03
*** numans has joined #openstack-infra12:07
*** e0ne has quit IRC12:07
openstackgerritMerged openstack-infra/storyboard master: handle missing filters arg when creating a worklist  https://review.openstack.org/54516912:08
*** yuchangcai has quit IRC12:11
*** yamamoto has joined #openstack-infra12:20
*** e0ne has joined #openstack-infra12:24
*** yamamoto has quit IRC12:24
*** e0ne has quit IRC12:24
*** masber has joined #openstack-infra12:37
*** salv-orlando has joined #openstack-infra12:49
*** lpetrut has quit IRC12:53
*** salv-orlando has quit IRC12:54
*** markvoelker has joined #openstack-infra12:56
*** rossella_s has quit IRC12:58
*** rossella_s has joined #openstack-infra13:00
*** tosky has joined #openstack-infra13:11
*** olaph1 has joined #openstack-infra13:12
*** danpawlik has joined #openstack-infra13:12
*** olaph has quit IRC13:12
*** yamamoto has joined #openstack-infra13:17
*** tosky has quit IRC13:18
*** slaweq has quit IRC13:20
*** danpawlik has quit IRC13:20
*** slaweq has joined #openstack-infra13:20
*** yamamoto has quit IRC13:21
*** slaweq has quit IRC13:25
*** markvoelker has quit IRC13:29
*** yuchangcai has joined #openstack-infra13:34
*** lbragstad has joined #openstack-infra13:35
*** danpawlik has joined #openstack-infra13:37
*** yuchangcai has quit IRC13:38
*** salv-orlando has joined #openstack-infra13:41
*** danpawlik has quit IRC13:41
*** olaph has joined #openstack-infra13:47
*** masber has quit IRC13:48
*** olaph1 has quit IRC13:49
*** armaan has quit IRC13:52
*** armaan has joined #openstack-infra13:53
*** yuchangcai has joined #openstack-infra13:56
*** pcichy has quit IRC13:57
*** dbecker has joined #openstack-infra13:59
*** rossella_s has quit IRC13:59
*** auristor1 has joined #openstack-infra13:59
*** auristor has quit IRC13:59
*** vivsoni_ has joined #openstack-infra14:01
*** dbecker_ has joined #openstack-infra14:04
*** thorre_se has joined #openstack-infra14:05
*** tonyb_ has joined #openstack-infra14:05
*** panda has joined #openstack-infra14:06
*** imacdonn has quit IRC14:07
*** sshnaidm_ has joined #openstack-infra14:07
*** imacdonn has joined #openstack-infra14:07
*** lbragstad has quit IRC14:07
*** MapspaM has joined #openstack-infra14:09
*** panda|sick has quit IRC14:10
*** dbecker has quit IRC14:10
*** vivsoni has quit IRC14:10
*** SpamapS has quit IRC14:10
*** tobberydberg has quit IRC14:10
*** sileht has quit IRC14:10
*** tonyb has quit IRC14:10
*** sshnaidm|off has quit IRC14:10
*** thorre has quit IRC14:10
*** thorre_se is now known as thorre14:10
*** tobberydberg has joined #openstack-infra14:10
*** bhujay has joined #openstack-infra14:10
*** sileht has joined #openstack-infra14:11
*** armaan has quit IRC14:12
*** johnthetubaguy_ has quit IRC14:12
*** rossella_s has joined #openstack-infra14:13
*** lbragstad has joined #openstack-infra14:14
*** johnthetubaguy has joined #openstack-infra14:15
*** caphrim007_ has quit IRC14:17
*** caphrim007 has joined #openstack-infra14:17
*** yamamoto has joined #openstack-infra14:18
*** johnthetubaguy has quit IRC14:22
*** bhujay has quit IRC14:22
*** tobberydberg has quit IRC14:22
*** johnthetubaguy has joined #openstack-infra14:23
*** rossella_s has quit IRC14:23
*** tobberydberg has joined #openstack-infra14:23
*** bhujay has joined #openstack-infra14:23
*** rossella_s has joined #openstack-infra14:23
*** yamamoto has quit IRC14:25
*** markvoelker has joined #openstack-infra14:26
*** pgadiya has joined #openstack-infra14:27
*** lbragstad has quit IRC14:28
*** markvoelker has quit IRC14:29
*** markvoelker has joined #openstack-infra14:29
*** rossella_s has quit IRC14:30
dmsimardAJaeger: abandoned, thanks14:30
*** rossella_s has joined #openstack-infra14:33
*** bhujay has quit IRC14:37
*** bhujay has joined #openstack-infra14:42
*** rossella_s has quit IRC14:42
*** rossella_s has joined #openstack-infra14:43
openstackgerritMerged openstack-infra/release-tools master: add missing lib files for presentation  https://review.openstack.org/54768914:44
*** danpawlik has joined #openstack-infra14:44
*** armaan has joined #openstack-infra14:45
*** markvoelker has quit IRC14:47
*** danpawlik has quit IRC14:48
*** salv-orlando has quit IRC14:49
*** rossella_ has joined #openstack-infra14:50
*** yamamoto has joined #openstack-infra14:55
*** yamamoto has quit IRC14:56
*** yamamoto has joined #openstack-infra14:56
*** rossella_ has quit IRC14:57
*** rossella_ has joined #openstack-infra14:58
*** rossella_s has quit IRC14:58
*** yuchangcai has quit IRC15:07
*** baoli has joined #openstack-infra15:09
*** lbragstad has joined #openstack-infra15:09
*** salv-orlando has joined #openstack-infra15:12
*** yuchangcai has joined #openstack-infra15:26
*** yuchangcai has quit IRC15:30
*** bhujay has quit IRC15:35
*** bhujay has joined #openstack-infra15:35
*** yamamoto_ has joined #openstack-infra15:39
*** bhujay has quit IRC15:41
*** jtomasek has joined #openstack-infra15:42
*** yamamoto has quit IRC15:43
*** yamahata has joined #openstack-infra15:46
*** jtomasek has quit IRC15:48
*** markvoelker has joined #openstack-infra15:57
*** danpawlik has joined #openstack-infra15:57
*** markvoelker has quit IRC15:58
*** olaph1 has joined #openstack-infra16:06
*** olaph has quit IRC16:06
*** olaph1 is now known as olaph16:07
*** VW has joined #openstack-infra16:07
*** armaan has quit IRC16:10
*** armaan has joined #openstack-infra16:10
*** bhujay has joined #openstack-infra16:13
*** danpawlik has quit IRC16:21
*** lbragstad has quit IRC16:21
ssbarneazaro_: hi! are you here?16:27
zaro_ssbarnea: hello16:28
ssbarneain fact @openstack-jjb was maybe better suited for this. i about reviewing and merging few python-jenkins improvements.16:29
ssbarnealike https://review.openstack.org/#/c/205637/16:30
*** baoli has quit IRC16:32
*** VW has quit IRC16:33
*** VW has joined #openstack-infra16:33
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776516:34
*** VW has quit IRC16:38
*** MapspaM is now known as SpamapS16:44
*** danpawlik has joined #openstack-infra16:47
*** slaweq has joined #openstack-infra16:48
*** armaan has quit IRC16:49
*** armaan has joined #openstack-infra16:49
*** slaweq has quit IRC16:50
*** slaweq has joined #openstack-infra16:50
*** jtomasek has joined #openstack-infra16:52
*** danpawlik has quit IRC16:53
*** olaph1 has joined #openstack-infra16:55
*** olaph has quit IRC16:55
*** danpawlik has joined #openstack-infra17:00
*** lpetrut has joined #openstack-infra17:02
*** danpawlik has quit IRC17:07
*** danpawlik has joined #openstack-infra17:17
*** bhujay has quit IRC17:18
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776517:19
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776517:25
*** armaan has quit IRC17:30
*** danpawlik has quit IRC17:30
*** armaan has joined #openstack-infra17:30
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776517:39
*** baoli has joined #openstack-infra17:42
*** baoli_ has joined #openstack-infra17:45
*** yamamoto_ has quit IRC17:46
*** baoli has quit IRC17:47
*** danpawlik has joined #openstack-infra17:53
*** agopi has joined #openstack-infra17:57
*** lpetrut has quit IRC17:58
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776518:00
*** baoli_ has quit IRC18:01
*** yamahata has quit IRC18:03
openstackgerritSorin Sbarnea proposed openstack/python-jenkins master: Skip kerberos tests on darwin with py3  https://review.openstack.org/54777618:04
*** baoli has joined #openstack-infra18:06
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776518:08
*** baoli has quit IRC18:11
*** baoli has joined #openstack-infra18:12
*** armaan has quit IRC18:12
*** armaan has joined #openstack-infra18:14
openstackgerritSorin Sbarnea proposed openstack/python-jenkins master: Skip kerberos tests on darwin with py3  https://review.openstack.org/54777618:14
*** baoli has quit IRC18:16
*** baoli has joined #openstack-infra18:20
*** pgadiya has quit IRC18:22
*** baoli has quit IRC18:25
*** baoli has joined #openstack-infra18:25
*** baoli has quit IRC18:29
*** pgadiya has joined #openstack-infra18:36
*** baoli has joined #openstack-infra18:37
*** pgadiya has quit IRC18:38
*** danpawlik has quit IRC18:41
*** baoli has quit IRC18:41
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776518:44
*** danpawlik has joined #openstack-infra18:45
*** yamamoto has joined #openstack-infra18:46
*** spligak has quit IRC18:48
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776518:50
*** danpawlik has quit IRC18:52
*** xarses_ has joined #openstack-infra18:54
*** andreww has joined #openstack-infra18:55
*** yamamoto has quit IRC18:55
*** xarses_ has quit IRC18:58
*** aviau has quit IRC19:00
*** aviau has joined #openstack-infra19:00
*** yuchangcai has joined #openstack-infra19:06
*** danpawlik has joined #openstack-infra19:09
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776519:09
mnaserAJaeger: taking advantage of an empty gate?  It’s so open!19:10
*** yuchangcai has quit IRC19:10
AJaegermnaser: trying to import a job - and then figuring out that it's broken since Zuul v3 conversion ;(19:10
AJaegerso, now the question is : How to fix properly? ;(19:11
AJaegerfun ;)19:11
AJaegermnaser: want to remove some broken jobs running? https://review.openstack.org/547362 is a small non-critical change...19:12
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776519:13
AJaegerargh, pre.yaml broken. Quotation mark fun ;(19:13
*** danpawlik has quit IRC19:13
* tobiash never saw the openstack zuul that empty19:20
fungimost of our developer community are on airplanes19:22
fungior sitting in airports (like i am)19:22
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776519:24
AJaegertobiash: take a screenshot ;)19:25
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776519:33
*** lpetrut has joined #openstack-infra19:35
*** andreww has quit IRC19:36
*** apetrich has joined #openstack-infra19:38
openstackgerritDirk Mueller proposed openstack-infra/zuul-jobs master: Adjust SuSEfirewall configuration for Tumbleweed  https://review.openstack.org/54777919:39
openstackgerritDirk Mueller proposed openstack-infra/openstack-zuul-jobs master: DNM  https://review.openstack.org/54684619:40
*** andreww has joined #openstack-infra19:44
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776519:45
*** xarses_ has joined #openstack-infra19:45
*** andreww has quit IRC19:48
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776519:52
*** jtomasek has quit IRC19:54
*** danpawlik has joined #openstack-infra19:58
*** danpawlik has quit IRC20:08
mnaseri just had a fun little encounter20:09
mnaserwent all the way through security and an hour before boarding i get called up to the gate and back into us border security...20:10
clarkbthey checked you twice?20:10
*** danpawlik has joined #openstack-infra20:10
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776520:10
mnaseryeah.  i went through normal borders all a-ok20:10
clarkb:/20:11
mnaserand then after that they called me out of the terminal back into the borders lol20:11
mnaser"someone happens to have the same name as you"20:11
mnaserfingerprints and all, but anyways .. it def passed some time lol20:11
clarkbwhat airport are you in? I just landed at jfk20:12
funginot a pleasant way to pass the time20:12
mnaserclarkb: i'm still in montreal (the us has a border 'crossing' point in the airport)20:13
clarkbah20:13
fungi(getting accosted by border security _or_ experiencing jfk intl.)20:13
mnaserclarkb: are you doing jfk -> dub ?20:13
mnaseri'm doing montreal -> bos -> jfk -> dub (fun time ha)20:13
clarkbmnaser: yup20:13
mnaserdelta 44? :p20:13
clarkbyup :)20:13
clarkbparty plane20:13
mnaseroh cool20:13
fungii should have brought us a disco ball20:14
mnaseri have 51 minutes to make my connection in jfk20:14
fungiouch20:14
fungiwe'll wave as you sprint by20:14
mnaseri've done 45 in chicago o'hare, i can do it :p20:14
mnaserim at 42A20:15
mnaserlooks like its a full flight20:15
fungi23c for me20:16
mnaserfungi: oou if things continue this way you might find an empty seat next to you20:16
mnasernice and spacious :p20:16
fungii'm rarely so lucky20:17
fungiit'll probably be a screaming infant with a head cold next to me20:17
mnaseraha20:18
*** danpawlik has quit IRC20:22
*** tosky has joined #openstack-infra20:25
clarkbI'm going to find the gate a whole 6 hours early then find lunch20:25
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776520:26
*** sshnaidm has joined #openstack-infra20:27
*** caphrim007_ has joined #openstack-infra20:27
*** sshnaidm_ has quit IRC20:27
*** caphrim007 has quit IRC20:30
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776520:41
mnaserhttps://review.openstack.org/#/q/topic:fix-nova-timeout20:42
mnaserif any infra's are core this is really useful to unblock stable/pike grenade gates20:42
*** oanson has quit IRC20:44
AJaegermnaser: clark and corvus are - see https://review.openstack.org/#/admin/groups/50,members20:44
mnaserwouldnt want to other clark on his quality time hanging around jfk20:45
mnaser:p20:45
clarkbI'm getting lunch I really need food20:46
clarkbI dont rhink I have +2 on stabke grenade though20:46
clarkbI seem to recall vhecking20:47
*** oanson has joined #openstack-infra20:47
openstackgerritMerged openstack-infra/project-config master: Add doc job for qinling  https://review.openstack.org/54771920:48
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776520:50
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: DNM: Import integration test  https://review.openstack.org/54776520:51
fungimnaser: #openstack-qa is a better bet for running down grenade reviewers if you haven't already tried20:51
*** lpetrut has quit IRC20:51
AJaegerfungi, clarkb, it's devstack, not grenade that the fix is for20:52
clarkbah20:53
clarkbonce I eat I'll find a quiet apot and take a look20:53
openstackgerritMerged openstack-infra/project-config master: have tag-releases skip closed series  https://review.openstack.org/54680120:54
* fungi isn't a devstack core reviewer either20:54
mnaserIt’s devstack affecting grenade :P20:57
openstackgerritAndreas Jaeger proposed openstack-dev/hacking master: Import integration test  https://review.openstack.org/54776521:02
AJaegera hack ;( for hacking ^21:03
*** xarses_ has quit IRC21:04
pabelangerI'm down at hotel bar myself now21:10
mnaserI booked too late to get space at the hotel for the event21:11
mnaserI’m a 10 minute walk out21:11
clarkbits gonna snow all week too :)21:12
*** gongysh has joined #openstack-infra21:15
mnaserclarkb: as someone who’s from Montreal, Dublin snow is nothing for me :P21:16
*** slaweq has quit IRC21:17
*** slaweq has joined #openstack-infra21:18
pabelangerit was nice today, I was walking around in t-shirt at time.21:18
clarkbya I'm sure you'll ve fine. We finally got some sniw in portland last week and now I get another week of snow"21:18
clarkbI'm sure it will be like portland snow all night melt by dinner rinse and repeat21:18
*** auristor1 has quit IRC21:19
mnaserThat was my safe assumption so  didn’t come with boots :p21:20
*** slaweq has quit IRC21:22
*** d0ugal has quit IRC21:22
*** apetrich has quit IRC21:22
*** auristor has joined #openstack-infra21:26
*** olaph has joined #openstack-infra21:33
*** olaph1 has quit IRC21:35
*** d0ugal has joined #openstack-infra21:40
clarkbmnaser: mriedem has a commwnt on th changes that I think is good but not -1'ing over it21:40
clarkbbasically make the doc trail a bit better21:40
*** vivsoni_ has quit IRC21:45
*** olaph1 has joined #openstack-infra21:58
*** olaph has quit IRC21:59
*** danpawlik has joined #openstack-infra22:19
*** olaph has joined #openstack-infra22:20
*** xarses_ has joined #openstack-infra22:22
*** olaph1 has quit IRC22:22
*** andreww has joined #openstack-infra22:23
*** xarses_ has quit IRC22:27
mnaserclarkb: cool thanks, I’ll throw a recheck too in there cause looks like this change didn’t get scheduled with the issues yesterday22:36
*** markvoelker has joined #openstack-infra22:40
*** markvoelker has quit IRC22:45
*** markvoelker has joined #openstack-infra22:46
*** markvoelker has quit IRC22:51
*** markvoelker has joined #openstack-infra22:54
*** olaph1 has joined #openstack-infra23:03
*** olaph has quit IRC23:04
*** tosky has quit IRC23:07
*** Goneri has joined #openstack-infra23:44

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!