| dansmith | did anyone else just lose gerrit? | 00:03 |
|---|---|---|
| harlowja | me | 00:03 |
| harlowja | can't seem to login :( | 00:03 |
| dansmith | I was mid comment | 00:03 |
| clarkb | dansmith: daily database backups happen at 0000UTC | 00:03 |
| dansmith | okay | 00:03 |
| clarkb | it should be fine in a minute or two | 00:03 |
| harlowja | k | 00:03 |
| dansmith | there it goes | 00:03 |
| anteaya | that ics file has my thunderbird in knots | 00:05 |
| *** jrmyjohn has joined #openstack-infra | 00:05 | |
| anteaya | also it isn't very useful as the same event is listed four times apiece | 00:05 |
| harlowja | hmmm, on getting gerrit redirected to login, mine still just waits at redirecting :( | 00:05 |
| clarkb | anteaya: lunch? | 00:05 |
| clarkb | there are apparently several different lunches | 00:05 |
| * harlowja just wants to login, lol | 00:05 | |
| anteaya | the print off the pdf and fold the paper, put it in a bag and get on the plane seems appealing | 00:05 |
| anteaya | clarkb: 4 coat checks | 00:06 |
| clarkb | harlowja: try again? | 00:06 |
| anteaya | 4 exams for every exam period | 00:06 |
| anteaya | 4 of everything | 00:06 |
| harlowja | clarkb ya, no workie | 00:06 |
| anteaya | my thunderbird is unhappy | 00:06 |
| harlowja | 'Redirecting ...' :( | 00:06 |
| harlowja | but i think its a lie, lol, its not really redirecting, lol | 00:06 |
| clarkb | harlowja: ok it owrks here | 00:06 |
| harlowja | odd | 00:06 |
| dansmith | seems to work here now | 00:06 |
| dansmith | but slow | 00:06 |
| *** sridhar_ram has quit IRC | 00:09 | |
| *** jrmyjohn has quit IRC | 00:10 | |
| dstanek | anteaya: i'll let her know if i do | 00:11 |
| anteaya | dstanek: awesome | 00:12 |
| *** xarses has quit IRC | 00:14 | |
| *** EricGonczer_ has joined #openstack-infra | 00:15 | |
| *** mylu has joined #openstack-infra | 00:15 | |
| *** ddieterly has joined #openstack-infra | 00:19 | |
| *** bpokorny_ has joined #openstack-infra | 00:25 | |
| *** bpokorn__ has joined #openstack-infra | 00:26 | |
| *** bpokorny_ has quit IRC | 00:26 | |
| *** daemontool has quit IRC | 00:27 | |
| *** mtanino_ has joined #openstack-infra | 00:28 | |
| *** bpokorny has quit IRC | 00:29 | |
| *** mtanino has quit IRC | 00:29 | |
| *** avarner has quit IRC | 00:30 | |
| *** sputnik13 has quit IRC | 00:31 | |
| *** njohnston has joined #openstack-infra | 00:32 | |
| *** Swami has quit IRC | 00:33 | |
| *** bpokorn__ has quit IRC | 00:42 | |
| *** xarses has joined #openstack-infra | 00:43 | |
| *** bpokorny has joined #openstack-infra | 00:43 | |
| *** tqtran has quit IRC | 00:44 | |
| *** gyee has quit IRC | 00:44 | |
| *** roxanagh_ has joined #openstack-infra | 00:45 | |
| *** Qiming has quit IRC | 00:47 | |
| *** roxanagh_ has quit IRC | 00:49 | |
| *** Sukhdev has quit IRC | 00:50 | |
| *** stewie925 has quit IRC | 00:51 | |
| openstackgerrit | Paul Van Eck proposed openstack-infra/system-config: Add database backups for refstack.openstack.org https://review.openstack.org/254426 | 00:53 |
| *** baoli has joined #openstack-infra | 00:57 | |
| *** apoorvad has quit IRC | 00:58 | |
| *** sdague has quit IRC | 00:59 | |
| *** bpokorny_ has joined #openstack-infra | 01:01 | |
| *** mtanino has joined #openstack-infra | 01:03 | |
| *** rossella_s has quit IRC | 01:03 | |
| *** rossella_s has joined #openstack-infra | 01:04 | |
| *** mtanino_ has quit IRC | 01:04 | |
| *** bpokorny has quit IRC | 01:05 | |
| *** bpokorny_ has quit IRC | 01:06 | |
| *** dimtruck is now known as zz_dimtruck | 01:06 | |
| *** inc0 has joined #openstack-infra | 01:07 | |
| *** zhurong has joined #openstack-infra | 01:08 | |
| *** ayoung has joined #openstack-infra | 01:12 | |
| openstackgerrit | Davanum Srinivas (dims) proposed openstack-dev/pbr: Remember the insertion order for pbr.json https://review.openstack.org/307405 | 01:13 |
| *** inc0 has quit IRC | 01:17 | |
| *** edmondsw has quit IRC | 01:21 | |
| *** pvaneck has quit IRC | 01:24 | |
| *** mylu has quit IRC | 01:29 | |
| *** bpokorny has joined #openstack-infra | 01:30 | |
| *** bpokorny has quit IRC | 01:33 | |
| *** mylu has joined #openstack-infra | 01:33 | |
| *** bpokorny has joined #openstack-infra | 01:33 | |
| *** EricGonczer_ has quit IRC | 01:34 | |
| *** MarkAtwood has joined #openstack-infra | 01:35 | |
| *** ddieterly is now known as ddieterly[away] | 01:38 | |
| *** Qiming has joined #openstack-infra | 01:45 | |
| *** nwkarsten has joined #openstack-infra | 01:46 | |
| *** nwkarsten has quit IRC | 01:50 | |
| *** ddieterly[away] is now known as ddieterly | 01:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Restore diskimage-builder-releases permissions to tag https://review.openstack.org/307531 | 01:51 |
| *** asettle has quit IRC | 01:52 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: WIP: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 01:53 |
| *** newjersey has quit IRC | 01:53 | |
| *** newjersey has joined #openstack-infra | 01:54 | |
| *** newjersey has quit IRC | 01:54 | |
| *** newjersey has joined #openstack-infra | 01:55 | |
| *** salv-orlando has joined #openstack-infra | 01:56 | |
| *** ddieterly is now known as ddieterly[away] | 01:57 | |
| *** newjersey has quit IRC | 01:59 | |
| *** newjersey has joined #openstack-infra | 02:00 | |
| *** ddieterly[away] is now known as ddieterly | 02:01 | |
| *** salv-orlando has quit IRC | 02:02 | |
| *** eliqiao_ is now known as eliqiao | 02:02 | |
| *** ddieterly is now known as ddieterly[away] | 02:02 | |
| *** IlyaG has joined #openstack-infra | 02:02 | |
| *** kengo_sakai has joined #openstack-infra | 02:03 | |
| *** newjersey has quit IRC | 02:08 | |
| *** woodster_ has quit IRC | 02:08 | |
| *** newjersey has joined #openstack-infra | 02:08 | |
| *** keedya has quit IRC | 02:10 | |
| *** rfolco has quit IRC | 02:12 | |
| *** rfolco has joined #openstack-infra | 02:13 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/project-config: Restore diskimage-builder-releases permissions to tag https://review.openstack.org/307531 | 02:13 |
| *** r-mibu has quit IRC | 02:14 | |
| *** EricGonczer_ has joined #openstack-infra | 02:15 | |
| *** MarkAtwood has quit IRC | 02:17 | |
| *** rfolco has quit IRC | 02:17 | |
| *** r-mibu has joined #openstack-infra | 02:19 | |
| dstufft | anyone know offhand one of the PyPI mirror URLs ya'll have? :] | 02:20 |
| clarkb | dstufft: mirror.dfw.rax.openstack.org | 02:21 |
| *** salv-orlando has joined #openstack-infra | 02:22 | |
| dstufft | clarkb: thanks! | 02:22 |
| dstufft | awesome, http://mirror.dfw.rax.openstack.org/pypi/packages/e0/fb/e94d8e95e97b0a51033e9c5afbc1d11941b8bda9e70222795c2e65ff04ce/dstufft.testpkg-2016.4.18.1.tar.gz#md5=d45b728fe3e3c79aa8ad734a1cd2e66e worked and got pulled in via bandersnatch just fine | 02:22 |
| *** newjersey has quit IRC | 02:24 | |
| *** salv-orlando has quit IRC | 02:26 | |
| *** EricGonczer_ has quit IRC | 02:27 | |
| *** tphummel has quit IRC | 02:27 | |
| *** asselin_ has joined #openstack-infra | 02:28 | |
| *** mtanino has quit IRC | 02:29 | |
| *** asselin__ has quit IRC | 02:29 | |
| openstackgerrit | Merged openstack-infra/storyboard: Fix updating automatic worklists with no filters https://review.openstack.org/307362 | 02:33 |
| *** gongysh has joined #openstack-infra | 02:34 | |
| *** yamahata has quit IRC | 02:40 | |
| *** Guest18995 is now known as Vivek | 02:45 | |
| *** Vivek has quit IRC | 02:45 | |
| *** Vivek has joined #openstack-infra | 02:45 | |
| *** esikachev has joined #openstack-infra | 02:48 | |
| *** zz_dimtruck is now known as dimtruck | 02:50 | |
| *** esikachev has quit IRC | 02:52 | |
| *** thorst_ has quit IRC | 02:52 | |
| *** thorst_ has joined #openstack-infra | 02:53 | |
| *** kzaitsev_mb has quit IRC | 02:53 | |
| *** thorst_ has quit IRC | 03:01 | |
| *** ddieterly[away] has quit IRC | 03:02 | |
| *** mylu has quit IRC | 03:06 | |
| jlvillal | tonyb, Thanks for the explanation on the start_date value :) | 03:14 |
| *** EricGonczer_ has joined #openstack-infra | 03:14 | |
| tonyb | jlvillal: np I know I'm kinda self-contradictory but I'm working (very slowly) on allowing yamil2ical to store the history like we used to have. | 03:15 |
| tonyb | jlvillal: and I think I just came up with a neat idea on how to maintain it long term | 03:16 |
| jlvillal | Seems like a good idea to be able to look at the history. At the moment is it basically the git log or is there something I'm missing? | 03:16 |
| *** tqtran has joined #openstack-infra | 03:17 | |
| jlvillal | tonyb, And I was playing some with the yaml2ical code this weekend. Proposed an '--irc-channels' option. | 03:17 |
| tonyb | jlvillal: I saw that. | 03:18 |
| jlvillal | My thought was to get to the point where when a conflict occurs it could maybe say, "hey channel 3 has an opening at that time" | 03:18 |
| tonyb | jlvillal: I like the idea and it removes some extra code form irc-meetings | 03:18 |
| jlvillal | Not there yet. But that was my sort of long term goal | 03:18 |
| * jlvillal has done the old, update irc channel, conflict, try another channel. conflict, try another channel, conflict. And finally figure out all channels were in-use. | 03:19 | |
| tonyb | jlvillal: we'll you *can* view the claendar in $ical viewer to make that go away but I agree it'd be nice to have some reporting in yaml2ical | 03:20 |
| *** amotoki has quit IRC | 03:20 | |
| jlvillal | Okay dinner time :) | 03:20 |
| jlvillal | Ciao | 03:20 |
| tonyb | jlvillal: cool cya | 03:20 |
| *** yamahata has joined #openstack-infra | 03:22 | |
| *** mylu has joined #openstack-infra | 03:22 | |
| *** EricGonczer_ has quit IRC | 03:22 | |
| *** salv-orlando has joined #openstack-infra | 03:23 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Go to story from card https://review.openstack.org/306454 | 03:25 |
| *** tphummel has joined #openstack-infra | 03:30 | |
| *** salv-orlando has quit IRC | 03:30 | |
| *** armax has quit IRC | 03:30 | |
| *** baoli has quit IRC | 03:32 | |
| *** uberjay has quit IRC | 03:34 | |
| *** Vivek has quit IRC | 03:34 | |
| *** tchaypo has quit IRC | 03:34 | |
| *** tchaypo has joined #openstack-infra | 03:35 | |
| *** Guest21642 has joined #openstack-infra | 03:35 | |
| *** roxanagh_ has joined #openstack-infra | 03:35 | |
| *** uberjay has joined #openstack-infra | 03:37 | |
| *** hrubi has quit IRC | 03:37 | |
| *** avarner has joined #openstack-infra | 03:37 | |
| *** morgabra has quit IRC | 03:39 | |
| *** charz_ has quit IRC | 03:39 | |
| *** morgabra has joined #openstack-infra | 03:40 | |
| *** persia has quit IRC | 03:40 | |
| *** links has joined #openstack-infra | 03:41 | |
| *** amotoki has joined #openstack-infra | 03:41 | |
| *** hrubi has joined #openstack-infra | 03:41 | |
| *** charz has joined #openstack-infra | 03:42 | |
| *** ianw has quit IRC | 03:43 | |
| *** Guest21642 has quit IRC | 03:44 | |
| *** Vivek_ has joined #openstack-infra | 03:45 | |
| stevemar | so, silly question, why are there only 300ish nodes available? according to http://status.openstack.org/zuul/ anyway ? | 03:46 |
| stevemar | did a miss an announcement or mailing list about stuff and things? | 03:46 |
| *** nwkarsten has joined #openstack-infra | 03:47 | |
| prometheanfire | is firefox really needed for non-debian? firefox [!platform:debian] | 03:47 |
| prometheanfire | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt | 03:47 |
| *** rlandy has quit IRC | 03:50 | |
| *** kzaitsev_mb has joined #openstack-infra | 03:50 | |
| *** persia has joined #openstack-infra | 03:50 | |
| eliqiao | hi, I see from zuul that jobs are queued for long time, is there any problem? | 03:50 |
| prometheanfire | fungi: do I have to map everything from http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/data/bindep-fallback.txt to gentoo? | 03:51 |
| prometheanfire | fungi: or is there an essential list | 03:51 |
| *** nwkarsten has quit IRC | 03:51 | |
| prometheanfire | eliqiao: a provider is down, forget the name | 03:52 |
| prometheanfire | it's the one from france iirc | 03:52 |
| fungi | prometheanfire: that is the list of stuff we used to install on bare-.* workers before we switched all jobs to run on minimal dib images, so it's now the list of packages installed at job runtime for most tox-based jobs unless the project declares a specific set of packages for their jobs to use | 03:54 |
| fungi | prometheanfire: in this case, probably the minimal set of packages necessary to run whatever basic things you expect to run on gentoo might suffice | 03:55 |
| prometheanfire | fungi: ok, my goal though is for openstack-ansible only, which likely doesn't need/want things like firefox | 03:55 |
| *** Nisha has joined #openstack-infra | 03:56 | |
| fungi | eliqiao: we had to take both ovh regions offline, as well as the rackspace region where we have the most available quota | 03:56 |
| prometheanfire | fungi: osic is down too? or rackspace public cloud? | 03:56 |
| fungi | prometheanfire: none of the providers are down themselves | 03:56 |
| *** avarner has quit IRC | 03:57 | |
| prometheanfire | just shifting jobs around? | 03:57 |
| fungi | we have an open ticket with rackspace because network connectivity between their ord and dfw regions is too unreliable for us to run jobs on now, and we have a ticket open with ovh because they seem to have deleted the project where our mirror servers are | 03:57 |
| prometheanfire | it's not too much extra work to do all the other mappings at least | 03:57 |
| prometheanfire | osic project? | 03:57 |
| *** _amrith_ is now known as amrith | 03:58 | |
| fungi | osic is separate, we don't have any significant issues there afaik but our quota in osic is pretty small by comparison to those others | 03:58 |
| prometheanfire | ah, k | 03:58 |
| *** thorst_ has joined #openstack-infra | 03:59 | |
| *** amotoki has quit IRC | 04:03 | |
| *** thorst_ has quit IRC | 04:06 | |
| *** gongysh has quit IRC | 04:08 | |
| clarkb | fungi prometheanfire in osic and bluevox we are leaking fips making both regions basically offline | 04:08 |
| clarkb | at least last I looked | 04:08 |
| *** ianw has joined #openstack-infra | 04:08 | |
| clarkb | not sure if mordred got that under control | 04:08 |
| clarkb | I believe we are effectively in 2 rax regions and internap right now | 04:10 |
| *** ayoung has quit IRC | 04:11 | |
| openstackgerrit | Catherine Diep proposed openstack-infra/irc-meetings: Re-schedule RefStack weekly team meeting from Monday to Tuesday. https://review.openstack.org/307450 | 04:11 |
| *** zhurong has quit IRC | 04:17 | |
| *** Vivek_ has quit IRC | 04:17 | |
| *** IlyaG has quit IRC | 04:18 | |
| prometheanfire | clarkb: you open a bug for that, if so I could probably bug someone at work to get attention on it | 04:18 |
| *** Vivek has joined #openstack-infra | 04:19 | |
| *** Vivek is now known as Guest40626 | 04:19 | |
| *** tqtran has quit IRC | 04:20 | |
| fungi | prometheanfire: not really. it's a nodepool regression | 04:21 |
| fungi | i suspect we need to roll nodepool back to a known-good commit and restart it | 04:21 |
| fungi | we really can't have 80% of our available resources offline like this | 04:21 |
| prometheanfire | ah | 04:22 |
| openstackgerrit | Matthew Thode proposed openstack-infra/project-config: add gentoo support to bindep-fallback.txt https://review.openstack.org/307553 | 04:22 |
| prometheanfire | fungi: ^ | 04:22 |
| prometheanfire | fungi: it's not perfect but something we can work from, also, without my bindep commit merged this WILL break bindep, the slashes in package names... | 04:23 |
| prometheanfire | just realized that now | 04:23 |
| * prometheanfire needs to remember how to do external deps in reviews | 04:23 | |
| jhesketh | infra-root: is anybody else having trouble loading logs.o.o? is somebody working on it? | 04:24 |
| *** abregman has quit IRC | 04:24 | |
| fungi | trouble loading? | 04:24 |
| prometheanfire | it's slow for me | 04:24 |
| jhesketh | fungi: my browser is stuck on 'waiting for logs.o.o'... I'm not sure if it'll timeout yet | 04:24 |
| prometheanfire | ya, think this is going to timeout | 04:25 |
| fungi | indeed | 04:25 |
| jhesketh | apache is running | 04:25 |
| fungi | load average is 0 | 04:25 |
| fungi | rrr | 04:25 |
| fungi | 40 | 04:25 |
| jhesketh | load looks normal | 04:25 |
| prometheanfire | 40 is a nice load | 04:26 |
| fungi | and climbing | 04:26 |
| jhesketh | sorry, I meant CPU | 04:26 |
| fungi | 100% iowait on one cpu | 04:26 |
| fungi | en, on 2 cpus actually | 04:26 |
| prometheanfire | ok, closing connection, sorry fot the load | 04:26 |
| prometheanfire | won't do it again | 04:27 |
| fungi | dmesg is full of io errors | 04:27 |
| *** ianw has quit IRC | 04:27 | |
| *** salv-orlando has joined #openstack-infra | 04:27 | |
| fungi | i wonder if rackspace is doing maintenance impacting access to our cinder volumes, or if they're having a failure | 04:27 |
| jhesketh | fungi: ah yes, I see cpu1 now | 04:28 |
| prometheanfire | fungi: iirc, no | 04:28 |
| prometheanfire | at least nothing I know of maint wise | 04:28 |
| jhesketh | hmm back to idle and logs.o.o working for me | 04:28 |
| prometheanfire | fungi: curiousity, does anyone in infra work for rackspace? | 04:28 |
| fungi | look like it started with [Tue Apr 19 04:25:22 2016] end_request: I/O error, dev xvdb, sector 898498984 | 04:28 |
| jhesketh | prometheanfire: I do | 04:29 |
| prometheanfire | jhesketh: castle? | 04:29 |
| jhesketh | but not for public cloud | 04:29 |
| jhesketh | prometheanfire: aus | 04:29 |
| prometheanfire | jhesketh: ah, going to summit? I'm in rpc going to summit | 04:29 |
| prometheanfire | rpc-o that is | 04:29 |
| *** Guest40626 has quit IRC | 04:29 | |
| jhesketh | prometheanfire: yep | 04:29 |
| *** kzaitsev_mb has quit IRC | 04:30 | |
| *** Vivek_ has joined #openstack-infra | 04:31 | |
| prometheanfire | see you at summit :P | 04:31 |
| *** yamamoto has quit IRC | 04:31 | |
| jhesketh | indeed :-) | 04:33 |
| jhesketh | looks like logs.o.o is behaving again... probably got scared when fungi logged in | 04:33 |
| *** amotoki has joined #openstack-infra | 04:33 | |
| *** ianw has joined #openstack-infra | 04:33 | |
| prometheanfire | that sounds right, he scares me too | 04:34 |
| *** Vivek_ has quit IRC | 04:35 | |
| *** vhosakot has quit IRC | 04:36 | |
| *** amrith is now known as _amrith_ | 04:38 | |
| *** asettle has joined #openstack-infra | 04:38 | |
| *** asselin_ has quit IRC | 04:41 | |
| AJaeger | jhesketh, ianw, could you put this one on your review list, please? https://review.openstack.org/#/c/307402/ It fixes a translation sync problem. | 04:43 |
| *** kengo_sa_ has joined #openstack-infra | 04:43 | |
| *** salv-orlando has quit IRC | 04:43 | |
| *** roxanagh_ has quit IRC | 04:44 | |
| prometheanfire | more devs need to do DIB reviews :( | 04:44 |
| jhesketh | AJaeger: lgtm | 04:44 |
| AJaeger | thanks, jhesketh. Now waiting that it finishes ;) | 04:47 |
| *** esikachev has joined #openstack-infra | 04:48 | |
| *** Vivek_ has joined #openstack-infra | 04:51 | |
| *** esikachev has quit IRC | 04:53 | |
| *** asettle has quit IRC | 04:54 | |
| *** asettle has joined #openstack-infra | 04:55 | |
| *** cody-somerville has quit IRC | 04:57 | |
| *** mylu has quit IRC | 05:02 | |
| *** cody-somerville has joined #openstack-infra | 05:03 | |
| *** flwang1 has quit IRC | 05:03 | |
| *** thorst_ has joined #openstack-infra | 05:04 | |
| *** mylu has joined #openstack-infra | 05:05 | |
| openstackgerrit | Merged openstack-infra/irc-meetings: switch HA meetings to daylight savings https://review.openstack.org/307002 | 05:08 |
| *** zhurong has joined #openstack-infra | 05:10 | |
| *** rcernin has joined #openstack-infra | 05:10 | |
| *** thorst_ has quit IRC | 05:12 | |
| *** eric_lopez has joined #openstack-infra | 05:12 | |
| *** elo has quit IRC | 05:13 | |
| *** asettle has quit IRC | 05:13 | |
| *** asettle has joined #openstack-infra | 05:13 | |
| *** yamamoto has joined #openstack-infra | 05:14 | |
| *** kzaitsev_mb has joined #openstack-infra | 05:15 | |
| *** chlong has quit IRC | 05:16 | |
| *** IlyaG has joined #openstack-infra | 05:18 | |
| *** kzaitsev_mb has quit IRC | 05:20 | |
| *** IlyaG has quit IRC | 05:23 | |
| *** ramishra has quit IRC | 05:28 | |
| *** ramishra has joined #openstack-infra | 05:28 | |
| *** mylu has quit IRC | 05:30 | |
| *** Sukhdev has joined #openstack-infra | 05:30 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Use infra-cloud east in nodepool https://review.openstack.org/283573 | 05:30 |
| *** tphummel has quit IRC | 05:30 | |
| yolanda | good morning | 05:30 |
| prometheanfire | moin | 05:31 |
| *** bpokorny has quit IRC | 05:35 | |
| *** asettle has quit IRC | 05:37 | |
| *** asettle has joined #openstack-infra | 05:38 | |
| *** oanson has joined #openstack-infra | 05:39 | |
| *** salv-orlando has joined #openstack-infra | 05:40 | |
| *** nadya has joined #openstack-infra | 05:41 | |
| *** masahito has joined #openstack-infra | 05:42 | |
| openstackgerrit | Merged openstack-infra/project-config: puppet/periodic: export ZUUL_REF https://review.openstack.org/307142 | 05:42 |
| *** cody-somerville has quit IRC | 05:43 | |
| eliqiao | hi yolanda. Morning. | 05:47 |
| *** asettle has quit IRC | 05:51 | |
| *** salv-orl_ has joined #openstack-infra | 05:52 | |
| *** salv-orl_ has quit IRC | 05:52 | |
| *** salv-orlando has quit IRC | 05:53 | |
| *** asettle has joined #openstack-infra | 05:53 | |
| tonyb | AJaeger: Any chnace you can backport the g-r portion of 305000 to mitaka and liberty (perhaps kilo too) | 05:53 |
| tonyb | AJaeger: if not I'll do it tomorrow | 05:53 |
| *** salv-orlando has joined #openstack-infra | 05:53 | |
| *** abregman has joined #openstack-infra | 05:54 | |
| *** e0ne has joined #openstack-infra | 05:54 | |
| *** salv-orlando has quit IRC | 05:54 | |
| *** salv-orlando has joined #openstack-infra | 05:55 | |
| *** infra-red has joined #openstack-infra | 06:02 | |
| *** kushal has quit IRC | 06:04 | |
| *** Nisha has quit IRC | 06:04 | |
| *** e0ne has quit IRC | 06:05 | |
| *** thorst_ has joined #openstack-infra | 06:09 | |
| *** e0ne has joined #openstack-infra | 06:10 | |
| *** Yarboa has joined #openstack-infra | 06:11 | |
| *** infra-re_ has joined #openstack-infra | 06:12 | |
| *** zhurong has quit IRC | 06:14 | |
| *** zhurong has joined #openstack-infra | 06:15 | |
| *** asettle has quit IRC | 06:15 | |
| *** infra-red has quit IRC | 06:15 | |
| *** asettle has joined #openstack-infra | 06:15 | |
| *** kzaitsev_mb has joined #openstack-infra | 06:16 | |
| *** thorst_ has quit IRC | 06:17 | |
| openstackgerrit | Merged openstack-infra/project-config: Translations: Fix sync of releasenotes https://review.openstack.org/307402 | 06:17 |
| *** infra-re_ has quit IRC | 06:17 | |
| *** zhurong has quit IRC | 06:18 | |
| *** oomichi has quit IRC | 06:19 | |
| *** kushal has joined #openstack-infra | 06:20 | |
| *** zhurong has joined #openstack-infra | 06:20 | |
| *** oomichi has joined #openstack-infra | 06:20 | |
| *** shardy has joined #openstack-infra | 06:20 | |
| *** kzaitsev_mb has quit IRC | 06:21 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints https://review.openstack.org/306798 | 06:21 |
| *** Sukhdev has quit IRC | 06:21 | |
| *** tojuvone__ has joined #openstack-infra | 06:22 | |
| *** roxanagh_ has joined #openstack-infra | 06:24 | |
| *** zeih has joined #openstack-infra | 06:25 | |
| *** tojuvone_ has quit IRC | 06:25 | |
| *** zhurong has quit IRC | 06:27 | |
| *** amotoki has quit IRC | 06:27 | |
| *** zhurong has joined #openstack-infra | 06:28 | |
| *** roxanagh_ has quit IRC | 06:29 | |
| *** Qiming has quit IRC | 06:29 | |
| *** e0ne has quit IRC | 06:30 | |
| *** e0ne has joined #openstack-infra | 06:31 | |
| *** nbloom has joined #openstack-infra | 06:33 | |
| *** infra-red has joined #openstack-infra | 06:33 | |
| *** jaosorior has joined #openstack-infra | 06:37 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 06:39 |
| *** scheuran has joined #openstack-infra | 06:42 | |
| *** dmellado_ is now known as dmellado | 06:44 | |
| *** masahito has quit IRC | 06:45 | |
| *** e0ne has quit IRC | 06:47 | |
| *** fawadkhaliq has joined #openstack-infra | 06:49 | |
| AJaeger | tonyb: I won't have time today | 06:52 |
| AJaeger | tonyb: If you beat me to it, it would be appreciated ;) | 06:52 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: [WIP] use virt-sysprep to cleanup images https://review.openstack.org/307588 | 06:52 |
| *** kzaitsev_mb has joined #openstack-infra | 06:53 | |
| *** infra-red has quit IRC | 06:55 | |
| tonyb | AJaeger: cool. I'll try to get it done tormorrow | 06:55 |
| *** daemontool has joined #openstack-infra | 06:55 | |
| *** nmagnezi has joined #openstack-infra | 06:55 | |
| *** kzaitsev_mb has quit IRC | 06:58 | |
| *** ramishra has quit IRC | 06:58 | |
| *** ramishra has joined #openstack-infra | 06:58 | |
| *** dizquierdo has joined #openstack-infra | 07:01 | |
| *** hichihara has quit IRC | 07:02 | |
| *** arxcruz has joined #openstack-infra | 07:03 | |
| *** binbincong has quit IRC | 07:04 | |
| *** amotoki has joined #openstack-infra | 07:06 | |
| *** arxcruz has quit IRC | 07:08 | |
| *** arxcruz has joined #openstack-infra | 07:09 | |
| *** dimtruck is now known as zz_dimtruck | 07:10 | |
| *** jed56 has joined #openstack-infra | 07:12 | |
| *** fawadkhaliq has quit IRC | 07:12 | |
| *** nadya_ has joined #openstack-infra | 07:12 | |
| *** nadya has quit IRC | 07:14 | |
| *** thorst_ has joined #openstack-infra | 07:15 | |
| *** ramishra has quit IRC | 07:15 | |
| *** binbincong has joined #openstack-infra | 07:16 | |
| *** ramishra has joined #openstack-infra | 07:16 | |
| *** pcaruana has joined #openstack-infra | 07:16 | |
| *** vincentll has joined #openstack-infra | 07:17 | |
| *** esikachev has joined #openstack-infra | 07:17 | |
| *** zeih has quit IRC | 07:17 | |
| *** zeih has joined #openstack-infra | 07:18 | |
| *** matrohon has joined #openstack-infra | 07:18 | |
| *** amotoki_ has joined #openstack-infra | 07:20 | |
| *** eliqiao_ has joined #openstack-infra | 07:20 | |
| *** thorst_ has quit IRC | 07:22 | |
| *** amotoki has quit IRC | 07:22 | |
| *** pcaruana has quit IRC | 07:23 | |
| *** eliqiao has quit IRC | 07:24 | |
| *** tesseract has joined #openstack-infra | 07:24 | |
| *** tesseract is now known as Guest22945 | 07:24 | |
| *** zeih has quit IRC | 07:24 | |
| *** Rockyg has quit IRC | 07:24 | |
| *** andymaier has joined #openstack-infra | 07:25 | |
| *** zeih has joined #openstack-infra | 07:25 | |
| *** roxanagh_ has joined #openstack-infra | 07:26 | |
| *** jaosorior has quit IRC | 07:27 | |
| *** zz_dimtruck is now known as dimtruck | 07:29 | |
| *** jlanoux has joined #openstack-infra | 07:29 | |
| *** mpaolino has joined #openstack-infra | 07:30 | |
| *** roxanagh_ has quit IRC | 07:31 | |
| *** infra-red has joined #openstack-infra | 07:31 | |
| *** flepied has quit IRC | 07:32 | |
| *** gongysh has joined #openstack-infra | 07:35 | |
| *** mikelk has joined #openstack-infra | 07:37 | |
| *** zeih has quit IRC | 07:39 | |
| *** dimtruck is now known as zz_dimtruck | 07:39 | |
| *** mariusv has quit IRC | 07:48 | |
| matrohon | AJaeger, hi | 07:48 |
| matrohon | AJaeger, could you please have a look at https://review.openstack.org/#/c/307150/? | 07:48 |
| *** ifarkas has joined #openstack-infra | 07:49 | |
| *** jaosorior has joined #openstack-infra | 07:55 | |
| *** gildub has joined #openstack-infra | 07:56 | |
| *** clarkb has quit IRC | 08:00 | |
| *** Qiming has joined #openstack-infra | 08:00 | |
| *** nadya_ has quit IRC | 08:00 | |
| openstackgerrit | Zhenguo Niu proposed openstack/requirements: Add networking-generic-switch to project list https://review.openstack.org/307619 | 08:00 |
| *** nbloom has quit IRC | 08:00 | |
| *** flepied has joined #openstack-infra | 08:02 | |
| *** eliqiao_ is now known as eliqiao | 08:04 | |
| openstackgerrit | David Pursehouse proposed openstack-infra/lodgeit: Show the currently running version on the about page https://review.openstack.org/249113 | 08:04 |
| *** clarkb has joined #openstack-infra | 08:06 | |
| *** amotoki_ has quit IRC | 08:06 | |
| *** e0ne has joined #openstack-infra | 08:13 | |
| *** ihrachys has joined #openstack-infra | 08:14 | |
| zigo | AJaeger: I got the first Debian package built on upstream infra! | 08:15 |
| zigo | IT'S WORKING !!! | 08:15 |
| zigo | \o/ | 08:15 |
| zigo | Next step: publish packages ... | 08:15 |
| *** masahito_ has joined #openstack-infra | 08:15 | |
| zigo | fungi: clarkb: mordred: ^ | 08:15 |
| *** kengo_sa_ has quit IRC | 08:16 | |
| *** jistr has joined #openstack-infra | 08:16 | |
| *** thorst_ has joined #openstack-infra | 08:20 | |
| *** gongysh has quit IRC | 08:24 | |
| *** nbloom has joined #openstack-infra | 08:26 | |
| *** ihrachys has quit IRC | 08:26 | |
| *** thorst_ has quit IRC | 08:27 | |
| openstackgerrit | Noam Bloom proposed openstack-infra/project-config: Vitrage - Zuul/Jenkins fix https://review.openstack.org/307634 | 08:29 |
| *** skolekonov_away is now known as skolekonov | 08:30 | |
| *** zz_dimtruck is now known as dimtruck | 08:30 | |
| openstackgerrit | Dong Ma proposed openstack-infra/subunit2sql: Add db api tests for subunit2sql (1/3) https://review.openstack.org/305030 | 08:31 |
| *** salv-orlando has quit IRC | 08:34 | |
| *** zeih has joined #openstack-infra | 08:39 | |
| *** dimtruck is now known as zz_dimtruck | 08:40 | |
| *** asettle has quit IRC | 08:41 | |
| *** asettle has joined #openstack-infra | 08:41 | |
| *** tojuvone__ is now known as tojuvone | 08:44 | |
| *** Adri2000 has quit IRC | 08:47 | |
| *** Adri2000 has joined #openstack-infra | 08:47 | |
| *** derekh has joined #openstack-infra | 08:49 | |
| *** asettle has quit IRC | 08:51 | |
| *** sshnaidm has quit IRC | 08:52 | |
| *** asettle has joined #openstack-infra | 08:52 | |
| openstackgerrit | Steven Hardy proposed openstack-infra/tripleo-ci: Fix REPO_PREFIX for stable releases in tripleo.sh https://review.openstack.org/307644 | 08:53 |
| *** jordanP has joined #openstack-infra | 08:54 | |
| *** electrofelix has joined #openstack-infra | 08:56 | |
| openstackgerrit | Jiri Stransky proposed openstack-infra/tripleo-ci: Don't use caching in stable jobs https://review.openstack.org/307647 | 08:57 |
| *** dingyichen has quit IRC | 08:58 | |
| *** Nikolay_St has joined #openstack-infra | 08:59 | |
| Nikolay_St | hi, infra team | 08:59 |
| *** salv-orlando has joined #openstack-infra | 08:59 | |
| *** salv-orlando has quit IRC | 09:00 | |
| *** kzaitsev_mb has joined #openstack-infra | 09:01 | |
| *** salv-orlando has joined #openstack-infra | 09:01 | |
| *** Qiming has quit IRC | 09:04 | |
| *** ociuhandu has joined #openstack-infra | 09:05 | |
| Nikolay_St | we have 545 commits in a check queue at http://status.openstack.org/zuul/ | 09:05 |
| Nikolay_St | is known problem? and is it a problem?) | 09:05 |
| yolanda | hi Nikolay_St , yes, we lost one of our providers for nodepool, so we are limited in number of vms we can spin | 09:06 |
| Nikolay_St | yolanda: thank for clarification | 09:07 |
| *** jaosorior has quit IRC | 09:07 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Hide the postci logs https://review.openstack.org/296375 | 09:07 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Revert split into 2 shell scripts https://review.openstack.org/307654 | 09:07 |
| openstackgerrit | sandhya proposed openstack/diskimage-builder: Procedure to build UEFI based images https://review.openstack.org/307655 | 09:07 |
| *** oanson has quit IRC | 09:10 | |
| *** esikachev has quit IRC | 09:12 | |
| *** lucasagomes_ has joined #openstack-infra | 09:13 | |
| *** lucasagomes has quit IRC | 09:13 | |
| *** roxanagh_ has joined #openstack-infra | 09:13 | |
| *** esikachev has joined #openstack-infra | 09:14 | |
| *** kengo_sa_ has joined #openstack-infra | 09:16 | |
| *** sambetts|afk is now known as sambetts | 09:17 | |
| *** amotoki has joined #openstack-infra | 09:17 | |
| *** roxanagh_ has quit IRC | 09:18 | |
| *** lucasagomes_ has quit IRC | 09:18 | |
| *** lucasagomes has joined #openstack-infra | 09:19 | |
| evgenyl | Hi, is there a way to delete git tag in the repository? | 09:19 |
| *** masahito_ has quit IRC | 09:21 | |
| *** hogepodge has joined #openstack-infra | 09:23 | |
| *** imcsk8 is now known as imcsk8|zZz | 09:25 | |
| *** thorst_ has joined #openstack-infra | 09:25 | |
| *** kengo_sa_ has quit IRC | 09:25 | |
| openstackgerrit | Niall Bunting proposed openstack-infra/project-config: Enable g-api and n-api for glance_store functional https://review.openstack.org/307670 | 09:27 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Add Ryan Petello as astara meeting chair https://review.openstack.org/307460 | 09:28 |
| *** vponomaryov1 has joined #openstack-infra | 09:30 | |
| *** vgridnev has joined #openstack-infra | 09:31 | |
| *** zz_dimtruck is now known as dimtruck | 09:31 | |
| *** thorst_ has quit IRC | 09:31 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 09:31 |
| *** dtantsur|afk is now known as dtantsur | 09:40 | |
| *** dimtruck is now known as zz_dimtruck | 09:41 | |
| *** vhejral has joined #openstack-infra | 09:43 | |
| *** amotoki has quit IRC | 09:43 | |
| *** vhejral has quit IRC | 09:44 | |
| *** nadya has joined #openstack-infra | 09:44 | |
| *** zhurong has quit IRC | 09:46 | |
| zigo | mordred: When you're up again, could you add this etherpad to the topic for packaging-deb please? https://etherpad.openstack.org/p/newton-deb-packaging | 09:46 |
| zigo | I'm really not sure how I could do it myself, probably I can't... | 09:47 |
| *** tosky has joined #openstack-infra | 09:47 | |
| zigo | (I mean, for the content at https://www.openstack.org/summit/austin-2016/summit-schedule/events/9380?goback=1) | 09:47 |
| *** zhurong has joined #openstack-infra | 09:47 | |
| *** oberezovskyi has joined #openstack-infra | 09:47 | |
| oberezovskyi | Hi! I've got problems with logging in into https://wiki.openstack.org/wiki/Main_Page . Does any one else can't login? (I'm getting HTTP 500 error) | 09:48 |
| vponomaryov1 | oberezovskyi: works ok | 09:48 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Explicitly set num_engine_workers for mitaka and below https://review.openstack.org/306296 | 09:49 |
| *** zhurong has quit IRC | 09:49 | |
| *** nwkarsten has joined #openstack-infra | 09:52 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 09:52 |
| *** IlyaG has joined #openstack-infra | 09:55 | |
| *** asettle has quit IRC | 09:56 | |
| *** nwkarsten has quit IRC | 09:57 | |
| *** asettle has joined #openstack-infra | 09:57 | |
| *** gildub has quit IRC | 09:59 | |
| *** IlyaG has quit IRC | 09:59 | |
| *** amotoki has joined #openstack-infra | 10:00 | |
| openstackgerrit | Merged openstack-infra/shade: Fix string formatting https://review.openstack.org/307454 | 10:01 |
| *** mattt has joined #openstack-infra | 10:01 | |
| *** sshnaidm has joined #openstack-infra | 10:04 | |
| *** nadya has quit IRC | 10:06 | |
| *** kushal has quit IRC | 10:07 | |
| *** _degorenko|afk is now known as degorenko | 10:08 | |
| *** sdague has joined #openstack-infra | 10:08 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Remove pointless variable creation https://review.openstack.org/306401 | 10:08 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient: Show a modal when editing worklists in a board https://review.openstack.org/306400 | 10:08 |
| *** claudiub has joined #openstack-infra | 10:10 | |
| *** amotoki has quit IRC | 10:14 | |
| *** yamahata has quit IRC | 10:20 | |
| *** nadya has joined #openstack-infra | 10:24 | |
| *** electrofelix has quit IRC | 10:24 | |
| *** Qiming has joined #openstack-infra | 10:25 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: WIP: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 10:26 |
| *** nbloom has quit IRC | 10:28 | |
| *** thorst_ has joined #openstack-infra | 10:29 | |
| *** jlanoux_ has joined #openstack-infra | 10:30 | |
| *** jlanoux has quit IRC | 10:31 | |
| *** zz_dimtruck is now known as dimtruck | 10:32 | |
| openstackgerrit | Merged openstack-infra/yaml2ical: Remove intermediate datetime.datetime().date() usage https://review.openstack.org/307355 | 10:32 |
| *** electrofelix has joined #openstack-infra | 10:34 | |
| *** vhejral has joined #openstack-infra | 10:34 | |
| *** asselin_ has joined #openstack-infra | 10:37 | |
| *** Qiming has quit IRC | 10:37 | |
| *** thorst_ has quit IRC | 10:37 | |
| *** markvoelker has joined #openstack-infra | 10:37 | |
| *** jlanoux_ has quit IRC | 10:38 | |
| *** kzaitsev_mb has quit IRC | 10:38 | |
| *** jlanoux has joined #openstack-infra | 10:39 | |
| *** dimtruck is now known as zz_dimtruck | 10:41 | |
| *** markvoelker has quit IRC | 10:42 | |
| *** jaosorior has joined #openstack-infra | 10:42 | |
| *** nbloom has joined #openstack-infra | 10:42 | |
| *** asselin_ has quit IRC | 10:43 | |
| *** kushal has joined #openstack-infra | 10:48 | |
| *** kzaitsev_mb has joined #openstack-infra | 10:49 | |
| *** |-paul-| has joined #openstack-infra | 10:50 | |
| *** tpsilva has joined #openstack-infra | 10:52 | |
| *** kushal has quit IRC | 10:55 | |
| *** kushal has joined #openstack-infra | 10:56 | |
| *** rfolco has joined #openstack-infra | 11:00 | |
| *** rfolco has quit IRC | 11:00 | |
| *** rhallisey has joined #openstack-infra | 11:01 | |
| *** ihrachys has joined #openstack-infra | 11:01 | |
| *** roxanagh_ has joined #openstack-infra | 11:01 | |
| *** shardy is now known as shardy_lunch | 11:03 | |
| *** bdemers_ has quit IRC | 11:04 | |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Properly populate Worklist.items in automatic worklists https://review.openstack.org/307712 | 11:06 |
| *** roxanagh_ has quit IRC | 11:06 | |
| sshnaidm | now when jenkins is unavailable, how to view the last statuses of particular job? | 11:10 |
| *** gildub has joined #openstack-infra | 11:10 | |
| *** gongysh has joined #openstack-infra | 11:11 | |
| *** julim has joined #openstack-infra | 11:12 | |
| *** vhejral has quit IRC | 11:14 | |
| *** eliqiao_ has joined #openstack-infra | 11:15 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Fix dashboard template to show the right information https://review.openstack.org/307448 | 11:16 |
| *** eliqiao has quit IRC | 11:19 | |
| *** esikachev has quit IRC | 11:19 | |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: WIP: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 11:19 |
| *** kengo_sakai has quit IRC | 11:24 | |
| therve | Keystone liberty branch seems broken ? | 11:25 |
| therve | Fails with pkg_resources.ContextualVersionConflict: (pbr 0.11.1 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('pbr>=1.8'), set(['testresources'])) | 11:25 |
| therve | Ah someone just sent an email about that | 11:26 |
| *** shardy_lunch is now known as shardy | 11:27 | |
| *** gongysh has quit IRC | 11:28 | |
| *** weshay has quit IRC | 11:28 | |
| *** yamamoto has quit IRC | 11:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 11:30 |
| *** doug-fish has joined #openstack-infra | 11:30 | |
| *** ldnunes has joined #openstack-infra | 11:32 | |
| *** zz_dimtruck is now known as dimtruck | 11:32 | |
| *** salv-orlando has quit IRC | 11:34 | |
| ryanpetrello | anybody in infra know what causes http://eavesdrop.openstack.org/#Astara_Meeting to be updated? | 11:36 |
| ryanpetrello | https://review.openstack.org/#/c/307460/ merged and I'm wondering if it doesn't affect that page, or if there's something else that I need to do | 11:37 |
| *** thorst_ has joined #openstack-infra | 11:37 | |
| *** mtanino has joined #openstack-infra | 11:38 | |
| *** markvoelker has joined #openstack-infra | 11:38 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 11:39 |
| *** ddieterly has joined #openstack-infra | 11:39 | |
| *** dimtruck is now known as zz_dimtruck | 11:42 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Don't allow users to interact with cards in automatic worklists https://review.openstack.org/306497 | 11:42 |
| *** markvoelker has quit IRC | 11:42 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: grafana: added periodic oslo master jobs to unit test 'gate' dashboard https://review.openstack.org/307511 | 11:44 |
| *** lucasagomes is now known as lucas-hungry | 11:47 | |
| *** dtardivel has joined #openstack-infra | 11:47 | |
| *** salv-orlando has joined #openstack-infra | 11:47 | |
| *** gordc has joined #openstack-infra | 11:48 | |
| therve | ryanpetrello, Looks like an issue in the post job? It's supposed to be done right there I believe | 11:49 |
| dims | therve : see backports in keystone branches reviews. | 11:49 |
| therve | dims, Which ones? | 11:49 |
| dims | https://review.openstack.org/#/c/307318/ and https://review.openstack.org/#/c/307335/ | 11:49 |
| therve | dims, Yeah but it looks like kilo is uninstallable? | 11:50 |
| therve | Which would explain why https://review.openstack.org/#/c/307318/ fails | 11:50 |
| dims | therve : i haven't looked at kilo, we should probably ask someone on #openstack-stable or keystone channels | 11:51 |
| *** Goneri has quit IRC | 11:51 | |
| therve | OK | 11:51 |
| dims | therve : at the moment, we are trying to cleanup babel references in oslo.* and get a release out soon-ish. so should be back on its feet after that | 11:52 |
| therve | dims, Well, not if we can't land the patches that fix it :) | 11:52 |
| dims | therve : +1, just saying what i was working on :) | 11:52 |
| *** nwkarsten has joined #openstack-infra | 11:53 | |
| *** Guest22945 is now known as tesseract | 11:54 | |
| *** tesseract is now known as Guest73397 | 11:54 | |
| *** rodrigods has quit IRC | 11:54 | |
| *** rodrigods has joined #openstack-infra | 11:55 | |
| *** dprince has joined #openstack-infra | 11:55 | |
| *** nwkarsten has quit IRC | 11:58 | |
| *** ddieterly is now known as ddieterly[away] | 11:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 11:59 |
| *** matt-borland has joined #openstack-infra | 12:00 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Make the Add Lane button show the new worklist modal https://review.openstack.org/306399 | 12:01 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Upload the overcloud images to cache https://review.openstack.org/306499 | 12:01 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: WIP: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 12:03 |
| *** weshay has joined #openstack-infra | 12:03 | |
| *** gildub has quit IRC | 12:06 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 12:07 |
| *** _amrith_ is now known as amrith | 12:07 | |
| *** yamamoto has joined #openstack-infra | 12:09 | |
| *** Qiming has joined #openstack-infra | 12:11 | |
| *** zhurong has joined #openstack-infra | 12:12 | |
| *** trown|outtypewww is now known as trown | 12:12 | |
| *** newjersey has joined #openstack-infra | 12:12 | |
| amrith | AJaeger, I fixed up https://review.openstack.org/#/c/307161/2. after speaking with peterstac just now I figured that it is best to leave the example to where it was (it is in the paragraph that talks about the depends anyway) and just indicate that the change is to add doc-publisher-site. | 12:13 |
| amrith | he's +1'ed it. | 12:13 |
| amrith | AJaeger ^^ | 12:13 |
| *** Qiming_ has joined #openstack-infra | 12:14 | |
| mordred | infra-root: ok. I think I may actually have the fip leak issue sorted | 12:15 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add final success/failure message to ping test https://review.openstack.org/304864 | 12:16 |
| *** Qiming has quit IRC | 12:16 | |
| *** baoli has joined #openstack-infra | 12:16 | |
| *** aysyd has joined #openstack-infra | 12:17 | |
| *** mtreinish has quit IRC | 12:18 | |
| *** baoli_ has joined #openstack-infra | 12:18 | |
| *** markvoelker has joined #openstack-infra | 12:19 | |
| *** oanson has joined #openstack-infra | 12:19 | |
| openstackgerrit | Merged openstack/requirements: Require stevedore>=1.9.0 https://review.openstack.org/307475 | 12:19 |
| *** slogan has quit IRC | 12:19 | |
| *** jheroux has joined #openstack-infra | 12:20 | |
| amrith | ttx, ^^ re: https://review.openstack.org/#/c/307161/2 | 12:20 |
| *** mtreinish has joined #openstack-infra | 12:21 | |
| *** baoli has quit IRC | 12:21 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** pblaho has quit IRC | 12:24 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Cleanup leaked floating ips on startup https://review.openstack.org/307755 | 12:24 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Avoid false positives when filtering projects by subscriber https://review.openstack.org/307756 | 12:24 |
| *** pradk_ has joined #openstack-infra | 12:24 | |
| *** ddieterly[away] has quit IRC | 12:30 | |
| *** pblaho has joined #openstack-infra | 12:31 | |
| *** lezbar has quit IRC | 12:31 | |
| *** lezbar has joined #openstack-infra | 12:32 | |
| *** zz_dimtruck is now known as dimtruck | 12:33 | |
| *** yamamoto has joined #openstack-infra | 12:34 | |
| ihrachys | mordred: is the fip leak issue showing as an empty IP for nodes allocated? as in http://logs.openstack.org/01/306901/3/check/gate-grenade-dsvm-neutron-dvr-multinode/aaa16c9/console.html#_2016-04-18_16_42_35_370 ? | 12:36 |
| sdague | mordred et al, is there a status update about why we're down to 250 nodes? | 12:37 |
| *** yamamoto has quit IRC | 12:39 | |
| yolanda | infra-root, so ovh mirrors are back to life now | 12:42 |
| *** zeih has quit IRC | 12:42 | |
| pabelanger | yolanda: \o/ | 12:42 |
| pabelanger | also, morning | 12:42 |
| yolanda | good morning | 12:42 |
| yolanda | we can restore the ovh provider then | 12:42 |
| mordred | yolanda: woot! | 12:42 |
| *** dimtruck is now known as zz_dimtruck | 12:43 | |
| mordred | ihrachys: no - it's the "sometimes floating ips do not delete even thought the delete API call is synchronous and it does not return with an error | 12:43 |
| mordred | sdague: it's multuple reasons that have all happened at the same time, lucky us | 12:43 |
| amrith | sdague, would you please review https://review.openstack.org/#/c/306716/; I'd like to start getting the trove team using that dashboard. I believe that you are the maintainer on that repository. | 12:43 |
| *** yamamoto has joined #openstack-infra | 12:44 | |
| *** kengo_sakai has joined #openstack-infra | 12:44 | |
| *** yamamoto has quit IRC | 12:45 | |
| sdague | amrith: done | 12:45 |
| amrith | thanks sdague | 12:45 |
| amrith | sdague, very nifty tool, would love it if I could sort the results a different way but I guess that's not something that gerrit-dash-creator can influence. | 12:46 |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Revert "Temporarily disable ovh-bhs provider" https://review.openstack.org/307791 | 12:46 |
| ihrachys | mordred: I am trying to understand why the failure rate for neutron multinode jobs is high lately, and that's what I see in all failure logs - nodepool files are empty, so no IPs for nodes are extracted from there, and hence everything fails to set tunnels and all. | 12:46 |
| *** zz_ja has quit IRC | 12:46 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/project-config: Revert "Temporarily disable OVH GRA, as we are seeing failures" https://review.openstack.org/307792 | 12:46 |
| pabelanger | yolanda: I can confirm cacti.o.o sees the mirrors again too | 12:47 |
| yolanda | woot! | 12:47 |
| pabelanger | Did ovh respond to our emails? | 12:48 |
| pabelanger | but ya, happy to see it back online | 12:48 |
| *** zeih has joined #openstack-infra | 12:48 | |
| pabelanger | 741 check pipeline, here we come | 12:48 |
| yolanda | pabelanger not to email, but i was chatting by irc with Jean-Daniel | 12:48 |
| yolanda | he has been very helpful | 12:48 |
| pabelanger | yolanda: cool! | 12:49 |
| sdague | amrith: in what kind of different way? | 12:49 |
| sdague | amrith: and, yes, it's probably not possible given the constraints, but would be interesting to hear | 12:49 |
| pabelanger | yolanda: as long as somebody at OVH knew the situation, I think we are safe to enable them again | 12:49 |
| amrith | sdague, I don't always want reviews sorted based on the recency of updates | 12:49 |
| *** lucas-hungry is now known as lucasagomes | 12:49 | |
| sdague | ah, right, not much that can be done on that | 12:49 |
| amrith | for example, I would like to group reviews (as I've done in the dashboard) but then I'd like to sort simply on the review number | 12:49 |
| amrith | which is the intuitive sort order based on the fact that it is the left most column | 12:50 |
| *** rlandy has joined #openstack-infra | 12:50 | |
| amrith | ala edward tufte | 12:50 |
| *** kengo_sakai has quit IRC | 12:50 | |
| *** zz_dimtruck is now known as dimtruck | 12:50 | |
| mordred | ihrachys: is it the private ip file that's empty? or all of them? | 12:50 |
| *** kengo_sakai has joined #openstack-infra | 12:51 | |
| ihrachys | mordred: /etc/nodepool/primary_node_private and /etc/nodepool/sub_nodes_private | 12:51 |
| mordred | ihrachys: yah. we've got a patch up to fix that | 12:51 |
| ihrachys | mordred: woot. link? | 12:51 |
| mordred | ihrachys: https://review.openstack.org/#/c/306835/ | 12:51 |
| *** swat30 has quit IRC | 12:51 | |
| mordred | ihrachys: so it SHOULD be fixed now | 12:51 |
| mordred | but probably didn't get picked up until I restarted nodepool this morning | 12:52 |
| ihrachys | great. we will monitor failure rate. our dashboards are 24h, so will take a bit of time before we see the full effect. | 12:52 |
| yolanda | infra-root, can you review https://review.openstack.org/307791 and https://review.openstack.org/307792 ? | 12:52 |
| pabelanger | yolanda: +2 | 12:53 |
| *** swat30 has joined #openstack-infra | 12:53 | |
| yolanda | i have nodepool disabled, so i can update by hand, i guess it will take ages to be on gate | 12:54 |
| *** zeih has quit IRC | 12:54 | |
| yolanda | puppet on nodepool | 12:54 |
| *** yamamoto has joined #openstack-infra | 12:54 | |
| *** HeOS has quit IRC | 12:55 | |
| *** pfallenop has quit IRC | 12:55 | |
| yolanda | modred, care +2 +a those? | 12:56 |
| *** ddieterly has joined #openstack-infra | 12:56 | |
| *** Goneri has joined #openstack-infra | 12:58 | |
| *** infra-red has quit IRC | 12:59 | |
| *** Goneri has quit IRC | 13:00 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch to shade specific statsd for FloatingIPTasks https://review.openstack.org/307233 | 13:00 |
| *** pfallenop has joined #openstack-infra | 13:01 | |
| *** dimtruck is now known as zz_dimtruck | 13:01 | |
| *** infra-red has joined #openstack-infra | 13:01 | |
| fungi | catching up now | 13:02 |
| *** yamamoto has quit IRC | 13:03 | |
| *** rossella_s has quit IRC | 13:03 | |
| *** aysyd has quit IRC | 13:04 | |
| *** eharney has joined #openstack-infra | 13:04 | |
| *** rossella_s has joined #openstack-infra | 13:04 | |
| mordred | yolanda, fungi: https://review.openstack.org/#/c/307439/ | 13:04 |
| *** keedya has joined #openstack-infra | 13:05 | |
| *** Goneri has joined #openstack-infra | 13:05 | |
| yolanda | so mordred, the problem was that command to delete floating ip failed? | 13:05 |
| *** yamamoto has joined #openstack-infra | 13:05 | |
| *** lezbar has quit IRC | 13:05 | |
| mordred | that is one of the problems | 13:05 |
| mordred | and it has the fip leak rate down to VERY low | 13:05 |
| mordred | I ahve one leaked fip on bluebox currently that I'm trying to figure out | 13:06 |
| *** lezbar has joined #openstack-infra | 13:06 | |
| *** pfallenop has quit IRC | 13:07 | |
| mordred | however, I have currently found no cause for it | 13:08 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add docstring for due dates DELETE endpoint https://review.openstack.org/307806 | 13:08 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Update API docs to include worklists, boards, and due dates https://review.openstack.org/307807 | 13:08 |
| Shrews | mordred: why do the fips sometimes fail to delete? do we know that yet? | 13:09 |
| mordred | Shrews: neutron bug | 13:09 |
| Shrews | get out of town | 13:09 |
| mordred | Shrews: sometimes it just returns success and didn't delete the fip | 13:09 |
| *** pfallenop has joined #openstack-infra | 13:09 | |
| mordred | I blame mestery | 13:09 |
| pabelanger | pew, Canada dodged a bullet there | 13:10 |
| fungi | infra-root: we likely need to figure out a solution for the infra-root@openstack.org inbox. that's where we had ovh configured to send our service expiration e-mails (apparently our service with them expires and we have to contact them periodically to renew it?) but that inbox is a dumping ground for backscatter so it's got ~25k unread messages at the moment | 13:10 |
| pabelanger | I don't know how to check that | 13:11 |
| jhesketh | fungi: I have to admit I didn't even know about the inbox (let alone where it goes or how to access it etc) | 13:12 |
| fungi | pabelanger: it's hosted on rackspace's e-mail service right now (details are in our list) | 13:12 |
| Shrews | mordred: hrm... we *could* consider keeping a cache of fips so that we have an "insta-delete" process, then a separate thread to process deletes in the background. maybe we don't need to think that complex yet, though | 13:12 |
| fungi | jhesketh: & | 13:12 |
| * jhesketh must have missed that | 13:12 | |
| fungi | "webmail" | 13:12 |
| fungi | it's lovely | 13:12 |
| *** jrmyjohn has joined #openstack-infra | 13:12 | |
| *** aysyd has joined #openstack-infra | 13:12 | |
| mordred | fungi: agree. I don't think the current solution is a great one | 13:13 |
| fungi | we might want to figure out a different distribution list or something to notify our actual humans and list that as the contact address with our providers | 13:13 |
| *** dizquierdo has quit IRC | 13:14 | |
| *** yamamoto has quit IRC | 13:14 | |
| *** zz_dimtruck is now known as dimtruck | 13:16 | |
| yolanda | yep, i was ignoring that inbox problem as well | 13:16 |
| *** yamamoto has joined #openstack-infra | 13:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 13:17 |
| *** zhurong has quit IRC | 13:19 | |
| *** yamahata has joined #openstack-infra | 13:20 | |
| *** zhurong has joined #openstack-infra | 13:21 | |
| *** esikachev has joined #openstack-infra | 13:22 | |
| AJaeger | yolanda: regarding https://review.openstack.org/307791 and https://review.openstack.org/307791: CAn you apply them manually on nodepool - or ninja-merge? | 13:24 |
| *** _ari_ has joined #openstack-infra | 13:24 | |
| yolanda | AJaeger, i manually updated that on nodepool, because anyway, i disabled puppet there since yesterday, when we had the failures | 13:24 |
| yolanda | so i just set the max-servers manually | 13:25 |
| *** trown is now known as trown|brb | 13:25 | |
| *** mriedem_away is now known as mriedem | 13:25 | |
| openstackgerrit | Merged openstack-infra/project-config: Move OpenStack-Ansible CentOS7 jobs to experimental queue https://review.openstack.org/307369 | 13:26 |
| *** mylu has joined #openstack-infra | 13:26 | |
| *** pfallenop has quit IRC | 13:26 | |
| *** pfallenop has joined #openstack-infra | 13:26 | |
| *** pfallenop has joined #openstack-infra | 13:26 | |
| AJaeger | yolanda: Thanks a lot for getting OVH back online! | 13:27 |
| pabelanger | I always thought you would enqueue a commit to the front of a pipeline. | 13:27 |
| *** adreznec has quit IRC | 13:27 | |
| pabelanger | guess it is promote I was confusing it with | 13:27 |
| *** nwkarsten has joined #openstack-infra | 13:27 | |
| fungi | enqueue tacks it onto the end (like zuul would normally) | 13:27 |
| yolanda | AJaeger, glad to help. It was easier for us this time, because of the timezones | 13:28 |
| fungi | pabelanger: if it's not enqueued yet and you want it inserted at the head of the line, then you end up needing to do both (enqueue, then promote) | 13:28 |
| pabelanger | fungi: ack | 13:28 |
| yolanda | mordred, i can see several errors when deleting floating ips on nodepool log. Is that expected? | 13:29 |
| fungi | yolanda: yes, thanks a bunch. now here's hoping i make some progress with rackspace on the network issues in ord | 13:29 |
| yolanda | fungi , any clue about the problems there? | 13:29 |
| openstackgerrit | Merged openstack-infra/project-config: Translations: Refactor https://review.openstack.org/307237 | 13:30 |
| *** xyang1 has joined #openstack-infra | 13:30 | |
| mordred | yolanda: yes. you should see errors like "attempted to delete floating ip, but failed, not deleting server" | 13:30 |
| yolanda | mordred, yes, several of those | 13:30 |
| fungi | yolanda: cloud support has gotten the noc involved finally, but no update since yesterday no | 13:30 |
| AJaeger | yolanda: What about a service announcement "#status notice we have recovered one cloud and a huuge backlog of jobs to process. Plesae have patience..." | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add releasenotes jobs for networking-ovn https://review.openstack.org/307394 | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config: Make gate-cue-python34 voting; add it as gate job https://review.openstack.org/306950 | 13:30 |
| openstackgerrit | Merged openstack-infra/project-config: Add release notes jobs for networking-bgpvpn https://review.openstack.org/307150 | 13:30 |
| mordred | yolanda: that's part of trying to not leak - if we can't delete it, we also don't delete the server, because otherwise we lose track of our need to delete the fip | 13:30 |
| mordred | and we'll try again later when we do a cleanup pass | 13:30 |
| yolanda | AJaeger, i'll send. It's always good to give good news | 13:30 |
| mestery | mordred: FIPs causing issues again? :( | 13:31 |
| *** xarses has quit IRC | 13:31 | |
| yolanda | floating ips... :( | 13:31 |
| fungi | mordred: oh, was that the solution? defer node deletion until the associated floating ip is no longer listed? | 13:31 |
| openstackgerrit | Merged openstack-infra/project-config: Add rally job for Senlin service https://review.openstack.org/307050 | 13:31 |
| *** oberezovskyi has left #openstack-infra | 13:31 | |
| *** jswarren_ has quit IRC | 13:31 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: cinderclient: Move pylint to non-voting check https://review.openstack.org/307818 | 13:31 |
| *** BigWillie has joined #openstack-infra | 13:31 | |
| *** jswarren_ has joined #openstack-infra | 13:32 | |
| fungi | mestery: they never stopped unfortunately. apparently at least in some deployments fip deletion is a best-effort call and seems to report success even if it eventually fails | 13:32 |
| pabelanger | fungi: yolanda running promote on the 2 patches... Mostly to say I've run the command before | 13:32 |
| *** xarses has joined #openstack-infra | 13:32 | |
| fungi | pabelanger: sure. it's a pretty unexciting command, but have fun | 13:32 |
| *** bdemers has joined #openstack-infra | 13:33 | |
| mestery | fungi: Ugh, that's just terrible :( | 13:33 |
| *** adreznec has joined #openstack-infra | 13:33 | |
| yolanda | #status notice We have recovered one of our cloud providers, but there is a huge backlog of jobs to process. Please have patience until your jobs are processed | 13:33 |
| fungi | statusbot is taking another holiday | 13:33 |
| yolanda | mm | 13:33 |
| *** mriedem has quit IRC | 13:33 | |
| *** trown|brb is now known as trown | 13:34 | |
| mordred | mestery: yes. yes it is | 13:34 |
| yolanda | i'm identified properly, so yes, statusbot looks bad | 13:34 |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 13:34 |
| *** piet has joined #openstack-infra | 13:34 | |
| *** mriedem has joined #openstack-infra | 13:35 | |
| openstackgerrit | Nathan Reller proposed openstack-infra/project-config: Add KMIP Experiment Check to Barbican https://review.openstack.org/307820 | 13:36 |
| *** mriedem has quit IRC | 13:36 | |
| *** mriedem has joined #openstack-infra | 13:36 | |
| *** openstackstatus has joined #openstack-infra | 13:37 | |
| *** ChanServ sets mode: +v openstackstatus | 13:37 | |
| fungi | yolanda: well, i said that because it wasn't in channel | 13:37 |
| *** yamahata has quit IRC | 13:37 | |
| yolanda | yep, not logging anything since 2 days ago | 13:37 |
| *** gongysh has joined #openstack-infra | 13:37 | |
| yolanda | i restarted | 13:37 |
| yolanda | #status notice We have recovered one of our cloud providers, but there is a huge backlog of jobs to process. Please have patience until your jobs are processed | 13:37 |
| fungi | 2016-04-17 11:19:44 <-- openstackstatus (~openstack@eavesdrop.openstack.org) has quit (Ping timeout: 244 seconds) | 13:37 |
| fungi | was the last i have on record for it | 13:38 |
| yolanda | yes, same date entries in logs | 13:38 |
| fungi | anyway, after you restart it, you need to wait at least a few minutes for it to rejoin all configured channels | 13:38 |
| fungi | you can tail its log to see when it stopps joining stuff | 13:38 |
| yolanda | yes, now it should be up but still not processing my entry, it's reconfiguring | 13:38 |
| openstackstatus | yolanda: sending notice | 13:38 |
| yolanda | woot! | 13:38 |
| fungi | there it goes. was just busy i guess | 13:39 |
| fungi | nice that it queues that up. i assumed you had to wait until it was done joining stuff | 13:39 |
| -openstackstatus- NOTICE: We have recovered one of our cloud providers, but there is a huge backlog of jobs to process. Please have patience until your jobs are processed | 13:39 | |
| *** ddieterly is now known as ddieterly[away] | 13:40 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: grafana: added grenade dvr multinode job to neutron dashboard https://review.openstack.org/307825 | 13:41 |
| dansmith | I think you mean "exercise patience" | 13:41 |
| dansmith | I have lots of patience stored up, but only use it when necessary :) | 13:41 |
| *** tonytan4ever has joined #openstack-infra | 13:41 | |
| openstackstatus | yolanda: finished sending notice | 13:41 |
| *** yamamoto has quit IRC | 13:41 | |
| Shrews | mordred: is nodepool running with your latest shade patch for deleting fips? | 13:41 |
| *** infra-red has quit IRC | 13:41 | |
| Shrews | mordred: b/c i think there is a big problem with it | 13:42 |
| fungi | dansmith: we hoard patience here like squirrels gathering nuts for a nuclear winter | 13:42 |
| Shrews | SQUIRREL! | 13:42 |
| dansmith | heh | 13:42 |
| *** yamamoto has joined #openstack-infra | 13:43 | |
| mordred | Shrews: oh yeah? | 13:44 |
| Shrews | mordred: see comments | 13:44 |
| openstackgerrit | Carlos Camacho proposed openstack-infra/tripleo-ci: WIP: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 13:44 |
| *** asettle has quit IRC | 13:45 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/elastic-recheck: Ignore uncategorized failures for puppet jobs https://review.openstack.org/307827 | 13:46 |
| mordred | Shrews: so - tl;dr is I got rid of the boolean because there was no circumstances where returning false would make sense - but you're right about the if not wait line | 13:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 13:48 |
| AJaeger | sdague: could you review https://review.openstack.org/307828 for gerrit-dash-creator, please? | 13:48 |
| mordred | Shrews: how's about that | 13:48 |
| sdague | AJaeger: done | 13:48 |
| Shrews | mordred: i disagree | 13:48 |
| AJaeger | thanks, sdague | 13:49 |
| Shrews | mordred: for our other delete APIs, we return False if there was no resource to delete, True if it deleted, or raise exception on error | 13:49 |
| *** ayoung has joined #openstack-infra | 13:50 | |
| Shrews | mordred: you've altered one aspect of that | 13:50 |
| Shrews | the False one | 13:50 |
| mordred | Shrews: ah - ok. I see what you're saying | 13:50 |
| mordred | one sec | 13:50 |
| *** |-paul-| has quit IRC | 13:50 | |
| *** amitgandhinz has joined #openstack-infra | 13:52 | |
| Shrews | mordred: i am sad that we do not have unit tests that would have caught that. i think i'll write one | 13:53 |
| *** piet has quit IRC | 13:53 | |
| *** jordanP has quit IRC | 13:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 13:54 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack-infra/system-config: Apt mirror for MariaDB https://review.openstack.org/307831 | 13:54 |
| openstackgerrit | amrith proposed openstack/requirements: Add xmltodict to global-requirements https://review.openstack.org/286316 | 13:56 |
| EmilienM | AJaeger: hey, can I use your help please? I'm seeing an error related to my patch on ZUUL_REF http://logs.openstack.org/30/307330/1/check/gate-puppet-openstack-integration-3-scenario001-tempest-ubuntu-trusty/fbdc961/console.html#_2016-04-19_13_11_10_895 | 13:56 |
| EmilienM | AJaeger: the patch was https://review.openstack.org/#/c/307142/ | 13:56 |
| *** rfolco has joined #openstack-infra | 13:57 | |
| AJaeger | EmilienM: a quoting problem ;( | 13:57 |
| mordred | Shrews, fungi: so, also, the current patch seems to have fixed things for osic, but not for bluebox | 13:57 |
| mordred | which I have to say I find exceptionally strange | 13:58 |
| *** EricGonczer_ has joined #openstack-infra | 13:58 | |
| fungi | mordred: so we have more than one fip race to contend with? | 13:58 |
| odyssey4me | clarkb pabelanger mordred with regards to https://review.openstack.org/307831 I have no idea what I'm doing - that's a blind patch submission in the hope that I did the right thing, if you have a gap to review/comment I'd appreciate it... it'd go a long way to assisting OSA and I suspect some other deployment projects | 13:58 |
| EmilienM | AJaeger: well, the same syntax exiss in some other jjb builders | 13:58 |
| AJaeger | EmilienM: let's look at the log file: http://logs.openstack.org/42/307142/1/check/gate-project-config-compare-xml/9cf73be/console.html#_2016-04-18_17_22_30_473 | 13:58 |
| pabelanger | AJaeger: EmilienM: you shouldn't need double {{ }} | 13:58 |
| AJaeger | EmilienM: In a template with parameters, you need to quote { with using {{ | 13:58 |
| *** jlk has quit IRC | 13:58 | |
| *** inc0 has joined #openstack-infra | 13:58 | |
| AJaeger | EmilienM: in a job or a builder without parameters, use a single { | 13:58 |
| AJaeger | EmilienM: as pabelanger says | 13:58 |
| fungi | EmilienM: jjb unfortunately has different substitution behaviors if the macro is used in a job vs a job-template | 13:58 |
| *** jlk has joined #openstack-infra | 13:59 | |
| *** jlk has quit IRC | 13:59 | |
| *** jlk has joined #openstack-infra | 13:59 | |
| EmilienM | ok, my mistake | 13:59 |
| fungi | i agree it's crazily confusing | 13:59 |
| AJaeger | EmilienM: See the log file, it shows the wrong expansion | 13:59 |
| EmilienM | right | 13:59 |
| AJaeger | EmilienM: Sorry, I should have caught that when reviewing... | 13:59 |
| Shrews | mordred: with your previous patchset, it would never actually wait | 13:59 |
| *** ddieterly[away] is now known as ddieterly | 13:59 | |
| Shrews | mordred: because of the 'result' always being None thing | 13:59 |
| fungi | it makes some sense when it's directly in a job or job-template definition, but the fact that it extends to included macros makes it hard to share a macro between jobs and job-templates | 14:00 |
| *** xarses has quit IRC | 14:00 | |
| *** mtanino has quit IRC | 14:00 | |
| mordred | Shrews: ah. well that would do it | 14:00 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Fix ZUUL_REF on puppet-openstack-integration-checkout https://review.openstack.org/307836 | 14:00 |
| *** inc0 has quit IRC | 14:01 | |
| AJaeger | EmilienM: looks fine. Is this breaking all your changes? | 14:01 |
| EmilienM | yes | 14:01 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:01 | |
| fungi | if we had the design to do over again, i'd lobby for jobs and job-templates to be synonymous. jobs with parameterized names able to be used as templates, and jobs without parameterized names just instantiate themselves | 14:01 |
| AJaeger | pabelanger: could you promote 307836 to reduce the number of fails, please? | 14:01 |
| *** baoli_ has quit IRC | 14:01 | |
| EmilienM | AJaeger: thanks | 14:02 |
| fungi | at least that way the parameter expansion could be consistent | 14:02 |
| AJaeger | fungi: Agreed! | 14:02 |
| pabelanger | odyssey4me: left comment. Aside from that, did you test the code locally by standing up a mirror-update? | 14:02 |
| mordred | openstackgerrit: so - it's a good start - but I think we'll actually need to add three different mirrors | 14:02 |
| pabelanger | AJaeger: sure, let me check | 14:02 |
| AJaeger | thanks, pabelanger ! | 14:02 |
| *** psilvad has joined #openstack-infra | 14:02 | |
| *** bdemers has quit IRC | 14:02 | |
| odyssey4me | pabelanger nope, I have no idea how to do that :/ my understanding of puppet is close to zero, never mind anything to do with system-config | 14:03 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Conditionally request a test env from geard https://review.openstack.org/306506 | 14:03 |
| *** links has quit IRC | 14:03 | |
| *** inc0 has joined #openstack-infra | 14:03 | |
| pabelanger | odyssey4me: okay, can you add that to the review. Somebody will have to schedule some time to validate reporepro runs as expected | 14:03 |
| mordred | pabelanger, odyssey4me: kind of like with ceph hammer - we have a repo for the hammer relase of ceph on debian-based distros - I think this is likely to be "deb-mariadb-5" "deb-mariadb-10" and "deb-mariadb-10-1" or something | 14:03 |
| pabelanger | mordred: makes sense\ | 14:04 |
| mordred | or it might be ubutun-trusty-mariadb-5 | 14:04 |
| mordred | since it looks like each repo has a precise and a trusty release in it | 14:04 |
| *** mylu has quit IRC | 14:05 | |
| mordred | pabelanger, odyssey4me: I imagine it'll probably be easier for one of us to help get that patch ready - I think you've got the info in there that someone can work from at least | 14:05 |
| AJaeger | pabelanger: can you manually put 307836 directly in the gate, please - promote and enqueue? | 14:05 |
| odyssey4me | mordred I'd appreciate that - my intent was simply to get the ball rolling and to hope to get some help. | 14:06 |
| mordred | yah. I think that's right - it's a system/process that we're still feeling our way around a little bit | 14:06 |
| *** piet has joined #openstack-infra | 14:06 | |
| *** mylu has joined #openstack-infra | 14:07 | |
| odyssey4me | mordred I've added a review comment to invite anyone to update the patch accordingly and to note that I have no idea what I'm doing. :p | 14:08 |
| *** infra-red has joined #openstack-infra | 14:09 | |
| pabelanger | personally, I'd like to land https://review.openstack.org/#/c/289520/ before we start adding more edge case mirrors | 14:11 |
| pabelanger | AJaeger: let me check | 14:12 |
| AJaeger | pabelanger: that'S the one you just approved for EmilienM | 14:13 |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use the cached ironic-python-agent images https://review.openstack.org/301699 | 14:13 |
| odyssey4me | pabelanger fair point, and happy with that - I'm just hoping to get it into the queue :) | 14:14 |
| pabelanger | AJaeger: done | 14:15 |
| *** rbrndt_ has joined #openstack-infra | 14:15 | |
| odyssey4me | I'm happy if my patch gets put behind that one as a dep | 14:15 |
| AJaeger | thanks, pabelanger. EmilienM, your change should merge quickly! | 14:15 |
| EmilienM | awesome, thanks a ton | 14:15 |
| pabelanger | fungi: AJaeger: do you have any info on 307553? Adding gentoo support to bindep-fallback.txt? | 14:16 |
| AJaeger | jroll: for a new report that you add to your project: The goverance change can go in after we merge the project-config change - if you approve the change, ttx will consider it as bookkeeping and just merge it. So, the change needs to exist but orderin here is: project-config first. | 14:16 |
| AJaeger | pabelanger: no info on that. prometheanfire was working on that last night but I didn't read all backscroll | 14:17 |
| pabelanger | fungi: AJaeger: I left a 0 comment on it, asking why we need it. Especially since we don't have any gentoo nodes in the gate | 14:17 |
| pabelanger | AJaeger: okay, let me check backscroll | 14:17 |
| jroll | AJaeger: sorry, meant to say "until the governance patch is up" :) | 14:17 |
| AJaeger | jroll: Ah;) That's perfectly fine then ;) | 14:17 |
| fungi | pabelanger: i was talking to prometheanfire about it last night and providing some guidance but haven't actually had time to review the change yet. i think the main reason to have something in there is to make it easier to introduce an effective functional test of bindep on a gentoo worker (probably non-voting/advisory) | 14:17 |
| AJaeger | pabelanger: around 4:00 UTC | 14:18 |
| *** oanson has quit IRC | 14:18 | |
| fungi | AJaeger: ^ | 14:18 |
| *** cloudtrainme has joined #openstack-infra | 14:18 | |
| pabelanger | AJaeger: fungi: Ya, that is the missing piece. I wasn't sure the status of getting a gentoo worker online | 14:18 |
| *** asettle has joined #openstack-infra | 14:18 | |
| AJaeger | fungi, thanks for context | 14:19 |
| *** woodster_ has joined #openstack-infra | 14:19 | |
| fungi | i think there's also a corresponding change to introduce gentoo support into bindep | 14:19 |
| AJaeger | fungi, pabelanger : One question on review is ordering: Alphabetical or logical? prometheanfire added logical but it'S not logical currently (like IceWeasel is alphabetical) | 14:19 |
| fungi | it's a multi-pronged effort and he's working on the various pieces in parallel | 14:19 |
| pabelanger | AJaeger: I would be okay with a check to sort for alphabetic being added | 14:20 |
| *** rbrndt_ has left #openstack-infra | 14:20 | |
| fungi | i think the last time i cleaned up that file i sorted with LC_COLLATE=C.UTF-8 | 14:20 |
| *** rbrndt_ has joined #openstack-infra | 14:20 | |
| *** piet has quit IRC | 14:21 | |
| fungi | but i think generally we're not going to be updating that file much | 14:21 |
| fungi | so a special sort detection job for it is probably overkill | 14:21 |
| AJaeger | pabelanger: I don't think we need a check, agree with fungi | 14:21 |
| AJaeger | But wanted to check for review consistency, whether a -1 would be ok... | 14:21 |
| fungi | unlike some of the other stuff in project-config where people are adding things weekly or even daily | 14:21 |
| fungi | could just use a sane sort on the file and push up a new patchset instead of leaving a -1 | 14:22 |
| AJaeger | pabelanger, fungi : Let me sort the change directly and add a comment... | 14:22 |
| pabelanger | jenkins already left a -1 :) | 14:23 |
| AJaeger | then let me just comment... | 14:23 |
| *** asettle has quit IRC | 14:23 | |
| *** salv-orlando has quit IRC | 14:23 | |
| fungi | yeah, if it needs a new patchset anyway to pass tests, i'd just leave a comment asking for consistent ordering | 14:23 |
| *** asselin_ has joined #openstack-infra | 14:24 | |
| fungi | then he can knock them out at teh same time | 14:24 |
| AJaeger | agreed - and commented | 14:24 |
| AJaeger | yeah, zuul check queue is getting shorter, "only" 679 jobs in the queue... | 14:26 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: Updaate get summit event endpoints https://review.openstack.org/307846 | 14:26 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard: Add offset/limit pagination to subscription events https://review.openstack.org/307847 | 14:26 |
| pabelanger | AJaeger: were over the hump | 14:26 |
| *** jordanP has joined #openstack-infra | 14:27 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix ZUUL_REF on puppet-openstack-integration-checkout https://review.openstack.org/307836 | 14:27 |
| AJaeger | pabelanger: I hope - normally it increases around this time with the US starting to work, so let's see... | 14:27 |
| *** nbloom has left #openstack-infra | 14:28 | |
| *** bdemers has joined #openstack-infra | 14:29 | |
| *** piet has joined #openstack-infra | 14:29 | |
| *** inc0 has quit IRC | 14:30 | |
| *** ddieterly is now known as ddieterly[away] | 14:31 | |
| *** bdemers has quit IRC | 14:31 | |
| *** ddieterly[away] is now known as ddieterly | 14:31 | |
| *** _ari_ has quit IRC | 14:34 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources: Updaate get summit event endpoints https://review.openstack.org/307846 | 14:35 |
| *** _ari_ has joined #openstack-infra | 14:36 | |
| *** zhurong has quit IRC | 14:37 | |
| pabelanger | fungi: I'm going to start work on puppetdb.o.o replacement server, at least launching it | 14:38 |
| *** bdemers has joined #openstack-infra | 14:38 | |
| dims | AJaeger : fungi : are we low on test nodes? | 14:39 |
| *** mtanino has joined #openstack-infra | 14:39 | |
| *** dizquierdo has joined #openstack-infra | 14:39 | |
| mordred | dims: yes. we're fighting with several provider outages | 14:39 |
| dims | mordred : ouch thanks for the update | 14:39 |
| fungi | dims: less low now, but recovery will still take a bit of time | 14:39 |
| mordred | dims: we're actually doing better now since ovh is back | 14:39 |
| *** xarses has joined #openstack-infra | 14:39 | |
| mordred | dims: but we're down an entire rackspace | 14:40 |
| AJaeger | dims: we were very low the last 28hours, and start recovering now | 14:40 |
| *** infra-red has quit IRC | 14:41 | |
| dims | thanks fungi and AJaeger | 14:42 |
| *** MarkAtwood has joined #openstack-infra | 14:42 | |
| *** nbloom has joined #openstack-infra | 14:42 | |
| *** mylu has quit IRC | 14:42 | |
| *** amotoki has joined #openstack-infra | 14:43 | |
| *** bdemers has quit IRC | 14:43 | |
| *** eric_lopez has quit IRC | 14:43 | |
| clarkb | fungi: mestery and by some deployments you mean every deployment we have had to use fips on :) | 14:44 |
| *** avarner has joined #openstack-infra | 14:44 | |
| fungi | clarkb: well, as it turns out the two we have fips on exhibit different races | 14:44 |
| mordred | fungi: it's possible I was wrong abot that | 14:44 |
| fungi | oh | 14:45 |
| mordred | fungi: Shrews found a bug in my patch | 14:45 |
| mordred | mestery: however, clarkb is quite right - this is a problem in every neutron that exists | 14:45 |
| mordred | mestery: I _guess_ someone should file a bug or something maybe | 14:45 |
| clarkb | hp, bb, and psic | 14:45 |
| clarkb | er osic | 14:45 |
| fungi | gotcha, didn't connect the false returns masking former exceptions as the explanation for the "new" race in bluebox | 14:45 |
| clarkb | the race has always been, delete fip, get back success, delete server, except fip didnt delete | 14:46 |
| mordred | yup | 14:47 |
| clarkb | and now we have no transitive record of it that can be used for deletion | 14:47 |
| *** eric_lopez has joined #openstack-infra | 14:47 | |
| mordred | clarkb: btw -I put up a patch to do periodic fip cleanips for neutron clouds, since we _can_ do that safely | 14:47 |
| *** e0ne has quit IRC | 14:47 | |
| mordred | clarkb: should mitigate while we continue to try to get a bulletproof leak prevention | 14:48 |
| openstackgerrit | Rossella Sblendido proposed openstack-infra/reviewday: Use pip to install gerrit-dash-creator https://review.openstack.org/307510 | 14:48 |
| mescanef | hello:-) please review puppet nodepools changes: https://review.openstack.org/#/c/299661/ and https://review.openstack.org/#/c/303306/ | 14:48 |
| pabelanger | mordred: fungi: which venv should I be running launch-node.py from on puppetmaster.o.o these days? | 14:49 |
| fungi | pabelanger: i've just been using the system environment | 14:50 |
| *** infra-red has joined #openstack-infra | 14:50 | |
| pabelanger | that's what I thought: http://paste.openstack.org/show/494651/ | 14:50 |
| *** infra-red has quit IRC | 14:50 | |
| *** mgould has joined #openstack-infra | 14:51 | |
| *** infra-red has joined #openstack-infra | 14:51 | |
| fungi | oh, right, broken newer novaclient | 14:51 |
| AJaeger | jeblair: gerrit-dash-creator is now on pypi and we can install it from there - https://review.openstack.org/#/c/307510 | 14:51 |
| fungi | is ~/git/openstack-infra/system-config up to date with teh tip of master? | 14:51 |
| rodrigods | AJaeger, ping... could you take a look again at https://review.openstack.org/#/c/298696/ whenever you have some time? | 14:51 |
| fungi | pabelanger: i've been running the version in /opt/system-config/production/launch | 14:51 |
| pabelanger | fungi: Ah, good call. let me rebase to be sure | 14:52 |
| *** mylu has joined #openstack-infra | 14:52 | |
| pabelanger | fungi: okay, that helped. Thanks | 14:52 |
| AJaeger | rodrigods: LGTM | 14:52 |
| *** nwkarste_ has joined #openstack-infra | 14:52 | |
| AJaeger | bbl | 14:52 |
| *** hashar has joined #openstack-infra | 14:52 | |
| mordred | no venv | 14:52 |
| rodrigods | AJaeger, thanks | 14:52 |
| mordred | pabelanger: sorry, no venv | 14:53 |
| *** nwkarsten has quit IRC | 14:53 | |
| pabelanger | mordred: thanks, fungi's tip to rebase helped | 14:53 |
| mordred | woot | 14:53 |
| *** ddieterly is now known as ddieterly[away] | 14:53 | |
| clarkb | mordred: what code are we running on nodepool now? | 14:54 |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 14:54 |
| *** jlanoux has quit IRC | 14:54 | |
| mordred | clarkb: nodepool is master. shade is https://review.openstack.org/#/c/307439/ | 14:54 |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Cap testresources<2.0.0 https://review.openstack.org/307858 | 14:55 |
| *** ddieterly[away] is now known as ddieterly | 14:56 | |
| *** yottatsa has joined #openstack-infra | 14:57 | |
| *** bdemers has joined #openstack-infra | 14:57 | |
| prometheanfire | fungi: pabelanger AJaeger: ya, noticed that it was alphabetical, but given how we have package categories I wasn't sure what to do | 14:57 |
| prometheanfire | fungi: pabelanger AJaeger: also, the fallback commit needs this first https://review.openstack.org/307493 | 14:58 |
| prometheanfire | I realized that after I submitted the change, it's because of the / in package names | 14:58 |
| *** armax has joined #openstack-infra | 14:58 | |
| *** dtantsur is now known as dtantsur|brb | 14:59 | |
| *** jaosorior has quit IRC | 15:00 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/glean: Read skip_dns_write option from config file https://review.openstack.org/307581 | 15:00 |
| nibalizer | good morning | 15:01 |
| *** amotoki has quit IRC | 15:02 | |
| openstackgerrit | Matthew Treinish proposed openstack/requirements: Cap testresources<2.0.0 https://review.openstack.org/307858 | 15:02 |
| *** nwkarste_ has quit IRC | 15:02 | |
| *** nwkarsten has joined #openstack-infra | 15:02 | |
| anteaya | morning nibalizer | 15:05 |
| *** nmagnezi has quit IRC | 15:05 | |
| *** cloudtrainme has quit IRC | 15:06 | |
| *** baoli has joined #openstack-infra | 15:07 | |
| *** mylu has quit IRC | 15:07 | |
| *** cloudtrainme has joined #openstack-infra | 15:08 | |
| clarkb | mordred: http://grafana.openstack.org/dashboard/db/nodepool-osic what changed at 1130UTC (looking at fip delete and error node graphs) | 15:08 |
| *** trown is now known as trown|afk | 15:09 | |
| *** mylu has joined #openstack-infra | 15:10 | |
| pabelanger | nibalizer: o/ | 15:10 |
| nibalizer | how is our happy little system this morning | 15:11 |
| *** arxcruz has quit IRC | 15:11 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add job to build generic debian packages https://review.openstack.org/307742 | 15:11 |
| mordred | clarkb: woot! I now know at least one reason why the shade mitigations for this are not helping | 15:12 |
| clarkb | mordred: this can either be good or terrible news :) | 15:13 |
| mordred | clarkb: it's good news ... | 15:13 |
| *** Nikolay_St has quit IRC | 15:13 | |
| mordred | clarkb: (I may have just added some debug trace logging so that I can what the **** is happening) | 15:13 |
| *** nwkarste_ has joined #openstack-infra | 15:13 | |
| mordred | clarkb: turns out ... we don't pass wait=True to deleteServer - so the "make sure the fip is deleted" code we just added to shade just flat isn't getting called | 15:13 |
| anteaya | claire tells me that app support questions that get posted in here are to be directed to summitapp@openstack.org | 15:14 |
| *** nwkarsten has quit IRC | 15:14 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for testresources release pbr conflict on kilo https://review.openstack.org/307866 | 15:15 |
| openstackgerrit | Ian Y. Choi proposed openstack-infra/system-config: Manifest for configuration translation checksite https://review.openstack.org/276466 | 15:16 |
| *** MarkAtwood has quit IRC | 15:16 | |
| *** MarkAtwood has joined #openstack-infra | 15:17 | |
| mpaolino | Hello, I am implementing a new devstack plugin, and I am not sure I understood everything about how to push this through gerrit | 15:18 |
| *** kengo_sa_ has joined #openstack-infra | 15:18 | |
| mpaolino | the plugin wiki page says: "best practice is to build a dedicated openstack/devstack-plugin-FOO project" | 15:18 |
| mpaolino | but it is not written how to build it | 15:18 |
| mpaolino | however, I created a blueprint and a launchpad project | 15:19 |
| *** abregman has quit IRC | 15:19 | |
| clarkb | mpaolino: http://docs.openstack.org/infra/manual/creators.html | 15:19 |
| mpaolino | the objective is to port this devstack patch to a plugin (as requested by the community): https://review.openstack.org/#/c/108714/ | 15:19 |
| clarkb | mordred: so what changed at 1130? was that working around the lack of the wait flag? | 15:20 |
| *** ayoung has quit IRC | 15:20 | |
| mpaolino | clarkb, thanks, I didn't get there. I'll have a look | 15:20 |
| *** kengo_sakai has quit IRC | 15:21 | |
| *** tonytan4ever has quit IRC | 15:22 | |
| *** kengo_sa_ has quit IRC | 15:22 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/elastic-recheck: Add query for testresources release pbr conflict on kilo https://review.openstack.org/307866 | 15:22 |
| mordred | clarkb: that might have just been me cleaning out fips manually, or something with patchset three | 15:22 |
| *** tonytan4ever has joined #openstack-infra | 15:23 | |
| *** jlanoux has joined #openstack-infra | 15:23 | |
| *** nwkarsten has joined #openstack-infra | 15:23 | |
| *** nwkarste_ has quit IRC | 15:23 | |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Re-write tripleo-jobs to use gerrit https://review.openstack.org/307532 | 15:24 |
| openstackgerrit | Dan Prince proposed openstack-infra/tripleo-ci: Switch website to use tripleo-jobs-gerrit.py https://review.openstack.org/307872 | 15:24 |
| *** fhubik has joined #openstack-infra | 15:25 | |
| *** infra-red has quit IRC | 15:26 | |
| *** jordanP has quit IRC | 15:26 | |
| zigo | clarkb: Finally, I fixed the issue by doing "ssh localhost 'cd ${WORKSPACE} ; gbp buildpackage", which does relogin which sbuild expects ! So, now, we finally have built a package! \o/ | 15:26 |
| zigo | clarkb: https://review.openstack.org/#/c/307706/ <--- See the successful build log. | 15:27 |
| *** armax has quit IRC | 15:27 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 15:27 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Wait on server deletion to deal with FIP leaks https://review.openstack.org/307875 | 15:27 |
| zigo | clarkb: mordred: pabelanger: Now that we have an initial package built, how should I work on publishing it on the infra's AFS ? | 15:28 |
| mordred | zigo: that's the thing we need to design next week at the summit | 15:28 |
| zigo | mordred: https://etherpad.openstack.org/p/newton-deb-packaging | 15:28 |
| *** nbloom has left #openstack-infra | 15:28 | |
| mordred | yup. that also should be on the summit schedule now | 15:28 |
| zigo | mordred: Can you insert this etherpad link in the submit's schedule ? | 15:28 |
| mordred | already done | 15:28 |
| zigo | Ah, cool ! :) | 15:28 |
| zigo | Thanks. | 15:28 |
| zigo | mordred: do you think we should use something like mini-dinstall, mini-dak, etc. ? | 15:29 |
| mordred | we use reprepro currently | 15:29 |
| zigo | mordred: If so, i can investigate and try them all. | 15:29 |
| mordred | but that's not the hard bit | 15:29 |
| zigo | mordred: What's the hard bit then? | 15:29 |
| zigo | reprepro doesn't have a facility to upload packages, that's one out of multiple issues it has. | 15:30 |
| mordred | we're going to need to figure out a two-stage system like we do for other publications - where we upload to tarballs.o.o at the end of the build, and then have something trusted download from tarballs and add to the mirror | 15:30 |
| *** oanson has joined #openstack-infra | 15:30 | |
| mordred | zigo: it does not need one | 15:30 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 15:30 |
| mordred | zigo: because of how we use things | 15:30 |
| zigo | mordred: Sure. | 15:30 |
| mordred | we will never have a job "upload" packages to a repo | 15:30 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 15:31 |
| mordred | because of privilege issues | 15:31 |
| zigo | mordred: Though what you're saying about "two-stage system like we do for other publications - where we upload to tarballs.o.o at the end of the build", well that's what something like mini-dinstall would do. | 15:31 |
| zigo | Or even dak ... :P | 15:31 |
| mordred | right. but we won't run those on tarballs.o.o | 15:31 |
| zigo | But I'm not supposing we'll go through setting-up something as complicated as dak. | 15:31 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 15:31 |
| zigo | mordred: So you want to continue with reprepro? | 15:31 |
| mordred | I do not see any benefit in not continuing to use it currently - we mostly just have to figure out what "uploading the job results" looks like, and what "downloading them from tarballs.o.o" looks like | 15:32 |
| mordred | it's possible that using dget will even work | 15:32 |
| *** nwkarsten has quit IRC | 15:33 | |
| mordred | I just think its complex enough that talking through it in person is worthwhile | 15:33 |
| *** nwkarsten has joined #openstack-infra | 15:33 | |
| mordred | it's also possible that you're right and dak or mini-dinstall is the thing we need to do | 15:33 |
| pabelanger | I for see a white board too | 15:33 |
| mordred | but yes | 15:33 |
| mordred | I think a white board and all of us fully grokking all of the bits is super important :) | 15:33 |
| *** ZZelle has joined #openstack-infra | 15:34 | |
| zigo | mordred: Ok, so I *should* try to investigate these repo management tools before the summit session then. | 15:34 |
| zigo | It'd be useless to just go to the session and say hello... | 15:34 |
| mordred | heh | 15:34 |
| zigo | It's best to have content ready. | 15:34 |
| mordred | totally | 15:34 |
| mordred | I can walk through the reprepro flow for folks | 15:34 |
| zigo | :) | 15:34 |
| mordred | it's pretty easy to publish locally built packages with it | 15:34 |
| mordred | zigo: it might be useful for you to look at how we're using it for mirroring | 15:35 |
| zigo | Yup, but IMO, it's even more easy to *not* use reprepro and use your own shell script based on apt-ftparchive and dpkg-scanpackages. | 15:35 |
| mordred | which is slighly different | 15:35 |
| *** hdd has joined #openstack-infra | 15:36 | |
| *** weshay has quit IRC | 15:36 | |
| zigo | For mirroring, if you're using reprepro, then it will have to be redone in another way, because reprepro wont be able to pickup modified packages of the same (or older) versions. | 15:36 |
| mordred | what do you mean? | 15:36 |
| zigo | And we do need that feature, since we can modify a package before uploading to distros (and we want to keep debian/changelog a history *in distros). | 15:36 |
| mordred | these will be different repos | 15:36 |
| openstackgerrit | Merged openstack-infra/project-config: Vitrage - Zuul/Jenkins fix https://review.openstack.org/307634 | 15:36 |
| mordred | we will not publish the packages to the mirror repo | 15:37 |
| zigo | Ah, right. | 15:37 |
| mordred | we've also talked about moving to rsync for mirroring - but we need to develop a repo validity check tool | 15:37 |
| mordred | before we can do that | 15:37 |
| zigo | BTW, I intend to help with the packaging of everything that infra is using. | 15:37 |
| mordred | so it's possible that apt-ftparchive or whatever wouldbe fine | 15:37 |
| *** dtantsur|brb is now known as dtantsur | 15:37 | |
| mordred | neat! | 15:37 |
| zigo | I already attempted to package zuul and nodepool. | 15:37 |
| zigo | I have that kind of ready, but couldn't upload to Sid, due to statsd 3.x compat (which should be solved these days). | 15:38 |
| *** Qiming_ has quit IRC | 15:38 | |
| *** vgridnev has quit IRC | 15:38 | |
| zigo | mordred: Ok, thanks for your input, will do the repo management tool survey now. | 15:38 |
| zigo | mordred: In the mean while: https://review.openstack.org/#/c/307742/ | 15:39 |
| zigo | :) | 15:39 |
| mordred | zigo: thanks! | 15:39 |
| pabelanger | fungi: clarkb: What is the best way to have launch-node use an open review of system-config? 284874 | 15:39 |
| *** EricGonczer_ has quit IRC | 15:40 | |
| pabelanger | clarkb: fungi I guess I would update it to support both trusty / precise and get it landed first | 15:40 |
| *** dprince has quit IRC | 15:40 | |
| anteaya | I'm out Wednesday afternoon, all day Thursday and Friday morning | 15:40 |
| clarkb | pabelanger: on the puppet side? you could use a puppet env in the past | 15:41 |
| clarkb | not sure if mordreds new masterless stuff supports similar | 15:41 |
| nibalizer | any git repo with a checkout of system-config would work | 15:41 |
| nibalizer | can just do that in your homedir | 15:42 |
| *** apoorvad has joined #openstack-infra | 15:42 | |
| clarkb | nibalizer: but how does it know to use that? | 15:42 |
| clarkb | the current ansible copies from/opt/system-config/production iirc | 15:42 |
| nibalizer | 'it' ? | 15:42 |
| clarkb | launch node | 15:42 |
| nibalizer | well that would be fine right | 15:42 |
| * nibalizer looks at the patch in question | 15:43 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Wait for floating ip deletion as part of server deletion https://review.openstack.org/307439 | 15:43 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Have delete_server use the timed server list cache https://review.openstack.org/307887 | 15:43 |
| mordred | clarkb: ^^ those two and the nodepool patch I sent up a few seconds ago | 15:43 |
| nibalizer | ohhhh | 15:43 |
| *** vincentll has quit IRC | 15:43 | |
| nibalizer | pabelanger: sweet lets do it | 15:43 |
| mordred | clarkb: should get us into the active codepath for retrying teh deletes | 15:43 |
| pabelanger | Ya, just patching it to support both for the moment | 15:44 |
| mordred | clarkb: I believe we should also implement a 5-second cache for floating_ip_list - since that's now a thing we're gonna run more with this | 15:44 |
| mordred | clarkb: but I think we can do that as an add-on | 15:44 |
| nibalizer | pabelanger: ya i think that makes the most sense | 15:44 |
| *** EricGonczer_ has joined #openstack-infra | 15:44 | |
| mordred | clarkb: since we're currently running it without a 5-second cache and nothing is unhappy about it | 15:44 |
| nibalizer | running two then decomissioning one is what we're actually doing, so our config mgmt might as well reflect that | 15:44 |
| *** links has joined #openstack-infra | 15:45 | |
| clarkb | nibalizer: mordred but how do you get launch node to run the unmerged code? | 15:45 |
| nibalizer | clarkb: i don't think we do | 15:45 |
| clarkb | ok thats a bug then | 15:45 |
| nibalizer | not necessarily | 15:46 |
| nibalizer | why would we run unmerged code? | 15:46 |
| clarkb | it is a loss of functionality | 15:46 |
| nibalizer | that sounds like a bug | 15:46 |
| fungi | yeah, in the long-ago, in the beforetime, we could pass --environment=foo to get it to use /opt/system-config/foo instead of /opt/system-config/production when trying to launch a server | 15:46 |
| clarkb | to test for one | 15:46 |
| mordred | nibalizer: yah - equiv to what fungi is saying there | 15:46 |
| nibalizer | in 99% of cases you can go to a host and checkout a review and test | 15:46 |
| fungi | it was one of the only solution we had, really, for figuring out whether a full deployment of a server could bootstrap successfully | 15:47 |
| fungi | s/one/of// | 15:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add puppetdb.o.o trusty support https://review.openstack.org/284874 | 15:47 |
| mordred | yah - but there is a thing that has chnaged | 15:47 |
| pabelanger | nibalizer: ^ | 15:47 |
| mordred | which is that servers don't need to be registered with puppetmaster | 15:47 |
| mordred | which makes the bringup process potentially slightly different | 15:47 |
| pabelanger | opps | 15:47 |
| nibalizer | so what can be done is this, run launch node without a nod definition, it will get the default node definition and come up | 15:47 |
| pabelanger | 1 sec | 15:47 |
| fungi | now we're back to submitting an endless stream of changes and spending a couple days retrying to launch a server until it boots clean and error-free the first time | 15:47 |
| mordred | we could ACTUALLY skip having launch_node do anything other than ... yes, what nibalizer just said | 15:48 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config: Add puppetdb.o.o trusty support https://review.openstack.org/284874 | 15:48 |
| pabelanger | nibalizer: Ya, that works too | 15:48 |
| nibalizer | then you can go to /opt/system-config/production on the host and check out a review and run it | 15:48 |
| *** kengo_sakai has joined #openstack-infra | 15:48 | |
| clarkb | lets not make this even more complocated... | 15:48 |
| *** gongysh has quit IRC | 15:48 | |
| clarkb | the current system is 100% automated once you select ehat you need | 15:49 |
| *** tosky has quit IRC | 15:49 | |
| clarkb | there is a goal to further automate this | 15:49 |
| pabelanger | I have to walk my daughter to school, so holding off on launch-node ATM. Will read backscroll once I get back | 15:49 |
| clarkb | its why we ansible in the first place | 15:49 |
| fungi | well, unless we actually switch to a two-phase bootstrapping process in launch-node.py (which i'm not particularly opposed to for reproducibility), puppeting it once with an empty node definition and then again with content is not a valid test of whether it can bootstrap in one shot | 15:49 |
| clarkb | a suggestion to manually login and fiddlr with stuff is a regression | 15:49 |
| nibalizer | clarkb: i disagee | 15:49 |
| nibalizer | either way there was fiddling | 15:50 |
| nibalizer | either you fiddled on the puppetmatser or you fiddle on the host | 15:50 |
| *** gongysh has joined #openstack-infra | 15:50 | |
| nibalizer | basically the operator has to set up a set of git repos and call 'execute' | 15:50 |
| clarkb | ya but the fiddle on puppetmaster is automated | 15:50 |
| fungi | the up side to when we fiddled on the puppetmaster was that it could be done in advance of launching the node | 15:50 |
| *** vhosakot has joined #openstack-infra | 15:50 | |
| fungi | so that we could confirm it would successfully bootstrap in one go | 15:50 |
| *** lakshmiS has joined #openstack-infra | 15:51 | |
| fungi | including checking out changes to other puppet modules in the dev env being used, not just system-config | 15:51 |
| nibalizer | so if we want to we can hack ansible to be environment aware | 15:51 |
| nibalizer | nothing we've done prevents that | 15:51 |
| fungi | i get that's something we give up by going masterless, i just think (and probably disagreeing with clarkb slightly) that whatever our test process is should also reflect the actual deployment process step-wise | 15:52 |
| *** mikelk has quit IRC | 15:52 | |
| *** daemontool has quit IRC | 15:52 | |
| clarkb | I just want equivalent functiobality... | 15:52 |
| clarkb | I still use the master to launch stuff because we do t have that | 15:52 |
| *** daemontool has joined #openstack-infra | 15:53 | |
| nibalizer | what? | 15:53 |
| clarkb | the ansibke launch script is inconplete and did/does not have featurea I require | 15:53 |
| clarkb | it was merged before I could review it | 15:53 |
| fungi | i on the other hand am fine not having feature-parity with the pre-ansible launch-node.py, but do want whatever mechanism we have for testing un-merged changes in the launch process to be an accurate representation of the real launch process | 15:53 |
| nibalizer | no i mean the ansible role called 'puppet' is where the feature would need to be added | 15:54 |
| clarkb | I checkout old launch scripy and use master | 15:54 |
| nibalizer | basically right now launch-node.py calls out to ansible's puppet role | 15:54 |
| nibalizer | which would need to learn to copy thigns based on environment | 15:54 |
| nibalizer | it can already remotely execute puppet in a different environment | 15:54 |
| *** gyee has joined #openstack-infra | 15:54 | |
| nibalizer | basically it just needs to know to synchronize other files in different environments | 15:55 |
| nibalizer | it might already have it | 15:55 |
| nibalizer | we currently pass src: "{{ manifest_base }}/{{ puppet_environment }}" in to the 'copy puppet modules' task | 15:56 |
| *** kengo_sakai has quit IRC | 15:56 | |
| *** tosky has joined #openstack-infra | 15:56 | |
| *** ddieterly is now known as ddieterly[away] | 15:57 | |
| mordred | yes. we just default to production as the environment | 15:57 |
| *** bpokorny has joined #openstack-infra | 15:57 | |
| *** tphummel has joined #openstack-infra | 15:57 | |
| *** links has quit IRC | 15:57 | |
| *** Rockyg has joined #openstack-infra | 15:57 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add report URL to status.json https://review.openstack.org/307891 | 15:57 |
| clarkb | I think that there are two major pieces that we neglected. One is volumes because afs and mirrors and all that are far more important today. This is mostly addressed but needs merging? and how to bootstrap things for testing | 15:58 |
| *** gokrokve has joined #openstack-infra | 15:58 | |
| *** nwkarste_ has joined #openstack-infra | 15:58 | |
| clarkb | the first was neglected in old script too (reverted entirely iirc) then I put it back in place whrn I had to make a bunch of mirrora | 15:58 |
| jeblair | 307891 is part 1/2 to enable links to logs.o.o on the status page | 15:59 |
| *** nwkarsten has quit IRC | 15:59 | |
| *** doug-fish has quit IRC | 16:00 | |
| fungi | also --keep probably just needs to become default now. it used to be that if a launch failed, having the instance automatically deleted was fine because all the puppet stdout was streamed to your console. now the only real diagnostic we have for the puppet phase of things is the syslog on the instance, which is lost if you don't pass --keep | 16:00 |
| *** Guest73397 has quit IRC | 16:00 | |
| *** doug-fish has joined #openstack-infra | 16:00 | |
| fungi | so the situations where not using --keep are useful are few and far between | 16:00 |
| clarkb | fungi: +1 | 16:00 |
| *** kushal has quit IRC | 16:00 | |
| mordred | we could configure out setup to pass the puppet stdout back to ansible, fwiw | 16:00 |
| clarkb | but this is why I proposed a workroom at summit | 16:00 |
| mordred | s/out/our/ | 16:00 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config: Zuul status: Link to logs.o.o on completed builds https://review.openstack.org/307892 | 16:01 |
| jeblair | part 2/2 ^ | 16:01 |
| mordred | jeblair: neat! | 16:01 |
| *** ddieterly[away] is now known as ddieterly | 16:01 | |
| clarkb | there are a lot of minor issues in the ansible that only seem to bite you when ypu actually try to do things that arent run in cron | 16:01 |
| clarkb | and a few that hit that that we have ignored | 16:01 |
| clarkb | so set time aside to work on them and make it better | 16:01 |
| mordred | ++ | 16:01 |
| mordred | clarkb: I am now running nodepool with the nodepool and shade patches outstanding for the fip issue applied | 16:02 |
| clarkb | mordred: this includes periodic cleanup? | 16:02 |
| *** nadya has quit IRC | 16:02 | |
| mordred | clarkb: no, I did not apply that | 16:02 |
| *** zz_ja has joined #openstack-infra | 16:02 | |
| clarkb | ok that should let us see how we leak now | 16:02 |
| *** zzzeek has quit IRC | 16:02 | |
| mordred | clarkb: because that requires also adding a config value to nodepool.yaml | 16:02 |
| mordred | clarkb: yah. I'm optimistic | 16:02 |
| *** zzzeek has joined #openstack-infra | 16:03 | |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Enable an environment aware launch node script https://review.openstack.org/307894 | 16:03 |
| mordred | clarkb: I am seeing debug logging indicating that the wait logic is now being exercised | 16:03 |
| nibalizer | clarkb: ok so there you go that probably does it | 16:03 |
| *** sshnaidm has quit IRC | 16:03 | |
| nibalizer | definitely going to be some bugs in there and in the ansible role but those can be ironed out in testing | 16:03 |
| mordred | clarkb: if you want to watch along, I'm doing tail -f /var/log/nodepool/debug.log | grep "DEBUG shade" | grep -v "Waiting ..0 seconds" | grep -v "Endpoint not found" | 16:04 |
| *** andymaier has quit IRC | 16:04 | |
| clarkb | nibalizer: can we add it back in as a script arg? | 16:04 |
| *** scheuran has quit IRC | 16:04 | |
| nibalizer | clarkb: thats literally what I did? | 16:04 |
| nibalizer | i just cheesed out of using getopt or whatever the hip python thing is | 16:05 |
| clarkb | nibalizer: sort of look at the end of the script | 16:05 |
| openstackgerrit | Ken Dreyer proposed openstack/python-jenkins: add get_whoami() https://review.openstack.org/307896 | 16:05 |
| clarkb | nibalizer: ya you cant do that we already argparse | 16:05 |
| nibalizer | oh ok i better fix that up then | 16:05 |
| clarkb | I would comment on change but it just crashed firefox mobile | 16:06 |
| *** matrohon has quit IRC | 16:06 | |
| mordred | of course it did | 16:06 |
| jeblair | clarkb: i got you covered | 16:06 |
| fungi | firefox immobile | 16:06 |
| clarkb | mordred I am trying ti use firefox mobile because not google and ublock origin | 16:06 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Enable an environment aware launch node script https://review.openstack.org/307894 | 16:06 |
| clarkb | but then gerrit crashes it :( | 16:06 |
| mordred | yay | 16:06 |
| nibalizer | clarkb: reup'd | 16:07 |
| mordred | so - in case anyone is curious watching to see something _not_ happen is boring | 16:07 |
| *** Sukhdev has joined #openstack-infra | 16:07 | |
| *** asettle has joined #openstack-infra | 16:07 | |
| clarkb | I should make a concrete list of ansible improvements to try and get done today | 16:08 |
| anteaya | mordred: requires much more focus than watching something happen | 16:08 |
| mordred | jeblair: if you have a sec, https://review.openstack.org/#/q/topic:shade-nodepool is ready for eyeballs | 16:08 |
| clarkb | will get that dine and share so others can add/update | 16:08 |
| *** Yarboa has quit IRC | 16:08 | |
| mordred | anteaya: yup | 16:08 |
| mordred | anteaya: well, also you don't know if it just hasn't happened yet, of if you've done a thing | 16:08 |
| fungi | clarkb: on the etherpad for the summit work session presumably | 16:08 |
| *** doug-fish has quit IRC | 16:09 | |
| *** nwkarsten has joined #openstack-infra | 16:09 | |
| *** nwkarste_ has quit IRC | 16:09 | |
| jeblair | nibalizer: almost -- missed some plumbing | 16:09 |
| jeblair | mordred: ack | 16:09 |
| clarkb | fungi: ya | 16:10 |
| jeblair | mordred: i saw that and went "WAH!" then added "status:open" to that and it's better. | 16:10 |
| mordred | jeblair: uhm | 16:10 |
| mordred | jeblair: so | 16:10 |
| mordred | jeblair: the leak may be even stranger than we might have thought | 16:10 |
| anteaya | mordred: yes | 16:10 |
| mordred | I now show 3 leaked osic fips | 16:10 |
| mordred | BUT | 16:10 |
| *** dtantsur is now known as dtantsur|brb | 16:10 | |
| mordred | I do not show any logging that they were ever even attempted to be deleted | 16:11 |
| jeblair | mordred: first thought -- could they be related to subnodes? | 16:11 |
| *** ayoung has joined #openstack-infra | 16:11 | |
| mordred | jeblair: maybe? | 16:11 |
| jeblair | mordred: can you etherpad all the info you have about those 3? | 16:11 |
| fungi | i've wondered for a while if some of our fip leaking might be around subnode lifecycle codepaths | 16:11 |
| *** yamahata has joined #openstack-infra | 16:11 | |
| clarkb | mordred: if you grep the ip you get the node id and if you grep that you will know if it is a subnode | 16:12 |
| *** stewie925 has joined #openstack-infra | 16:12 | |
| *** asettle has quit IRC | 16:12 | |
| jeblair | start with the etherpad so i can help | 16:12 |
| mordred | https://etherpad.openstack.org/p/vGZsfXNbp6 | 16:12 |
| jeblair | 2016-04-19 15:48:04,231 DEBUG shade: Need to delete fip:172.99.106.35:9b92bb94-7595-4790-85d5-5c05e1e5e202 for server a867850e-d112-4de9-ad80-6f85fbfd6235 | 16:14 |
| *** cfarquhar has quit IRC | 16:14 | |
| jeblair | mordred: you know that's not the same instance of it because the id (9b92) is different (!=41d7) right? | 16:14 |
| jeblair | mordred: (i'm just verifying my understanding of that log msg) | 16:14 |
| mordred | ah. good point | 16:14 |
| *** sdake has joined #openstack-infra | 16:14 | |
| *** kushal has joined #openstack-infra | 16:14 | |
| anteaya | yolanda: might you know Jean-Daniel's irc nick? when you said you were chatting with that person on irc, I take it that conversation didn't happen in this channel? | 16:14 |
| mordred | yah. so searching for the IP here is not connecting us with the server necessarily | 16:15 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add openstack-health RSS proxy to api server https://review.openstack.org/307905 | 16:15 |
| mtreinish | fungi: ^^^ that's my first pass at that. But I'm sure I screwed up the regex (I always screw up the regex) | 16:15 |
| jeblair | mordred: yeah, that fip was used on at least 7 servers in the past few hours | 16:16 |
| mordred | jeblair: so I'm going to take those lines out of the etherpad, because they're not relevant | 16:16 |
| *** stevelle has left #openstack-infra | 16:16 | |
| jeblair | mordred: i don't see 41d76dc7-92aa-4e61-8351-3b3bd005aee2 in the debug log | 16:16 |
| mordred | jeblair: I don't either | 16:16 |
| *** samueldmq has quit IRC | 16:17 | |
| jeblair | mordred: if we had that, we could establish a positive connection with a server; however, it seems likely that it's related to the last server that used it (on account of it's a leak, after all) | 16:17 |
| jeblair | mordred: however, the last mention of the ip is with a different id | 16:17 |
| mordred | yah - I tink I'd like to add a log message at fip creation time - we should have a server id when we're creating it in shade | 16:18 |
| jeblair | mordred: so it seems like perhaps the ip was somehow related to a server that never got as far as nodepool knowing what the ip is? | 16:18 |
| mordred | yah | 16:18 |
| mordred | that's what I'm thinking | 16:18 |
| mordred | oh! you know what ... | 16:18 |
| *** kushal has quit IRC | 16:19 | |
| *** tphummel has quit IRC | 16:19 | |
| *** yamamoto has quit IRC | 16:19 | |
| nibalizer | jeblair: thanks, will fix | 16:19 |
| *** cfarquhar has joined #openstack-infra | 16:19 | |
| *** cfarquhar has quit IRC | 16:19 | |
| *** cfarquhar has joined #openstack-infra | 16:19 | |
| *** zaro has quit IRC | 16:20 | |
| *** nwkarsten has quit IRC | 16:20 | |
| *** nwkarsten has joined #openstack-infra | 16:20 | |
| *** anteaya has quit IRC | 16:20 | |
| *** ifarkas has quit IRC | 16:20 | |
| jeblair | mordred: i think that's a good idea. i believe nodepool does learn the server id before fip creation/attachment (between create server and 'waiting for server'). so we should be able to track fips back to nodes with that. | 16:22 |
| openstackgerrit | Merged openstack-infra/storyboard: Allow filtering of worklists and boards by story or task https://review.openstack.org/288910 | 16:22 |
| mordred | Shrews: what the hell were we thinking on line 4092 ? | 16:22 |
| Shrews | eh? | 16:23 |
| fungi | you know, line 4092 | 16:24 |
| jeblair | get_container? | 16:24 |
| mordred | Shrews: at the end of _delete_server in the if has_sevice(volume) block | 16:24 |
| mordred | Shrews: just look at it and be sad | 16:24 |
| *** sdake_ has joined #openstack-infra | 16:25 | |
| *** doug-fish has joined #openstack-infra | 16:27 | |
| * jeblair looks at it and is confused | 16:27 | |
| *** mylu has quit IRC | 16:27 | |
| *** imcsk8|zZz is now known as imcsk8 | 16:27 | |
| jeblair | but then again, i'm looking at line 4064, so | 16:27 |
| *** yamahata has quit IRC | 16:27 | |
| *** mylu has joined #openstack-infra | 16:27 | |
| *** sridhar_ram has joined #openstack-infra | 16:28 | |
| *** yamahata has joined #openstack-infra | 16:28 | |
| *** aimeeU has joined #openstack-infra | 16:28 | |
| clarkb | https://etherpad.openstack.org/p/newton-infra-robustify-ansible-puppet thoughts written down there with two lists of issues | 16:29 |
| *** piet has quit IRC | 16:29 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 16:29 |
| *** stewie925 has quit IRC | 16:29 | |
| clarkb | mordred: nibalizer jeblair fungi ^ you may all have feedback on that etherpad before we sit down for the session | 16:29 |
| *** sdake has quit IRC | 16:29 | |
| pabelanger | and back. Catching up on backscroll | 16:30 |
| Shrews | mordred: that code seems to have snuck in while adding some functional tests. /me will not mention the author | 16:30 |
| *** piet has joined #openstack-infra | 16:31 | |
| *** johnny has joined #openstack-infra | 16:31 | |
| mordred | Shrews: I'm sure that the author had good intentions | 16:31 |
| Shrews | mordred: and yes, i am sad | 16:31 |
| *** johnny is now known as Guest74326 | 16:31 | |
| *** nwkarsten has quit IRC | 16:31 | |
| *** nwkarsten has joined #openstack-infra | 16:31 | |
| mordred | Shrews: I mean - all three lines are insane | 16:32 |
| mordred | Shrews: that has to be a rebase fail | 16:32 |
| mordred | Shrews: what does get_volume(server) even mean? | 16:33 |
| Shrews | mordred: https://review.openstack.org/251150 | 16:33 |
| jeblair | mordred: redarging 307875 -- i think there's some extra complexity | 16:33 |
| mordred | jeblair: yah? | 16:33 |
| *** mylu has quit IRC | 16:33 | |
| jeblair | mordred: nodepool expects deleteServer to be fast and then calls waitForServerDeletion when it wants to wait | 16:33 |
| jeblair | mordred: often times these calls are in immediate series, but not always | 16:34 |
| mordred | jeblair: ok. I see that | 16:34 |
| jeblair | mordred: for instance, when it wants to cause all of the subnodes to be deleted. it starts the process for all of them in parallel, then waits for them all in series | 16:34 |
| mordred | jeblair: I think we can do what we did for create_server | 16:34 |
| mordred | where we break the wait-for into a function that can be separately called | 16:34 |
| *** gyee has quit IRC | 16:34 | |
| mordred | _however_ | 16:34 |
| *** esikachev has quit IRC | 16:34 | |
| mordred | actually, no, I don't think we can | 16:35 |
| *** IlyaG has joined #openstack-infra | 16:35 | |
| *** IlyaG has quit IRC | 16:35 | |
| jeblair | mordred: (there already is one) | 16:35 |
| mordred | yah - I meant in shade | 16:35 |
| *** IlyaG has joined #openstack-infra | 16:35 | |
| *** EricGonczer_ has quit IRC | 16:36 | |
| Shrews | mordred: we can repurpose wait_for_server() for that | 16:36 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Add new virtualbmc project under ironic's umbrella https://review.openstack.org/307323 | 16:36 |
| Shrews | now it waits for active, but we could add a delete option | 16:36 |
| *** andymaier has joined #openstack-infra | 16:36 | |
| Shrews | or a general status | 16:36 |
| mordred | Shrews: we could - but the problem is, submitting an async delete server task is not generally meaningful | 16:37 |
| *** anteaya has joined #openstack-infra | 16:37 | |
| mordred | beacuse we can't actually submit the delete call until the fips are gone | 16:37 |
| jeblair | right | 16:37 |
| *** ddieterly is now known as ddieterly[away] | 16:37 | |
| *** IlyaG has quit IRC | 16:37 | |
| mordred | we might have to generate a fake shade-side thing to poll on that embodies the sequence of fips-then-servers | 16:37 |
| jeblair | nodepool used to do this itself in its deleteserver method; i'm having trouble figuring a way to get the desired behavior without going back to that | 16:37 |
| *** gongysh has quit IRC | 16:38 | |
| * mordred goes away to think for a sec | 16:38 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Add openstack-health RSS proxy to api server https://review.openstack.org/307905 | 16:38 |
| *** jistr has quit IRC | 16:39 | |
| *** jlanoux has quit IRC | 16:40 | |
| Shrews | mordred: i keep going back to my fip cache idea | 16:40 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Add a function to decide if using cache is appropriate https://review.openstack.org/306524 | 16:40 |
| *** tphummel has joined #openstack-infra | 16:42 | |
| *** nwkarste_ has joined #openstack-infra | 16:42 | |
| *** nwkarsten has quit IRC | 16:43 | |
| *** tosky_ has joined #openstack-infra | 16:43 | |
| anteaya | fungi: did you want to populate the infra section of https://wiki.openstack.org/wiki/Design_Summit/Newton/Etherpads | 16:43 |
| rcarrillocruz | Shrews, mordred : addressed comments on https://review.openstack.org/#/c/302770/ | 16:43 |
| *** ddieterly[away] is now known as ddieterly | 16:43 | |
| rcarrillocruz | rebasing the recordsets change based on ^ (i put zones change as dependency) | 16:44 |
| rodrigods | anyone available to take a look at https://review.openstack.org/#/c/298696/ ? :) It already has a +2 | 16:44 |
| anteaya | fungi: or do you want us to do it ourselves? I'm fine either way, just curious what you would like | 16:44 |
| ayoung | rodrigods, you were looking at https://review.openstack.org/#/c/298696/ | 16:44 |
| ayoung | ah..right | 16:44 |
| *** nwkarste_ has quit IRC | 16:45 | |
| *** fawadkhaliq has joined #openstack-infra | 16:45 | |
| *** tosky has quit IRC | 16:45 | |
| ayoung | rodrigods, what is that holding up? | 16:45 |
| fungi | anteaya: i was planning to, just hadn't gotten to it yet | 16:45 |
| *** nwkarsten has joined #openstack-infra | 16:45 | |
| fungi | i'll do that now before i move on to meeting prep | 16:45 |
| rodrigods | ayoung, tests reviews, starting at https://review.openstack.org/#/c/302299/ | 16:45 |
| *** cfarquhar has quit IRC | 16:45 | |
| anteaya | fungi: cool, thanks | 16:45 |
| rodrigods | ppl need to know if they are passing before reviewing | 16:45 |
| ayoung | rodrigods, ah right | 16:45 |
| *** sshnaidm has joined #openstack-infra | 16:46 | |
| ayoung | rodrigods, that is only running on Keystone checkins, right? | 16:46 |
| rodrigods | ayoung, right | 16:46 |
| rodrigods | ayoung, it should run "tempest -e all-plugin -- keysotne" | 16:46 |
| rodrigods | and the job is only added to keystone too (non-voting) | 16:47 |
| *** cfarquhar has joined #openstack-infra | 16:47 | |
| *** cfarquhar has quit IRC | 16:47 | |
| *** cfarquhar has joined #openstack-infra | 16:47 | |
| ayoung | fungi, can you kick off https://review.openstack.org/#/c/302299/ please? It will let Keystone drive on with functional testing. Which will make the world a better place. | 16:47 |
| *** salv-orlando has joined #openstack-infra | 16:48 | |
| *** sputnik13 has joined #openstack-infra | 16:48 | |
| fungi | ayoung: not sure what you mean by kick off | 16:48 |
| rodrigods | fungi, +3 i guess :) | 16:49 |
| rodrigods | after proper review, of course | 16:49 |
| fungi | i'm not a keystone core reviewer | 16:49 |
| ayoung | fungi, nah, this is an infra | 16:49 |
| fungi | though i can try to take a look at the change anyway if you need | 16:49 |
| rodrigods | fungi, https://review.openstack.org/#/c/298696/ | 16:49 |
| fungi | it says the change is for openstack/keystone | 16:49 |
| *** Nikolay_St has joined #openstack-infra | 16:49 | |
| rodrigods | correct URL ^ | 16:50 |
| *** nwkarsten has quit IRC | 16:50 | |
| ayoung | sorry wrong linkwe need to | 16:50 |
| fungi | rodrigods: thanks, that's totally not what ayoung linked ;) | 16:50 |
| * fungi is marginally less confused now | 16:50 | |
| ayoung | fungi, I was one too far in my tab list | 16:50 |
| *** EricGonczer_ has joined #openstack-infra | 16:50 | |
| ayoung | fungi, and if ever you need to chose between believing me and rodrigods you know which way to lean | 16:50 |
| rodrigods | lol | 16:51 |
| * fungi is a believer | 16:51 | |
| *** salv-orlando has quit IRC | 16:51 | |
| * ayoung is a dreamer | 16:51 | |
| *** ivc_ has joined #openstack-infra | 16:51 | |
| * rodrigods not a lier | 16:51 | |
| ayoung | fungi, anyway, Keystone discussed this in our meeting a couple times...we really want to head this way. | 16:52 |
| ayoung | hence two +1s from Keystone core | 16:52 |
| * anteaya queues up the monkees | 16:52 | |
| * ayoung herds them onto the last train to Clarksville. | 16:52 | |
| openstackgerrit | Derek Higgins proposed openstack-infra/tripleo-ci: Use the cached ironic-python-agent images https://review.openstack.org/301699 | 16:53 |
| *** oanson has quit IRC | 16:53 | |
| rodrigods | yay o/ | 16:54 |
| openstackgerrit | Ben Nemec proposed openstack-infra/tripleo-ci: Don't use cached undercloud image for element repos either https://review.openstack.org/307923 | 16:55 |
| *** hashar has quit IRC | 16:57 | |
| *** oanson has joined #openstack-infra | 17:00 | |
| *** brucet has joined #openstack-infra | 17:00 | |
| mpaolino | clarkb, after some investigations I have a project in the launchpad. It is still not clear to me how to use git review with this (the git "gerrit" remote does not exist) | 17:01 |
| clarkb | mpaolino: git review talks ot gerrit and has nothing to do with launchpad | 17:01 |
| *** derekh has quit IRC | 17:01 | |
| clarkb | mpaolino: the link I shared walks you through the process of getting a project in gerrit | 17:01 |
| openstackgerrit | Merged openstack-infra/project-config: Enable non-voting keystone tempest plugin tests https://review.openstack.org/298696 | 17:01 |
| *** jrmyjohn has quit IRC | 17:02 | |
| *** dprince has joined #openstack-infra | 17:02 | |
| *** rossella_s has quit IRC | 17:03 | |
| *** rcernin has quit IRC | 17:04 | |
| *** rossella_s has joined #openstack-infra | 17:04 | |
| mpaolino | clarkb, set up launchpad is part of the link you shared | 17:04 |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Show a modal when editing worklists in a board https://review.openstack.org/306400 | 17:05 |
| mpaolino | in any case I completely agree | 17:05 |
| openstackgerrit | Merged openstack-infra/project-config: cinderclient: Move pylint to non-voting check https://review.openstack.org/307818 | 17:05 |
| *** dtantsur|brb is now known as dtantsur | 17:06 | |
| mpaolino | however I don't see the gerrit/projects.yaml file (as that guide suggest) in any of the devstack plugins of the plugin registry.... thus I am wondering what's wrong :) | 17:07 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Add tarball jobs for all Puppet OpenStack modules https://review.openstack.org/307931 | 17:07 |
| *** mriedem has quit IRC | 17:08 | |
| *** tqtran has joined #openstack-infra | 17:08 | |
| *** ddieterly is now known as ddieterly[away] | 17:09 | |
| openstackgerrit | Merged openstack-infra/project-config: Implement higher job timeout for Ansible role jobs https://review.openstack.org/306573 | 17:10 |
| *** ddieterly[away] is now known as ddieterly | 17:10 | |
| *** annegentle has joined #openstack-infra | 17:10 | |
| *** vgridnev has joined #openstack-infra | 17:11 | |
| anteaya | mpaolino: are you already a contributor to openstack? are you familiar with this document? http://docs.openstack.org/infra/manual/developers.html | 17:12 |
| mpaolino | yes, I already did this in the past (not recently) | 17:13 |
| mpaolino | of course I have an account and signed the UCLA | 17:13 |
| anteaya | okay so then you know that gerrit is our canonical git repository for all our repos | 17:13 |
| mpaolino | ICLA | 17:13 |
| fungi | i hear they have a great basketball team | 17:13 |
| mpaolino | however the point is that I don't have a git clone something to do | 17:13 |
| *** infra-red has joined #openstack-infra | 17:14 | |
| mpaolino | fungi, UCLA for sure | 17:14 |
| anteaya | mpaolino: well if you recall your use fo gerrit, gerrit is our canonical git reposititory | 17:14 |
| anteaya | mpaolino: so the first thing you need to do is have your project on our gerrit | 17:14 |
| *** mriedem has joined #openstack-infra | 17:14 | |
| anteaya | mpaolino: which the link clarkb shared with you earlier instructs you in how to do | 17:15 |
| rodrigods | fungi, after merged, a new job starts working after... | 17:15 |
| rodrigods | just to figure out the whole process :) | 17:15 |
| AJaeger | mordred, zigo, fungi: Regarding deb-packing, I think adding one repository per package is the wrong approach. Adding another 300 repos to our already 1000 packages does not really scale. | 17:15 |
| *** Sukhdev has quit IRC | 17:15 | |
| zigo | AJaeger: I don't see how we could do otherwise ! | 17:15 |
| anteaya | AJaeger: I haven't been following the conversation, but I agree with your conclusion | 17:15 |
| fungi | rodrigods: automation runs to set up the new job configuraiton on all our jenkins masters and to reload zuul with the corresponding layout change | 17:16 |
| anteaya | having multiples of repos for every one repo is not a great model | 17:16 |
| fungi | rodrigods: probably ~30 minutes but can vary depending on confitions | 17:16 |
| zigo | anteaya: You missread. One repo per package. | 17:16 |
| rodrigods | fungi, hmm got it | 17:16 |
| rodrigods | thanks | 17:16 |
| anteaya | zigo: would that amount to another 300 repos for our already 1000 packages? | 17:17 |
| anteaya | as AJaeger states | 17:17 |
| fungi | on the one hand i agree adding another 300 repos for debian packages (and 300 for red hat packages? and 300 for gentoo packages? and...) is unsettling, i'd want to know what the alternative proposals are | 17:17 |
| zigo | anteaya: https://qa.debian.org/developer.php?login=openstack-devel@lists.alioth.debian.org <--- That's how many packages I maintain for OpenStack in Debian, though lots of these libraries, I wont maintain them within infra: only OpenStack upstream stuff, which isn't 371 packages (at least more than 100 are general purpose Python modules). | 17:19 |
| zigo | My current estimate is maybe 150 packages to begin with, and then it may grow a little bit. | 17:19 |
| zigo | I agree that's a lot, but like fungi wrote: what is the alternative? | 17:19 |
| anteaya | okay 150 | 17:19 |
| *** yamamoto has joined #openstack-infra | 17:20 | |
| anteaya | I think it is worth discussion, the alternative that is | 17:20 |
| pleia2 | for the config mgmt stuff we already do that, 130 for puppet | 17:20 |
| anteaya | pleia2: yes I know | 17:20 |
| *** sambetts is now known as sambetts|afk | 17:20 | |
| anteaya | puppet has their 100+, fuel has their 100+, etc | 17:20 |
| *** annegentle has quit IRC | 17:20 | |
| zigo | anteaya: The point I'm trying to make is that there's no alternative that I know about. | 17:20 |
| AJaeger | zigo, last time you said you wanted to add 300 more repositories. | 17:20 |
| AJaeger | zigo: but even 150 is a lot. | 17:21 |
| anteaya | fungi: thanks for setting up the etherpads on the wiki page | 17:21 |
| zigo | AJaeger: I'm not really sure yet, sorry. | 17:21 |
| anteaya | zigo: right, I get that | 17:21 |
| fungi | it's a matrix expansion of projects x deployment methodologies roughly | 17:21 |
| anteaya | fungi: yes | 17:21 |
| AJaeger | zigo: RPM packaging does not need that many, they have one where they store all spec files. | 17:21 |
| anteaya | and I think having a chat about the matrix expansion is worthwhile | 17:21 |
| *** infra-re_ has joined #openstack-infra | 17:21 | |
| zigo | AJaeger: Well, as you know, Debian packaging doesn't work this way (ie: we don't have a single .spec file to design...). | 17:22 |
| AJaeger | zigo, still, you can use a directory... | 17:22 |
| zigo | AJaeger: That's now how git-buildpackage works. | 17:22 |
| zigo | s/now/not/ | 17:22 |
| AJaeger | zigo: and rpm can use more than a spec file, just wanted to point out that they managed to do this without creating lots of new repos | 17:22 |
| AJaeger | zigo: then let's fix it... | 17:23 |
| *** annegentle has joined #openstack-infra | 17:23 | |
| *** infra-red has quit IRC | 17:23 | |
| zigo | AJaeger: We're really not there yet... | 17:23 |
| zigo | AJaeger: https://etherpad.openstack.org/p/newton-deb-packaging | 17:23 |
| zigo | That's the etherpad for Austin. | 17:24 |
| zigo | AJaeger: I'm currently investigating mini-dinstall and dak. | 17:24 |
| anteaya | have we set aside time for the matrix expansion of projects x deployment methodologies at summit yet? | 17:24 |
| *** salv-orlando has joined #openstack-infra | 17:24 | |
| zigo | I'm slowly leaning toward using dak... :P | 17:24 |
| AJaeger | zigo: and that's why I'm not going to review deb-spice-html5 - a package that is IMHO completely unrelated to OpenStack and where I first like to see a proper plan - so, let's wait until Austin and then write up a spec. | 17:25 |
| fungi | much of debian's packaging (not just for openstack) is maintained with a separate git repo to contain the packaging metadata/scripts/patches for each project being packaged, so while this may run counter to how rpm package maintenance is done, it would be a significant divergence within debian to have one git repo covering lots of packages | 17:25 |
| *** annegentle has quit IRC | 17:25 | |
| zigo | AJaeger: The point of spice-html5 is that it has no dependency, which is why I chose it. | 17:25 |
| zigo | AJaeger: If I take something like oslo.messaging, it will require half of all of OpenStack to even start building. | 17:25 |
| zigo | AJaeger: The point is not spice-html5, which I may maintain in Debian only. | 17:26 |
| zigo | AJaeger: I can search for a better toy to play with, more related to OpenStack, if you prefer. | 17:26 |
| *** zz_ja is now known as zz_zz_ja | 17:26 | |
| clarkb | mordred: jeblair Shrews is there a tl;dr on the nodepool + shade + fip plan? I think I saw that the current proposed and running patches are working but they make nodepool less async and serialize deletes because of the waits? | 17:26 |
| clarkb | are we planning on sticking with that? | 17:26 |
| mordred | clarkb: I'm about to give a tl;dr | 17:26 |
| mordred | have a new idea | 17:27 |
| mordred | but brb | 17:27 |
| zigo | AJaeger: Plus, I 100% agree with what fungi just wrote. | 17:27 |
| clarkb | ok | 17:27 |
| clarkb | fungi: and you mentioned that rax was looking into the networking stuff more ? | 17:27 |
| zigo | I wouldn't have write it better myself. | 17:27 |
| AJaeger | fungi, that's really bad - especially if zigo needs to package all dependencies. | 17:27 |
| openstackgerrit | Spencer Krum proposed openstack-infra/system-config: Enable an environment aware launch node script https://review.openstack.org/307894 | 17:27 |
| * clarkb wants to have system running well before we all disappear for a week | 17:27 | |
| *** ddieterly is now known as ddieterly[away] | 17:27 | |
| AJaeger | With fuel and charm we have already a lot - but those don't need a repo for each and every library | 17:27 |
| *** annegentle has joined #openstack-infra | 17:28 | |
| *** yamamoto has quit IRC | 17:28 | |
| fungi | AJaeger: yes, i agree that hosting packaging for a bunch of openstack dependencies was unforeseen | 17:28 |
| AJaeger | fungi, zigo : Why do we need this at all? | 17:28 |
| *** fhubik has quit IRC | 17:28 | |
| AJaeger | Debian has its own way of building packages, I don't get that we need to replicate that one as well. | 17:28 |
| *** dimtruck is now known as zz_dimtruck | 17:29 | |
| fungi | AJaeger: version control for the tree of files (metadata, scripts, occasional out-of-tree patches) used to build deb packages | 17:29 |
| *** ramishra has quit IRC | 17:29 | |
| AJaeger | fungi, plus also building and publishing them? | 17:30 |
| *** ramishra has joined #openstack-infra | 17:30 | |
| *** oanson has quit IRC | 17:31 | |
| fungi | AJaeger: zigo's goal is to maintain that data under version control, and test that proposed changes to it work (successfully build packages, install cleanly, pass validation tests once installed, et cetera) | 17:31 |
| *** mrmartin has joined #openstack-infra | 17:31 | |
| *** flepied has quit IRC | 17:31 | |
| *** weshay has joined #openstack-infra | 17:32 | |
| nibalizer | jeblair: are you going to be piping asterisk into some of the rooms this time? | 17:32 |
| zigo | fungi: AJaeger: And at the end, test that the package doesn't break a Tempest run too. | 17:32 |
| fungi | and also it can be cross-tested (for example, periodic pipeline jobs) against master branch state of the projects being packaged so that the maintainers can be more reactive to inevitable bitrot in their packaging | 17:32 |
| zigo | (final, ultimate goal, to make sure nothing ever breaks) | 17:32 |
| nibalizer | sounds like Hunner won't be able to make it for the ansible-puppet hardening session, but it would be great to conference him in | 17:32 |
| * zigo goes for dinner | 17:33 | |
| fungi | we could have Hunner-in-a-box next week! | 17:33 |
| * Hunner jumps out of the box | 17:34 | |
| * Clint shrieks | 17:35 | |
| anteaya | pleia2 pabelanger I have added some links to the wiki etherpad for next week: https://etherpad.openstack.org/p/newton-infra-wiki-upgrades | 17:35 |
| anteaya | do add your notes | 17:35 |
| AJaeger | fungi, zigo: I'm still scared by 300+ repos (even 150), especially when I see request for packaging of libraries that are needed but outside of OpenStack ecosystem like html-spice | 17:35 |
| pleia2 | anteaya: I would really like to see all this work in a story | 17:35 |
| pleia2 | but an etherpad for a session is a fine start | 17:35 |
| anteaya | okay thanks | 17:36 |
| anteaya | and I have no objection to it becoming a story | 17:36 |
| anteaya | pleia2: do you want to add your vision to the etherpad? | 17:36 |
| anteaya | AJaeger: I agree with your trepidation | 17:36 |
| jeblair | nibalizer: i have not had a chance to work on that more than what i had at infra-midcycle, but i could bring that hardware again -- it might be useful to try it out hooked up to the room av system | 17:37 |
| *** gokrokve has quit IRC | 17:37 | |
| nibalizer | ok cool | 17:37 |
| jeblair | nibalizer: (we can also just do it with a laptop which is what i did for clark-in-a-box) | 17:37 |
| pleia2 | anteaya: working on it :) | 17:37 |
| *** gokrokve has joined #openstack-infra | 17:37 | |
| nibalizer | i've been in the box, clark has been in the box, monty has been in the box | 17:37 |
| nibalizer | paul was in the box for tokyo | 17:37 |
| anteaya | pleia2: lovely addition so far, thank you | 17:38 |
| *** annegentle has quit IRC | 17:38 | |
| nibalizer | jeblair: i also have a suspicion we can con bkero into being a box-weilder | 17:38 |
| fungi | AJaeger: i agree. i'm not fully endorsing the idea, simply explaining | 17:38 |
| nibalizer | he often caries the right gear and fills that role in other situations | 17:38 |
| anteaya | fungi: my only conflict with the infra schedule seems to be the nova ci session at the same time as the new landing page session | 17:39 |
| *** gyee has joined #openstack-infra | 17:39 | |
| fungi | AJaeger: it's still worth deeper conversation as to whether and how we draw the line as a team as to what we're willing to review | 17:39 |
| anteaya | fungi: so I think I will have to miss the new landing page | 17:39 |
| anteaya | alas I feel I will be the only infra in the nova ci session | 17:39 |
| fungi | anteaya: yep, that's why i noted it in the announcement i sent. so people can plan ahead to figure out if there are others who can proxy their positions in some of those sessions | 17:39 |
| anteaya | which I think kind of is less than what jay was hoping for according to his mailing list posts | 17:40 |
| anteaya | yup, I didn't say anything at the time as I knew switching anything wouldn't help | 17:40 |
| AJaeger | fungi: sure, we need to discuss this. I wanted to explain the problems I see and why I'm not reviewing right now any further repos for this effort... | 17:40 |
| fungi | hopefully we can get some more infra people to go to that. the wiki landing page work session shouldn't be super exciting | 17:40 |
| anteaya | if I have to miss one of our sessions, this really is the only one I am willing to miss | 17:40 |
| anteaya | well it would be great to have more infra folk in the nova ci session | 17:41 |
| fungi | rather, non-wiki landing page | 17:41 |
| anteaya | as I think jay wants someone willing to say they will help put his idea into effect | 17:41 |
| fungi | i expect it will be a fairly cookie-cutter clone of what we're doing with governance, releases, et al | 17:41 |
| anteaya | and I can't do more than I am already doing regarding third party ci | 17:41 |
| bkero | nibalizer: box? | 17:41 |
| bkero | i can do hardware | 17:42 |
| nibalizer | bkero: jeblair often sets up his laptop with an external mic and external speakers | 17:42 |
| anteaya | so anyway those are my thoughts | 17:42 |
| nibalizer | then hooks it up to asterisk | 17:42 |
| bkero | Ah | 17:43 |
| nibalizer | as a way to bridge people in to meetings who couldn't be there | 17:43 |
| fungi | though i gather he's gotten some recentish pi boards to try implementing on | 17:43 |
| bkero | right right | 17:43 |
| bkero | Lots of opportunities for embedded boards | 17:43 |
| fungi | basically sip client with jackd filter chain to keep stuff audible | 17:43 |
| openstackgerrit | Merged openstack-infra/project-config: Revert split into 2 shell scripts https://review.openstack.org/307654 | 17:44 |
| fungi | if redone on an sbc, it's something we could in theory scatter hither and yon in session rooms for small-scale remote participation | 17:44 |
| bkero | Yeah. Can pi pretty easily | 17:44 |
| *** electrofelix has quit IRC | 17:44 | |
| bkero | set up sd cards with unique creds on each, have them autoconnect to asterisk conf calls | 17:45 |
| persia | There are a number of asterisk-compatible SIP speakerphones available commercially, which may avoid some integration work | 17:45 |
| fungi | mmm... integration work is the fun part! ;) | 17:45 |
| persia | Admittedly :) | 17:46 |
| *** oanson has joined #openstack-infra | 17:46 | |
| fungi | free software toolchains, open hardware design... | 17:46 |
| *** nwkarsten has joined #openstack-infra | 17:46 | |
| bkero | I imagine a lot of them grab conf from network | 17:46 |
| bkero | which might not be possible over greater conf network | 17:46 |
| pleia2 | anteaya: did you see pabelanger's comment about ffmpeg on https://review.openstack.org/#/c/285405/ ? | 17:46 |
| pleia2 | anteaya: ah, you mention that later | 17:47 |
| anteaya | yes, thanks | 17:48 |
| *** nwkarsten has quit IRC | 17:50 | |
| *** tosky_ has quit IRC | 17:52 | |
| *** zz_dimtruck is now known as dimtruck | 17:54 | |
| *** nadya has joined #openstack-infra | 17:55 | |
| pabelanger | pleia2: I did not, looking now | 17:58 |
| pabelanger | Oh | 17:58 |
| pabelanger | nm :) | 17:58 |
| pabelanger | anteaya: checking now | 17:59 |
| *** kzaitsev_mb has quit IRC | 17:59 | |
| *** Sukhdev has joined #openstack-infra | 18:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Retry floating ip deletion before deleting server https://review.openstack.org/307439 | 18:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Don't look for invalid image keys for volume invalidation https://review.openstack.org/307959 | 18:00 |
| *** doug-fis_ has joined #openstack-infra | 18:00 | |
| * AJaeger needs to leave urgently... | 18:01 | |
| *** piet has quit IRC | 18:01 | |
| anteaya | AJaeger: enjoy the rest of your day | 18:01 |
| anteaya | AJaeger: thank you | 18:01 |
| *** doug-fi__ has joined #openstack-infra | 18:01 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool: Cleanup leaked floating ips periodically https://review.openstack.org/307755 | 18:01 |
| *** mgould has quit IRC | 18:02 | |
| *** jrmyjohn has joined #openstack-infra | 18:03 | |
| *** mtreinish has quit IRC | 18:03 | |
| *** brucet has quit IRC | 18:03 | |
| pabelanger | anteaya: pleia2: Thanks for the link. I added a question about checking if there is an existing community puppet-mediawiki module. Maybe there is a newer solution to our existing module. I haven't looked at anything but something to maybe ask around about. I suspect there isn't however | 18:03 |
| *** doug-fish has quit IRC | 18:04 | |
| *** doug-fis_ has quit IRC | 18:04 | |
| anteaya | pabelanger: I responsed in the etherpad | 18:05 |
| *** kzaitsev_mb has joined #openstack-infra | 18:05 | |
| pabelanger | anteaya: great, thanks | 18:05 |
| anteaya | thank you | 18:06 |
| *** jrmyjohn has quit IRC | 18:07 | |
| *** mtreinish has joined #openstack-infra | 18:08 | |
| *** lucasagomes is now known as lucas-dinner | 18:10 | |
| pabelanger | EmilienM: I don't see puppet-aodh tarball on tarball.o.o, was it successful? | 18:12 |
| EmilienM | pabelanger: not yet the patch can wait, I'll -1 it | 18:13 |
| EmilienM | pabelanger: in fact, we should have tarball once post queue is cleared | 18:13 |
| *** kzaitsev_mb has quit IRC | 18:13 | |
| openstackgerrit | Ihar Hrachyshka proposed openstack-infra/project-config: networking-*: allow neutron-release group to push tags https://review.openstack.org/307965 | 18:13 |
| EmilienM | pabelanger: but I'll -1 my patch in the meantime | 18:13 |
| pabelanger | EmilienM: Sure, np. I was mostly curious to see the log file for it | 18:14 |
| *** kzaitsev_mb has joined #openstack-infra | 18:14 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Add support to launch-node for cinder attach https://review.openstack.org/285477 | 18:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/system-config: Only use one volume for /opt https://review.openstack.org/307967 | 18:16 |
| *** asselin_ has quit IRC | 18:16 | |
| *** ihrachys has quit IRC | 18:17 | |
| *** oanson has quit IRC | 18:17 | |
| *** sabari has quit IRC | 18:17 | |
| *** mhayden has left #openstack-infra | 18:17 | |
| *** dtantsur is now known as dtantsur|afk | 18:18 | |
| *** nadya has quit IRC | 18:19 | |
| *** krtaylor has quit IRC | 18:20 | |
| *** Guest74326 has quit IRC | 18:21 | |
| *** pvaneck has joined #openstack-infra | 18:21 | |
| openstackgerrit | Morgan Fainberg proposed openstack-infra/gitdm: Add alias for morgan fainberg https://review.openstack.org/307973 | 18:22 |
| openstackgerrit | Dan Prince proposed openstack-infra/project-config: Create an os-net-config ACL https://review.openstack.org/307975 | 18:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Don't look for invalid image keys for volume invalidation https://review.openstack.org/307959 | 18:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 18:25 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Retry floating ip deletion before deleting server https://review.openstack.org/307439 | 18:25 |
| *** Nisha has joined #openstack-infra | 18:25 | |
| Nisha | hello | 18:25 |
| *** flepied has joined #openstack-infra | 18:25 | |
| anteaya | Nisha: how can we help? | 18:26 |
| *** nwkarsten has joined #openstack-infra | 18:26 | |
| Nisha | anteaya, i had one quick ques | 18:26 |
| anteaya | please ask | 18:26 |
| Nisha | i have enabled two gates in our CI | 18:26 |
| Nisha | which are triggered comment based right now | 18:26 |
| Nisha | when i add the comment to the patch to trigger the CI, it triggers only one gate | 18:27 |
| Nisha | the other gate didnt get triggered | 18:27 |
| anteaya | yes | 18:27 |
| *** ddieterly[away] has quit IRC | 18:28 | |
| anteaya | our tools are designed for the gates to be exclusive | 18:28 |
| anteaya | so one gate is triggered | 18:28 |
| *** rbrndt_ has quit IRC | 18:28 | |
| anteaya | if you want to change that functionality, you will need to look at the triggering mechanism | 18:28 |
| Nisha | means? if i want both the gates to be triggered for a patch, how can i achieve that | 18:28 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones https://review.openstack.org/302770 | 18:28 |
| fungi | i have a feeling there may be a terminology impedance mismatch here | 18:28 |
| clarkb | mordred: commented on https://review.openstack.org/#/c/307439/9 note it is an older patchset | 18:28 |
| mordred | clarkb: thanks | 18:29 |
| anteaya | fungi: go ahead | 18:29 |
| clarkb | mordred: there is one thing I think you will want to do for defensive measures | 18:29 |
| fungi | Nisha: when you say "i have enabled two gates in our CI" do you mean two jobs? or two pipelines? or two of something else? | 18:29 |
| Nisha | yes | 18:29 |
| Nisha | fungi, yes | 18:29 |
| *** krtaylor has joined #openstack-infra | 18:29 | |
| fungi | which of those | 18:29 |
| *** nwkarsten has quit IRC | 18:29 | |
| fungi | ? | 18:29 |
| fungi | that was an either/or/or | 18:30 |
| Nisha | fungi, check https://review.openstack.org/#/c/306929/2 | 18:30 |
| clarkb | mordred: comment should still be valid on latest patchset I just didn't copy pasta it | 18:30 |
| *** sabari has joined #openstack-infra | 18:30 | |
| Nisha | This shows HP Proliant CI gate | 18:30 |
| *** nwkarsten has joined #openstack-infra | 18:30 | |
| *** sridhar_ram is now known as sridhar_ram_afk | 18:30 | |
| *** EricGonczer_ has quit IRC | 18:30 | |
| Nisha | there is one more for another driver, which didnt get triggered | 18:30 |
| *** sridhar_ram_afk has quit IRC | 18:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Don't look for invalid image keys for volume invalidation https://review.openstack.org/307959 | 18:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 18:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Retry floating ip deletion before deleting server https://review.openstack.org/307439 | 18:30 |
| fungi | Nisha: looks like you have one job running, for gate-tempest-ironic-ilo-driver-iscsi_ilo | 18:30 |
| mordred | clarkb: good call with max | 18:30 |
| Nisha | we have one more | 18:31 |
| Nisha | but that didnt got triggered | 18:31 |
| fungi | Nisha: sure, i said "running" (or at least reporting back to the change) | 18:31 |
| fungi | Nisha: what does your zuul layout.yaml file look like? can you put that on http://paste.openstack.org/ and let us know the url? | 18:32 |
| Nisha | fungi, both the jobs are running...but i dont know why the other one didnt got triggered with the comment as this one got | 18:32 |
| Nisha | both are running from same jenkinsmaster | 18:32 |
| fungi | Nisha: it sounds like what you want to do is run two jobs and have them report back on the change. zuul is capable of doing that as long as it's configured correctly | 18:33 |
| anteaya | fungi: somewhere in there if you can get her to test her ci on ci-sandbox, that would be great | 18:33 |
| Nisha | fungi, yeah correct. | 18:33 |
| morgan | dhellmann: ping - have a question about an update for an external library and best practices | 18:33 |
| Nisha | ok let me recheck my zuul if i have gate based check there | 18:33 |
| dhellmann | morgan : pong | 18:33 |
| morgan | dhellmann: adding a new dependency (wrapt) for @positional, should that be a 1.1 release? or a 2.0? | 18:33 |
| morgan | dhellmann: that is the only change (and using wrapt) | 18:34 |
| dhellmann | 1.1 | 18:34 |
| morgan | dhellmann: i lean towards 1.1 | 18:34 |
| fungi | anteaya: well, i'm mostly just trying to get Nisha to stop referring to jobs as "gates" as a start, since that seems to be a point of confusion | 18:34 |
| morgan | dhellmann: cool. jusrt checking before i do the tagging and all | 18:34 |
| *** nwkarsten has quit IRC | 18:34 | |
| *** kzaitsev_mb has quit IRC | 18:34 | |
| *** prad has joined #openstack-infra | 18:34 | |
| Nisha | fungi, thanks :) | 18:34 |
| morgan | dhellmann: :) want to release this *before* my week is shot preparing for travel. | 18:34 |
| *** Nikolay_St has quit IRC | 18:35 | |
| dhellmann | morgan : dims and I are working on releases as patches pass the check queue | 18:35 |
| fungi | Nisha: at least from an upstream gerrit perspective, no third-party ci system gates any changes at all. they are purely advisory reporters on proposed changes | 18:35 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add support for Designate zones https://review.openstack.org/302770 | 18:35 |
| morgan | dhellmann: since positional isn't in openstack land... | 18:35 |
| Nisha | fungi, ok | 18:35 |
| morgan | dhellmann: and is run off my github instead, you have no requirement for it ;) | 18:35 |
| morgan | dhellmann: to help that is. | 18:36 |
| *** doug-fi__ has quit IRC | 18:36 | |
| *** degorenko is now known as _degorenko|afk | 18:36 | |
| anteaya | fungi: agreed, it is a good place to begin | 18:36 |
| morgan | dhellmann: it's such a small package... it didn't seem worth looping in infra. | 18:36 |
| dhellmann | morgan : ah, right, carry on then :-) | 18:36 |
| *** johnny has joined #openstack-infra | 18:36 | |
| *** ivc_ has quit IRC | 18:37 | |
| *** johnny is now known as Guest35267 | 18:37 | |
| fungi | anteaya: anyway, i didn't mean to take over your troubleshooting assistance with Nisha, i mostly wanted to understand what "two gates" meant because that makes no sense in our vocabulary | 18:37 |
| morgan | oh ... crap. i can't find my gpg key :( | 18:37 |
| *** pradk_ has quit IRC | 18:38 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 18:38 |
| Clint | check under the cushions | 18:38 |
| fungi | morgan: checked your other pants? | 18:38 |
| *** pradk has quit IRC | 18:38 | |
| *** Nisha has quit IRC | 18:38 | |
| morgan | fungi: yeah :( it clearly went through the wash | 18:38 |
| *** ociuhandu has quit IRC | 18:38 | |
| morgan | fungi: do gpg keys weather the washingmachine well? | 18:38 |
| morgan | :P | 18:38 |
| fungi | depends on the spin (electron joke!) | 18:39 |
| anteaya | fungi: no you made a good call, yes everyone calling everything a "gate" doesn't help newcomers | 18:39 |
| anteaya | fungi: and my soup is ready so thank you | 18:39 |
| fungi | soup! | 18:39 |
| fungi | well, i also seem to have driven Nisha away | 18:40 |
| anteaya | she comes and goes | 18:40 |
| anteaya | she'll be back | 18:40 |
| anteaya | I have invited her to third party meetings | 18:40 |
| fungi | that will likely help more than anything | 18:40 |
| anteaya | apparently things are bad enough yet for her to attend | 18:40 |
| anteaya | I agree | 18:40 |
| fungi | it's almost like a support group therapy session | 18:40 |
| anteaya | hope she decides to show up sometime | 18:40 |
| anteaya | very much so | 18:40 |
| anteaya | and can be effective for things like vocabulary | 18:41 |
| *** sdake_ has quit IRC | 18:41 | |
| fungi | my name is fungi and i've been trying to set up a ci system for more than 3 years | 18:41 |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/shade: Add Designate recordsets support https://review.openstack.org/306502 | 18:42 |
| *** prad is now known as pradk | 18:43 | |
| *** trown|afk is now known as trown | 18:43 | |
| *** infra-red has joined #openstack-infra | 18:44 | |
| *** ddieterly has joined #openstack-infra | 18:45 | |
| mordred | fungi: let me know if you ever get one working | 18:46 |
| anteaya | fungi: ha ha ha | 18:46 |
| fungi | i'll post a blog | 18:46 |
| anteaya | fungi: I feel so much better now | 18:46 |
| fungi | (once i figure out what a blog is) | 18:46 |
| anteaya | I've only been trying about 18 months | 18:46 |
| *** esikachev has joined #openstack-infra | 18:46 | |
| anteaya | don't bother with blogs, very transitory | 18:46 |
| mordred | they're so last year | 18:47 |
| anteaya | exactly | 18:47 |
| mordred | I believe snapchats are the current rage | 18:47 |
| *** EricGonczer_ has joined #openstack-infra | 18:47 | |
| *** infra-re_ has quit IRC | 18:47 | |
| anteaya | really I though we have moved to imbedded implants already | 18:47 |
| anteaya | posting your hormone levels and gps co-ordinates | 18:47 |
| *** ddieterly is now known as ddieterly[away] | 18:48 | |
| *** sdake has joined #openstack-infra | 18:48 | |
| fungi | snapchat already sounds very close to that | 18:49 |
| morgan | fungi: ugh. whelp... looks like my gpgkey is gone :( time to get a new one... woo re-doing web of trust. | 18:49 |
| morgan | fungi: *sigh* | 18:49 |
| fungi | morgan: did you at least stash a revocation certificate somewhere so you can kill it? | 18:49 |
| *** pradk_ has joined #openstack-infra | 18:49 | |
| morgan | fungi: nope. the whole thing is gone. | 18:49 |
| fungi | though i guess if it's not expected to be compromised, there's no real risk | 18:49 |
| morgan | fungi: i serioiusly doubt it's compromised. | 18:50 |
| morgan | just... secret key is long gone. | 18:50 |
| morgan | checked in 5 different backups. | 18:50 |
| morgan | not in any of them. | 18:50 |
| morgan | :( | 18:50 |
| fungi | morgan: i'll have a stack of business cards with my fingerprint on it and multiple forms of photo identification including a passport, so happy to dance with you | 18:50 |
| morgan | LOL | 18:50 |
| morgan | fungi: just.. annoying is all | 18:51 |
| *** dahoo has joined #openstack-infra | 18:51 | |
| fungi | indeed | 18:52 |
| clarkb | mordred: my biggest concern with this retry ip deletes thing is that it doesn't garuntee anything | 18:52 |
| *** cody-somerville has joined #openstack-infra | 18:52 | |
| clarkb | mordred: it is a workaround best effort (which may solve the problem in 99% of cases but we will still have to run the periodic thing behind it to clean up what falls through) | 18:52 |
| mordred | clarkb: that is true - we'll combine it with nodepool doing unattached FIP deletes | 18:52 |
| mordred | so yes | 18:52 |
| clarkb | mordred: ya I am just cranky that this is a thing | 18:52 |
| mordred | TOTALLY | 18:52 |
| mordred | it's insane | 18:52 |
| clarkb | mordred: neutron should do what it says it does | 18:52 |
| mordred | yes | 18:53 |
| mordred | neutron should not return success on failure | 18:53 |
| mordred | and nova should have an option to delete floating ips when the server is deleted | 18:53 |
| mordred | since there is one for cinder already | 18:53 |
| mordred | I mean, it boggles my mind that we have an implementation of making sure to delete DATA when a server is deleted, but not to be able to delete a floating ip | 18:54 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade https://review.openstack.org/243239 | 18:54 |
| mordred | that, combined with not having the port id in the addresses dict, and not having any metadata to tell you which networks are external and which aren't, having the metadata about nat destination be implied by the subnet rather than a quality reported by the network | 18:55 |
| *** brucet has joined #openstack-infra | 18:55 | |
| mordred | it's a sequence of several usability issues that together drive me batty | 18:55 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Tempest for tripleo CI https://review.openstack.org/295844 | 18:56 |
| *** ZZelle_ has joined #openstack-infra | 18:56 | |
| *** maishsk has joined #openstack-infra | 18:56 | |
| *** nwkarsten has joined #openstack-infra | 18:57 | |
| *** nwkarsten has quit IRC | 18:57 | |
| Shrews | mordred: BREATHE! | 18:57 |
| * Shrews hands mordred a warm, fuzzy kitten and a cold, foamy beer | 18:58 | |
| *** amrith is now known as _amrith_ | 18:58 | |
| flaper87 | I don't seem to be able to add my order number to my openstack foundation account. Are there any docs/references I should read? It fails saying it couldn't process my request. My guess is that my primary email in eventbrite is not the same as the one I use for the foundation account | 18:58 |
| flaper87 | who should I ping about this? | 18:58 |
| clarkb | mordred: is latest 307439 running yet? it seems fine for me other than it being best effort | 18:58 |
| jeblair | flaper87: what's an order number? | 18:58 |
| mordred | clarkb: yah | 18:59 |
| flaper87 | jeblair: 483391834 | 18:59 |
| flaper87 | jeblair: ah, ops | 18:59 |
| mordred | oh - speaking of | 18:59 |
| clarkb | jeblair: it is the eventbrite invoice number basically | 18:59 |
| flaper87 | jeblair: the event brite order number | 18:59 |
| *** tonytan4ever has quit IRC | 18:59 | |
| mordred | clarkb, jeblair: we have enough debug logging now to track leaked fips to server | 18:59 |
| clarkb | jeblair: your unique ticket identifier, quick everyone go to summit as flaper87 | 18:59 |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: Add initial setup for magnum in shade https://review.openstack.org/243239 | 18:59 |
| mordred | clarkb, jeblair: e968d76d-258e-4e0f-82e4-1bbd4988c0bf is a leaked fip | 18:59 |
| jeblair | what does it mean to associate it with your foundation account? | 18:59 |
| *** esikachev has quit IRC | 18:59 | |
| fungi | it's that (weekly infra team meeting) time again! find us in #openstack-meeting for the next hour | 18:59 |
| flaper87 | clarkb: was that number supposed to be private? I had no idea. | 19:00 |
| *** tonytan4ever has joined #openstack-infra | 19:00 | |
| clarkb | flaper87: I don't think it matters :) | 19:00 |
| * flaper87 epic fail | 19:00 | |
| clarkb | flaper87: jst giving you a hard time | 19:00 |
| mordred | clarkb, jeblair: so if you search for it in debug log, you can find things | 19:00 |
| flaper87 | clarkb: phew | 19:00 |
| mordred | like 2016-04-19 18:30:52,337 DEBUG shade: Created FIP 172.99.106.137:e968d76d-258e-4e0f-82e4-1bbd4988c0bf for server 77ebb888-f3e4-4746-b3dd-740119f509c4 | 19:00 |
| *** Sukhdev has quit IRC | 19:00 | |
| *** fawadkhaliq has quit IRC | 19:00 | |
| *** bpokorny has quit IRC | 19:00 | |
| clarkb | jeblair: for raisisn in order to use the mobile app schedule you must tie your eventbrite ticket to your foundation account | 19:00 |
| flaper87 | apparently I'm not the only one that just found out about this | 19:01 |
| jeblair | i'm sorry i missed the design session for that. i'd love to know the reason. | 19:01 |
| anteaya | summitapp@openstack.org | 19:01 |
| anteaya | that is the app support contact | 19:01 |
| flaper87 | anteaya: thanks, will email them | 19:01 |
| anteaya | great | 19:01 |
| clarkb | jeblair: what I am told (and I still don't understand it) is that they want to make sure people leaving feedback on sessions are real attendees | 19:01 |
| *** ddieterly[away] is now known as ddieterly | 19:02 | |
| jeblair | clarkb: or are, at this point, at least flaper87 ;) | 19:02 |
| Zara | :S I didn't know about that, intended to leave it to the last minute to install it, thanks for the heads up (don't yet know if it'll cause problems or not, but at least I'll recognise it) | 19:02 |
| *** rocky_g has joined #openstack-infra | 19:03 | |
| dstufft | mordred: IPs are forever but data is transient, seems to be legit | 19:03 |
| *** dimtruck is now known as zz_dimtruck | 19:03 | |
| *** salv-orlando has quit IRC | 19:03 | |
| *** doug-fis_ has joined #openstack-infra | 19:04 | |
| mordred | dstufft: :) | 19:05 |
| *** rbrndt_ has joined #openstack-infra | 19:05 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:06 | |
| *** brucet has quit IRC | 19:06 | |
| *** brucet has joined #openstack-infra | 19:06 | |
| *** doug-fis_ has quit IRC | 19:09 | |
| *** EricGonczer_ has quit IRC | 19:10 | |
| *** zz_dimtruck is now known as dimtruck | 19:12 | |
| *** jrmyjohn has joined #openstack-infra | 19:13 | |
| *** infra-re_ has joined #openstack-infra | 19:13 | |
| *** doug-fish has joined #openstack-infra | 19:14 | |
| *** Sukhdev has joined #openstack-infra | 19:15 | |
| *** esker has joined #openstack-infra | 19:16 | |
| *** jrmyjohn has quit IRC | 19:17 | |
| *** infra-red has quit IRC | 19:18 | |
| *** Sukhdev has quit IRC | 19:18 | |
| *** Sukhdev has joined #openstack-infra | 19:19 | |
| *** brucet has quit IRC | 19:20 | |
| *** cody-somerville has quit IRC | 19:20 | |
| nibalizer | if you look at diff modules/openstack_project/manifests/openstackid_prod.pp modules/openstack_project/manifests/openstackid_dev.pp you can see there is really no reason for these to be separate files | 19:20 |
| nibalizer | just as an exmaple | 19:20 |
| *** brucet has joined #openstack-infra | 19:21 | |
| rcarrillocruz | yeah dude | 19:21 |
| rcarrillocruz | that's terrible | 19:21 |
| *** Guest35267 has quit IRC | 19:21 | |
| rcarrillocruz | some cargo cult from the gerrit prod gerrit dev i assume | 19:21 |
| anteaya | will changing this make individual puppet modules more easy to deploy locally? | 19:21 |
| anteaya | I can not figure out how to deploy the puppet-mediawiki module locally | 19:21 |
| anteaya | apache won't start because I can't figure out how to give it the ssl cert paths to certs that don't exist on my vm | 19:22 |
| SotK | nibalizer: it may be worth talking to pedroalvarez if you are interested in hosting/testing an ansible role for storyboard btw | 19:22 |
| * Zara updates story so it has context on the mention of Ansible. https://storyboard.openstack.org/#!/story/2000028 | 19:23 | |
| Zara | and yes, +5 to talking to pedroalvarez | 19:23 |
| *** apoorvad has quit IRC | 19:24 | |
| *** mylu has joined #openstack-infra | 19:25 | |
| *** nadya has joined #openstack-infra | 19:26 | |
| *** EricGonczer_ has joined #openstack-infra | 19:27 | |
| *** dizquierdo has quit IRC | 19:28 | |
| *** ddieterly is now known as ddieterly[away] | 19:28 | |
| *** infra-red has joined #openstack-infra | 19:29 | |
| pedroalvarez | I'll happily help with Ansible and storyboard, and more if needed | 19:30 |
| *** gyee has quit IRC | 19:31 | |
| Zara | :D | 19:31 |
| *** infra-re_ has quit IRC | 19:32 | |
| *** Latha has joined #openstack-infra | 19:32 | |
| *** ociuhandu has joined #openstack-infra | 19:32 | |
| *** ddieterly[away] is now known as ddieterly | 19:32 | |
| clarkb | nibalizer: rcarrillocruz until very recently the big difference there was in the realized users, obviously there are other ways of splitting the ocnfigs but this is one way and it works | 19:33 |
| *** brucet has quit IRC | 19:33 | |
| Latha | Hi I am having trouble in entering contact information I get this when i use gerit review Please review your contact information: https://review.openstack.org/#/settings/contact | 19:33 |
| rcarrillocruz | sure, i guess there was never a need to do a refactor | 19:33 |
| *** brucet has joined #openstack-infra | 19:33 | |
| openstackgerrit | yolanda.robla proposed openstack-infra/shade: WIP: Add baymodel list call for shade+magnum https://review.openstack.org/308009 | 19:33 |
| rcarrillocruz | i mean, the dev one is not spinned 'often' | 19:33 |
| openstackgerrit | Merged openstack-infra/tripleo-ci: Don't use cached undercloud image for element repos either https://review.openstack.org/307923 | 19:34 |
| clarkb | Latha: are you following http://docs.openstack.org/infra/manual/developers.html#account-setup ? that should walk you through the steps | 19:34 |
| clarkb | Latha: if not following that please read that section on acount setup and it should get you through the contact info stuff | 19:35 |
| *** Sukhdev has quit IRC | 19:36 | |
| *** pradk has quit IRC | 19:37 | |
| *** pradk_ has quit IRC | 19:37 | |
| *** maestro1 has joined #openstack-infra | 19:37 | |
| *** pradk has joined #openstack-infra | 19:39 | |
| *** mpaolino has quit IRC | 19:39 | |
| *** pradk has quit IRC | 19:40 | |
| *** Swami has joined #openstack-infra | 19:40 | |
| *** Sukhdev has joined #openstack-infra | 19:41 | |
| Latha | i could setup the account and able to login | 19:41 |
| *** mtreinish has quit IRC | 19:41 | |
| Latha | and error comes when submitting changes' | 19:41 |
| anteaya | you have to set up accounts with launchpad, gerrit and the foundation | 19:41 |
| Latha | I might have missed putting contact info while registering as user | 19:41 |
| anteaya | did you set up all the accounts | 19:41 |
| anteaya | and did you follow the link clarkb posted? | 19:42 |
| *** mtreinish has joined #openstack-infra | 19:42 | |
| *** pradk has joined #openstack-infra | 19:43 | |
| Latha | yes i only see regsiter a project register a team | 19:44 |
| sdake | mordred it looks like internap suffers from the same problem as ovh was suffering from | 19:44 |
| sdake | http://logs.openstack.org/91/306791/5/check/gate-kolla-dsvm-build-ubuntu-source/4500d6d/console.html#_2016-04-18_18_22_28_537 | 19:44 |
| *** hferenc has joined #openstack-infra | 19:44 | |
| *** dimtruck is now known as zz_dimtruck | 19:44 | |
| *** baoli has quit IRC | 19:44 | |
| anteaya | Latha: what are you looking at in http://docs.openstack.org/infra/manual/developers.html#account-setup that says register a project register a team | 19:45 |
| mordred | sdake: that was a log from 24 hours ago? | 19:45 |
| sdake | looks like it | 19:45 |
| sdake | i just triggered a new recheck | 19:45 |
| sdake | so we will see if the problem stil exists | 19:45 |
| mordred | sdake: the fix should have been running for at least the last 8 or so | 19:45 |
| sdake | oh i see | 19:46 |
| sdake | ok well i'll get back to youo after recheck finishes | 19:46 |
| hferenc | hello all! | 19:46 |
| *** shardy has quit IRC | 19:46 | |
| openstackgerrit | Sean M. Collins proposed openstack-infra/project-config: Make grenade-dsvm-neutron-multinode voting https://review.openstack.org/299843 | 19:48 |
| hferenc | finally, i was able to create a cirros image that has an i6300esb watchdog driver in it | 19:48 |
| Latha | i have an account already setup and at that time didn't put my contact info and now when i submit changes it is giving me this | 19:49 |
| Latha | Type 'yes' to confirm, other to cancel: yes fatal: ICLA contributor agreement requires current contact information. Please review your contact information: https://review.openstack.org/#/settings/contact fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. | 19:49 |
| *** mtreinish has quit IRC | 19:49 | |
| anteaya | Latha: did you follow the link clarkb and I gave you? | 19:50 |
| anteaya | particularly about ensuring you have a foundation account | 19:50 |
| anteaya | and the membership is set correctly | 19:50 |
| anteaya | and that the email in the foundation account match the email in your gerrit account? | 19:50 |
| hferenc | i used the lp:cirros repo as a base, however as it turned out the result is not as close to the 'official' 0.3.4 cirros image as i hoped it will be | 19:52 |
| *** Sukhdev has quit IRC | 19:53 | |
| hferenc | can somebody tell me how the 'official' images are created? | 19:53 |
| sc68cal | hferenc: whose 'official' cirros image are you referring to | 19:55 |
| sc68cal | and what do you mean by not as close | 19:55 |
| hferenc | i meant these: http://download.cirros-cloud.net/0.3.4/ - i think these are the default built in images e.g. in devstack | 19:57 |
| *** shardy has joined #openstack-infra | 19:58 | |
| Latha | Yes. I followed the guide and was trying the tutorial " Learn the Gerrit Workflow in the Sandbox." when I get to "git review" step to submit patch set i get the error as "fatal: ICLA contributor agreement requires current contact information." and then when I try to update the contact information in gerrit, I get the error "Server Error. Cannot store contact information" | 19:58 |
| anteaya | look at your email in gerrit | 19:59 |
| anteaya | and look at your email in your foundation account | 19:59 |
| anteaya | are they the same email? | 19:59 |
| pleia2 | mordred: your mission is to find me something kosher for passover to drink at the summit | 19:59 |
| *** elo has joined #openstack-infra | 20:00 | |
| pleia2 | agave tequila should be ok, no beer D: | 20:00 |
| pleia2 | it's going to be a rough week | 20:00 |
| bkero | fungi: care to bring your chorded keyboard to Summit? :) | 20:00 |
| *** eric_lopez has quit IRC | 20:00 | |
| Zara | a musical summit? :D | 20:01 |
| *** esikachev has joined #openstack-infra | 20:01 | |
| mordred | pleia2: if agave tequila is ok, then you're fine | 20:01 |
| mordred | pleia2: are avocados, chips or salsa kosher for passover? | 20:01 |
| clarkb | pleia2: I hear there are plenty of margaritas | 20:01 |
| *** johnny has joined #openstack-infra | 20:01 | |
| fungi | bkero: i shall. my gen-1 prototype is terribly bulky (it's in an old phone handset), but i'll make room | 20:01 |
| Latha | anteaya, yes it is the same e-mail | 20:01 |
| *** johnny is now known as Guest94200 | 20:02 | |
| *** pradk has quit IRC | 20:02 | |
| nibalizer | pleia2: i will join you in hunting out good tequilaa | 20:02 |
| pleia2 | mordred: depends on ths chips | 20:02 |
| pleia2 | mordred: but the rest is good | 20:02 |
| *** aimeeU has quit IRC | 20:02 | |
| pleia2 | and I can eat meat :d just no bread-things | 20:02 |
| pleia2 | texas is good for that | 20:02 |
| sc68cal | hferenc: ok. so, you build a cirros image using the tip of the cirros source and it differed from 0.3.4 - doesn't that sound like a truism? | 20:02 |
| *** piet has joined #openstack-infra | 20:02 | |
| mordred | pleia2: oh - so the stuffed avocado will not work though, because it's breaded | 20:02 |
| bkero | fungi: awesome. I'll probably be bringing my twiddler | 20:02 |
| pleia2 | nibalizer: hooray | 20:02 |
| anteaya | Latha: is your foundation membership a Foundation membership or a community membership? | 20:02 |
| pleia2 | mordred: ah, yeah | 20:03 |
| fungi | bkero: i can bring you a dev board too if you want. i have several extra pcbs. i don't have enough spare component bags though so if you want to build one you'll need to consult my digikey parts manifest | 20:03 |
| mordred | pleia2: otoh - you will be surrounded by so much brisket | 20:03 |
| pleia2 | hehe, right | 20:03 |
| Latha | community | 20:03 |
| bkero | fungi: That might be useful :) | 20:03 |
| bkero | Got a link to your page that you describe it on? | 20:03 |
| anteaya | Latha: you need to have a Foundation membership _not_ a community membership | 20:04 |
| SamYaple | pleia2: they make a few different good tequilas right there in austin. its one of the things i miss since i moved away | 20:04 |
| SamYaple | also, suprisingly, vodka | 20:04 |
| *** michaelxin has quit IRC | 20:05 | |
| fungi | bkero: yeah, http://www.yuggoth.org/wiki/ManMachine/KeyerPCB has details, though i haven't updated it since actually getting the printed boards back and putting one into service | 20:05 |
| pleia2 | SamYaple: lovely | 20:06 |
| *** salv-orlando has joined #openstack-infra | 20:06 | |
| bkero | fungi: are those self-etched? | 20:06 |
| mordred | clarkb, jeblair: I cannot see anything in the extra logging that indicates why the leaked fips leaked | 20:06 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:06 | |
| *** _amrith_ is now known as amrith | 20:07 | |
| *** nwkarsten has joined #openstack-infra | 20:07 | |
| fungi | bkero: no, i had http://fab.fritzing.org/fritzing-fab do them for me | 20:07 |
| bkero | Ah cool | 20:07 |
| *** michaelxin has joined #openstack-infra | 20:07 | |
| fungi | i like their service and ethos, so happy to support them with my monies | 20:07 |
| *** EricGonczer_ has quit IRC | 20:08 | |
| *** mylu has quit IRC | 20:08 | |
| *** mylu has joined #openstack-infra | 20:08 | |
| hferenc | sc68cal: sure it does, but i've not found the repo that has the sources of 0.3.4 | 20:10 |
| *** Goneri has quit IRC | 20:10 | |
| *** amitgandhinz has quit IRC | 20:10 | |
| *** pvaneck_ has joined #openstack-infra | 20:10 | |
| fungi | bkero: i need to pick this project back up, though the plan for the next-gen build is to integrate keyer into a 3d-printed housing for a forearm-mounted wearable system | 20:11 |
| clarkb | mordred: is there any common thread among the leaks? multinode, maybe a common distro, possibly a subrange of IPs? | 20:11 |
| *** salv-orlando has quit IRC | 20:11 | |
| mordred | clarkb: nope | 20:11 |
| Latha | hi anteaya,how to change it now from one to the other or have to re-register? | 20:11 |
| *** flepied has quit IRC | 20:11 | |
| anteaya | Latha: if you sign into your foundation account you should have the ability to change the membership | 20:11 |
| *** brucet has quit IRC | 20:11 | |
| fungi | Latha: https://ask.openstack.org/question/56720 has a screenshot of the button you're looking for there | 20:12 |
| anteaya | Latha: please don't re-register, that just makes things worse | 20:12 |
| *** madhu_ak has joined #openstack-infra | 20:12 | |
| *** daemontool has quit IRC | 20:12 | |
| sc68cal | hferenc: are you familiar with bzr? | 20:12 |
| mordred | clarkb: oh - well, actually - maybe | 20:12 |
| *** Yarboa has joined #openstack-infra | 20:12 | |
| fungi | Latha: you're looking for a button in the upper right of your profile edit screen which says "make me a foundation member" | 20:12 |
| jeblair | mordred: i'll start looking | 20:12 |
| mordred | the node I started looking at was involved in a timeout | 20:13 |
| bkero | fungi: sounds useful | 20:13 |
| mordred | 2016-04-19 19:30:42,714 ERROR nodepool.NodeLauncher: OpenStackCloudTimeout launching node id: 333045 in provider: osic-cloud1 error: | 20:13 |
| *** esker has quit IRC | 20:13 | |
| bkero | fungi: I'd be interested in how you'd solve the ergo problems of typing on opposite arms | 20:13 |
| *** pvaneck has quit IRC | 20:13 | |
| *** esker has joined #openstack-infra | 20:14 | |
| *** fawadkhaliq has joined #openstack-infra | 20:14 | |
| jeblair | mordred: updating https://etherpad.openstack.org/p/vGZsfXNbp6 | 20:14 |
| madhu_ak | hi there, I was trying to verify experimental job in octavia, as per the patch: https://review.openstack.org/#/c/306053/. It fails because of $PROJECTS variable used in https://github.com/openstack/octavia/blob/master/octavia/tests/contrib/pre_test_hook.sh. Error can be seen in devstack logs: http://logs.openstack.org/53/306053/2/experimental/gate-tempest-dsvm-octavia/4e55e16/logs/devstacklog.txt.gz. Is this the right way in adding PROJECTS | 20:14 |
| madhu_ak | variable in gate hooks than adding in job definition of octavia.yaml under project-config? | 20:14 |
| clarkb | pleia2: the googles tell me there is a group that certifies beer as kosher. sam adams has a big list | 20:15 |
| clarkb | pleia2: but you probably know more about all that than I or google do | 20:15 |
| fungi | bkero: i intend to wear it on my left. i'm mostly right-hand dominant so trained to chord-type with my left so that my right hand is free for other fine motor skills. the housing i'm imagining could probably just be inverted pretty trivially but wouldn't likely be easy to make one unit usable interchangeably by both hands | 20:15 |
| bkero | fungi: an alternate would be to wear it on your leg or hanging off a pant pocket | 20:16 |
| jeblair | mordred, clarkb: it timed out waiting for the fip to be attached | 20:16 |
| clarkb | jeblair: ahah! | 20:16 |
| pleia2 | clarkb: yeah, just tricky to find while out and about | 20:16 |
| clarkb | madhu_ak: no, the pre test hook runs after all of the setup is already done | 20:16 |
| fungi | bkero: i did a shoulder strap for the current wearable system i have and simply tether the keyer by a retractable usb cord, but it's a bit cumbersome. trying to condense a bunch in the next design | 20:17 |
| clarkb | madhu_ak: you need to set all that config before devstack gate runs | 20:17 |
| bkero | fungi: Yeah. At least with prototypes they can be as bulky as necessary. | 20:17 |
| clarkb | jeblair: mordred: in that case we don't delete the fip because it isn't attached which is where we get the info to know what to delete | 20:17 |
| greghaynes | dhellmann: Is there a page with info about the different release types and what they mean? I get the impression https://wiki.openstack.org/wiki/ReleaseTeam/ReleaseProcess is where it is meant to end up? | 20:18 |
| mordred | jeblair, clarkb: ah. hrm. | 20:18 |
| *** matrohon has joined #openstack-infra | 20:19 | |
| madhu_ak | clarkb, so should I define https://github.com/openstack/octavia/blob/master/octavia/tests/contrib/pre_test_hook.sh#L18 and 19 in https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/octavia.yaml#L24 ? | 20:19 |
| anteaya | greghaynes: he is in the tc meeting right now so might be a few before he replies | 20:19 |
| greghaynes | anteaya: ah, ty | 20:19 |
| anteaya | welcome | 20:19 |
| clarkb | madhu_ak: most of that script actually | 20:20 |
| clarkb | madhu_ak: I don't think the devstack local config will bubble out, but you can check the logs to be double sure | 20:20 |
| mordred | jeblair: awesome. so - I'd like to add two more pieces of debugging there | 20:20 |
| clarkb | madhu_ak: and typically you use the features.yaml to select which services to run | 20:20 |
| mordred | jeblair: because we actually SHOULD have enough context at that point in shade to delete | 20:20 |
| clarkb | madhu_ak: easiest way to see is to check the logs | 20:20 |
| mordred | but I'd like to make sure I understand what timing out means here | 20:20 |
| pabelanger | clarkb: so, looking into diskimage-builder. I believe this is how they handle setfiles: https://git.openstack.org/cgit/openstack/diskimage-builder/tree/elements/rpm-distro/finalise.d/11-selinux-fixfiles-restore So, we should be able to do the same | 20:20 |
| *** kzaitsev_mb has joined #openstack-infra | 20:20 | |
| *** zz_dimtruck is now known as dimtruck | 20:20 | |
| dhellmann | greghaynes : http://governance.openstack.org/reference/tags/index.html#release-management-tags | 20:21 |
| pabelanger | clarkb: We just need to figure out how to properly call it | 20:21 |
| clarkb | mordred: http://logs.openstack.org/53/306053/2/experimental/gate-tempest-dsvm-octavia/4e55e16/logs/localrc.txt.gz the rest of it looks fine | 20:21 |
| clarkb | er madhu_ak ^ | 20:21 |
| greghaynes | dhellmann: awesome, thanks | 20:21 |
| *** mylu has quit IRC | 20:21 | |
| pabelanger | ianw: thoughts on how we can breakout ^ | 20:21 |
| dhellmann | greghaynes : release info in the wiki is largely out of date, because we've moved to the project team guide and governance repo | 20:21 |
| bkero | fungi: Just read on the page. I happen to have a set of ST1080s :) | 20:22 |
| madhu_ak | clarkb, I could see these from devstack logs: | 20:22 |
| pabelanger | ianw: vs just copying it into our nodepool-base element | 20:22 |
| madhu_ak | echo 'The /opt/stack/new/barbican project was not found; if this is a gate job, add' | 20:22 |
| madhu_ak | 2016-04-14 23:04:36.561 | The /opt/stack/new/barbican project was not found; if this is a gate job, add | 20:22 |
| madhu_ak | 2016-04-14 23:04:36.563 | + functions-common:git_clone:533 : echo 'the project to the $PROJECTS variable in the job definition.' | 20:22 |
| madhu_ak | 2016-04-14 23:04:36.564 | the project to the $PROJECTS variable in the job definition. | 20:22 |
| madhu_ak | 2016-04-14 23:04:36.566 | + functions-common:git_clone:534 : die 534 'Cloning not allowed in this configuration' | 20:22 |
| *** fawadkhaliq has quit IRC | 20:22 | |
| clarkb | madhu_ak: yes, please don't paste large amounts of test to the channel, use a paste bin instead | 20:22 |
| clarkb | madhu_ak: that is happening becaue the PROJECTS list is one of the first things processed in the devstack gate run | 20:22 |
| clarkb | madhu_ak: you need to set that in your job as yo udescribed | 20:22 |
| *** fawadkhaliq has joined #openstack-infra | 20:22 | |
| clarkb | pabelanger: ok, I have little experience with that but can probablydig up man pages and figure it out if that helps | 20:23 |
| *** tonytan_brb has joined #openstack-infra | 20:23 | |
| madhu_ak | sure clarkb will push a patch on project-infra. Thanks for the info | 20:23 |
| *** austin81 has joined #openstack-infra | 20:23 | |
| *** dimtruck is now known as zz_dimtruck | 20:23 | |
| clarkb | pabelanger: looks like it is specfile then path | 20:24 |
| *** brucet has joined #openstack-infra | 20:24 | |
| *** tonytan4ever has quit IRC | 20:24 | |
| *** lyz is now known as lyz_ | 20:24 | |
| clarkb | pabelanger: actually, couldn't we just use that element? | 20:24 |
| clarkb | pabelanger: I will test using that element locally | 20:24 |
| clarkb | pabelanger: if that works we can remove our restorecon completely | 20:25 |
| *** pradk has joined #openstack-infra | 20:25 | |
| pabelanger | clarkb: Right, I don't know how to just use part of an element (eg: finalise.d part( | 20:25 |
| clarkb | pabelanger: looking at the other bits they seem harmless | 20:26 |
| *** nadya has quit IRC | 20:26 | |
| clarkb | pabelanger: and some may also be beneficial | 20:26 |
| *** sridhar_ram has joined #openstack-infra | 20:26 | |
| pabelanger | clarkb: I'll defer to ianw, since he did alot of the work. But if we can use it, great! | 20:26 |
| clarkb | ya I will run a test locally and if it works from what I can see then push a patch to nodepool.yaml | 20:27 |
| clarkb | that will give ianw something concrete to review | 20:27 |
| *** doug-fish has quit IRC | 20:27 | |
| *** Guest94200 has quit IRC | 20:29 | |
| *** mylu has joined #openstack-infra | 20:29 | |
| *** bpokorny has joined #openstack-infra | 20:30 | |
| fungi | bkero: what did you think of the st1080? i never got around to picking one up, and need to re-survey the current landscape of lightweight video goggles on the market now | 20:30 |
| greghaynes | dhellmann: hrm, so the question I am trying to find an answer to is - which release tags require the release team to approve a release and which ones dont. Is the independent tag the only one which doesnt require the release team to approve releases? | 20:30 |
| *** zz_dimtruck is now known as dimtruck | 20:30 | |
| *** brucet has quit IRC | 20:31 | |
| *** armax has joined #openstack-infra | 20:32 | |
| *** bdemers has quit IRC | 20:32 | |
| *** tqtran has quit IRC | 20:32 | |
| *** esikachev has quit IRC | 20:34 | |
| dhellmann | greghaynes : we should talk about this in #openstack-release, but all official projects are expected to use the reviewable release process this cycle | 20:34 |
| dhellmann | greghaynes : though that is being discussed in the ML thread about dib, and will likely be a subject of discussion in the release session at the summit | 20:35 |
| openstackgerrit | Madhusudhan Kandadai proposed openstack-infra/project-config: Include PROJECTS list for octavia job https://review.openstack.org/308023 | 20:35 |
| *** mylu has quit IRC | 20:35 | |
| *** ddieterly is now known as ddieterly[away] | 20:35 | |
| *** doug-fish has joined #openstack-infra | 20:36 | |
| madhu_ak | clarkb, please review ^^ once you get time. | 20:36 |
| *** BigWillie has quit IRC | 20:36 | |
| *** doug-fish has quit IRC | 20:36 | |
| *** doug-fish has joined #openstack-infra | 20:36 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 20:37 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on attach timeout when we created it https://review.openstack.org/308024 | 20:37 |
| mordred | jeblair: ^^ | 20:37 |
| *** ddieterly[away] is now known as ddieterly | 20:37 | |
| clarkb | pabelanger: ok I have a build running with that element in it | 20:37 |
| bkero | fungi: I've been a bit unimpressed. The strap to keep them attached is terrible and comes apart very easily. | 20:37 |
| clarkb | pabelanger: will probably take an hour or so | 20:37 |
| pabelanger | clarkb: great | 20:38 |
| bkero | The screens are of limited size and have terrible contrast | 20:38 |
| bkero | but it is 1080p as advertised | 20:38 |
| fungi | bkero: i'd quite possibly hack it up a bit anyway. mostly interested in screen resolution and focus | 20:38 |
| bkero | also it has a control box that gets really hot | 20:38 |
| fungi | eww, lots of waste heat means eats battery | 20:38 |
| bkero | yeah | 20:39 |
| *** dimtruck is now known as zz_dimtruck | 20:39 | |
| bkero | I can bring it to the summit | 20:39 |
| bkero | I have a wearable computer made with it | 20:39 |
| bkero | With a FXi Cotton Candy and Jorno folding keyboard. I have a very stylish Pokemon Blastoise case to hold it all. | 20:39 |
| *** EricGonczer_ has joined #openstack-infra | 20:40 | |
| *** ddieterly is now known as ddieterly[away] | 20:40 | |
| *** dprince has quit IRC | 20:40 | |
| *** Sukhdev has joined #openstack-infra | 20:40 | |
| *** armax_ has joined #openstack-infra | 20:40 | |
| bkero | fungi: https://twitter.com/dietrich/status/473914637960744960 | 20:40 |
| *** mtreinish1 has joined #openstack-infra | 20:41 | |
| fungi | heh, blastoise ftw! | 20:41 |
| *** zz_dimtruck is now known as dimtruck | 20:42 | |
| *** armax has quit IRC | 20:42 | |
| *** armax_ is now known as armax | 20:42 | |
| *** salv-orlando has joined #openstack-infra | 20:45 | |
| jeblair | mordred, clarkb, fungi: we're going to get a neat new behavior from zuulv2.5 -- when you shut down a launcher, it will release all of its nodes back to nodepool automatically | 20:46 |
| anteaya | yay! | 20:46 |
| jeblair | (i'm going to have it send fake completed events) | 20:46 |
| fungi | worried about thundering herd of deletes? | 20:46 |
| jeblair | fungi: not at all, i'm going to create one :) | 20:47 |
| jeblair | fungi: that's nodepool's job | 20:47 |
| jeblair | fungi: but this way we won't have to do the "i just shut down jenkins07, so now i need to delete all the ready nodes attached to it" thing | 20:47 |
| *** lyz_ is now known as lyz | 20:47 | |
| mordred | jeblair: nice! | 20:47 |
| fungi | well, we'll also stop needing to shut down jenkins07 (except for maybe one last time) | 20:48 |
| *** jrmyjohn has joined #openstack-infra | 20:48 | |
| *** EricGonczer_ has quit IRC | 20:48 | |
| jeblair | heh :) i'm assuming we might need to *occasionally* restart these things, but it's good to have goals. | 20:48 |
| fungi | yep! | 20:48 |
| *** baoli has joined #openstack-infra | 20:48 | |
| fungi | awesome | 20:48 |
| mordred | jeblair: NO RESTARTING THINGS | 20:49 |
| mordred | Shrews: see patch I posted above ... definitely one I'd like to have brains on | 20:49 |
| *** brucet has joined #openstack-infra | 20:49 | |
| pabelanger | jeblair: 2.5 sounds exciting | 20:50 |
| Shrews | mordred: i'll see if i can find someone matching those requirements | 20:50 |
| * Shrews looks anyway | 20:50 | |
| *** jrmyjohn has quit IRC | 20:51 | |
| *** jrmyjohn has joined #openstack-infra | 20:51 | |
| Shrews | mordred: ooh, was that the other leaky thing? | 20:51 |
| mordred | Shrews: that, it seems, actually might be the cause of the leaks | 20:52 |
| *** mrmartin has quit IRC | 20:53 | |
| mordred | at least in the neutron case | 20:53 |
| Shrews | mordred: we definitely need a log message there | 20:53 |
| mordred | kk | 20:53 |
| mordred | Shrews: error log you think? | 20:53 |
| Shrews | mordred: yeah. i assume we don't need this for the nova case? | 20:53 |
| *** tristanC_ is now known as tristanC | 20:54 | |
| mordred | well, the nova case leaks ina completely different way | 20:54 |
| mordred | but it's not possible to reason about what state we're in at this point in the stack for nova | 20:54 |
| *** _ari_ has quit IRC | 20:54 | |
| *** andymaier has quit IRC | 20:54 | |
| *** shardy has quit IRC | 20:55 | |
| clarkb | pabelanger: my git cache is very out of date and slowly updating | 20:55 |
| *** mylu has joined #openstack-infra | 20:55 | |
| clarkb | I should invest in better disk | 20:56 |
| *** brucet has quit IRC | 20:56 | |
| krotscheck | Can someone help set up the Gerrit ACL's on the openstack-infra/js-afs-blob-store? | 20:57 |
| *** brucet has joined #openstack-infra | 20:57 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on attach timeout when we created it https://review.openstack.org/308024 | 20:57 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 20:57 |
| mordred | Shrews: how's that | 20:57 |
| krotscheck | It should inherit from infra core. | 20:57 |
| krotscheck | Maybe we should create an infra-javascript group? One project doesn't seem like it's worth it yet though. | 20:58 |
| Shrews | mordred: fine, but... i'm not sure about the use of skip_attach in that logic... | 21:00 |
| mordred | Shrews: yah - that's the thing I wanted to talk about | 21:01 |
| *** rfolco has quit IRC | 21:01 | |
| *** ddieterly[away] is now known as ddieterly | 21:01 | |
| fungi | krotscheck: oh, we have a script to run as long as it's already in governance | 21:01 |
| mordred | Shrews: we know if skip_attach is passed that it's in a flow where the neutron fip create was done on the server port | 21:01 |
| mordred | Shrews: that means the user did not pass us in a pre-existing fip to attach to the server | 21:02 |
| *** sridhar_ram has quit IRC | 21:02 | |
| *** jheroux has quit IRC | 21:02 | |
| fungi | krotscheck: is the governance change for that addition still pending? | 21:02 |
| krotscheck | fungi: Ah, it's still RC-Vote. | 21:02 |
| krotscheck | https://review.openstack.org/#/c/304124/ | 21:02 |
| *** mylu has quit IRC | 21:02 | |
| *** dims_ has joined #openstack-infra | 21:02 | |
| mordred | Shrews: since we're actually hiding the fact that the ip is a fip from the user, I think we have to delete it - but it's dangerous if we can get there via another path | 21:02 |
| *** dims has quit IRC | 21:02 | |
| mordred | Shrews: we could also make a new flag "delete_fip_on_fail" or something and plumb that through | 21:02 |
| fungi | krotscheck: you need to petition your friendly local tc members to vote for that. i think they need a few more still | 21:03 |
| clarkb | mordred: couldn't you just delete if you created there | 21:03 |
| clarkb | mordred: and don't delete if the fip is handed to us and created elsewhere? | 21:03 |
| mordred | clarkb: right. that's the thing we're trying to accomplish | 21:03 |
| *** brucet has quit IRC | 21:03 | |
| *** rossella_s has quit IRC | 21:03 | |
| clarkb | btw I am pretty sure this case is handled in old nodepool, we traded one leak for another :) | 21:03 |
| mordred | clarkb: zomg. its completely different | 21:03 |
| mordred | and no we didn;t | 21:04 |
| *** mylu has joined #openstack-infra | 21:04 | |
| mordred | the entire mechanism is 100% different | 21:04 |
| openstackgerrit | Armando Migliaccio proposed openstack-infra/project-config: Remove publisher site for vendor specific networking-* projects https://review.openstack.org/308033 | 21:04 |
| *** rossella_s has joined #openstack-infra | 21:04 | |
| clarkb | mordred: if fip attach failed in old nodepool we deleted the fip right there | 21:04 |
| *** vponomaryov1 has quit IRC | 21:04 | |
| mordred | clarkb: yes. this is not fip attach failing | 21:04 |
| clarkb | mordred: but we didn't handle the fip didn't delete properly nad now server is gone case | 21:04 |
| clarkb | mordred: it isn't? | 21:04 |
| mordred | no | 21:04 |
| clarkb | then I don't understand "Delete floating IP on attach timeout when we created it" | 21:05 |
| mordred | this is nova server record not refreshing itself | 21:05 |
| *** ddieterly is now known as ddieterly[away] | 21:05 | |
| mordred | we use the attach logic to do the server polling to see the nova record refresh after the fip creation | 21:05 |
| *** amrith is now known as _amrith_ | 21:05 | |
| mordred | since polling the server for an active fip is the same no matter how it was created or attached | 21:05 |
| *** psilvad has quit IRC | 21:05 | |
| *** aysyd has quit IRC | 21:05 | |
| fungi | so the race is service-side? | 21:05 |
| mordred | it's just a failure service side | 21:05 |
| mordred | there is so far no indication that the server side will ever recover | 21:06 |
| fungi | ahh, it doesn't magically show up later | 21:06 |
| mordred | it _might_ | 21:06 |
| mordred | we've never waited long enough :) | 21:06 |
| clarkb | mordred: does the fip function though? | 21:06 |
| *** baoli_ has joined #openstack-infra | 21:06 | |
| clarkb | (honestly that seems ot me that fip attach failed, regardless of return code from neutron) | 21:06 |
| mordred | so - if it's like the tripleo cloud fail (where this just happens 100% of the time) | 21:07 |
| mordred | then yes, the fip does function | 21:07 |
| mordred | it's just that nova doesn't show it to you | 21:07 |
| *** gyee has joined #openstack-infra | 21:07 | |
| mordred | but that's broken enough for _us_ | 21:07 |
| *** sridhar_ram has joined #openstack-infra | 21:07 | |
| clarkb | I see, any chance we can interrogate the othe rdirection (and ignore nova net for a minute) and ask neutron if the thing is properly attached? | 21:07 |
| mordred | yes | 21:08 |
| mordred | we can totally do that | 21:08 |
| *** baoli__ has joined #openstack-infra | 21:08 | |
| mordred | it'll just be generally wonky because it woul require an additional set of neutron calls just to do a server get | 21:08 |
| clarkb | or as another alternative, bubble the error up out of this functio nand handle it in the calling context | 21:08 |
| mordred | but it would be totally possible | 21:08 |
| clarkb | because the calling context shold know if the fip was created by shade under the covers or explicitly by the user | 21:08 |
| *** mylu has quit IRC | 21:08 | |
| anteaya | so grafana zuul status says 1034 in check: http://grafana.openstack.org/dashboard/db/zuul-status | 21:08 |
| *** thorst_ has quit IRC | 21:08 | |
| *** mylu has joined #openstack-infra | 21:09 | |
| krotscheck | jeblair: Quick update. There should be a way for me to get the registry code to exit when it reaches a wait() state, I just need to hammer out with the yahoo team on where best to hook that in. | 21:09 |
| *** ddieterly[away] is now known as ddieterly | 21:09 | |
| anteaya | and http://status.openstack.org/zuul/ says 645 in check | 21:09 |
| mordred | clarkb: yes - looking at it - I believe we can do that | 21:09 |
| clarkb | anteaya: grafana counts the deps in check, status only counts the change under test iirc | 21:09 |
| krotscheck | jeblair: ETA on solution... probably sometime during the infra sessions at the summit. | 21:09 |
| anteaya | anyone know where grafana is getting its numbers for check? | 21:09 |
| mordred | clarkb: let me amend the patch - yeah, I thin kthat'll read clearer | 21:09 |
| *** gordc has quit IRC | 21:09 | |
| anteaya | clarkb: ah thank you | 21:09 |
| Shrews | mordred: k, i think i see your logic there. we're deleting in the case where auto_ip=True (basically), but not in other scenarios like where we say "use this pool of fips" | 21:09 |
| clarkb | Shrews: yup | 21:09 |
| mordred | Shrews: yah | 21:10 |
| *** ddieterly has quit IRC | 21:10 | |
| mordred | and clarkb's point about doing it up the stack I think is right on | 21:10 |
| Shrews | oh, i missed the scrollback. was groking code | 21:10 |
| *** baoli has quit IRC | 21:10 | |
| Shrews | yes, higher up would clarify the logic, i think | 21:10 |
| *** Sukhdev has quit IRC | 21:11 | |
| clarkb | pabelanger: my build seems to have failed on dep resolution installing puppet? I am going to retry to see if it is flaky mirror issues | 21:11 |
| *** brucet has joined #openstack-infra | 21:11 | |
| pabelanger | clarkb: ack | 21:11 |
| *** baoli__ has quit IRC | 21:11 | |
| *** rocky_g has quit IRC | 21:11 | |
| *** baoli_ has quit IRC | 21:11 | |
| *** baoli has joined #openstack-infra | 21:12 | |
| *** trown is now known as trown|outtypewww | 21:12 | |
| austin81 | AJaeger, anteaya, ianw, sdague: Would y'all mind taking a look at https://review.openstack.org/#/c/279317/ for me? I think it's ready to be merged | 21:13 |
| *** hferenc has quit IRC | 21:14 | |
| jeblair | krotscheck: cool | 21:15 |
| fungi | anteaya: my gut says it's counting all changes in the pipeline rather than only active changes (copies of dependent parent changes show up as inactive in independent pipelines( | 21:15 |
| clarkb | also I learned a new thing, the centos build appears to write an empty /etc/resolv.conf | 21:15 |
| clarkb | er centos-minimal | 21:15 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on attach timeout when we created it https://review.openstack.org/308024 | 21:15 |
| clarkb | so just checking if the file exists there may not be quite what we want | 21:15 |
| openstackgerrit | Steve Baker proposed openstack-infra/shade: Implement update_stack https://review.openstack.org/292593 | 21:15 |
| openstackgerrit | Steve Baker proposed openstack-infra/shade: delete_stack add wait argument https://review.openstack.org/292626 | 21:15 |
| openstackgerrit | Steve Baker proposed openstack-infra/shade: Document create_stack https://review.openstack.org/308042 | 21:15 |
| jeblair | fungi: i think i checked and i think your gut is wrong (which would be like the first time) | 21:15 |
| fungi | anteaya: should probably compare which graphite values are being pulled for the sparkline on the zuul status page and make sure the grafana yaml for the zuul-status dashboard is updated to use the right one(s) | 21:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on nova refresh failure https://review.openstack.org/308024 | 21:16 |
| *** ldnunes has quit IRC | 21:16 | |
| fungi | jeblair: my gut is frequently wrong | 21:16 |
| *** ok_delta has joined #openstack-infra | 21:16 | |
| *** ok___delta has joined #openstack-infra | 21:16 | |
| jeblair | fungi: you're wrong about that | 21:16 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: restore tagging ACLs for independent projects https://review.openstack.org/308044 | 21:16 |
| fungi | hrm, good point. my gut doesn't tell me that it's frequently wrong | 21:16 |
| Shrews | mordred: yes, that is much more logical | 21:17 |
| mordred | woot! | 21:17 |
| *** brucet has quit IRC | 21:17 | |
| jeblair | fungi, anteaya: oh -- grafana, not the number on status.o.o | 21:17 |
| jeblair | fungi's gut might be right, let me check | 21:17 |
| *** matrohon has quit IRC | 21:17 | |
| Shrews | mordred: though i think you should log the error before the delete call, in case the delete throws an exception | 21:18 |
| *** dtardivel has quit IRC | 21:18 | |
| jeblair | fungi: i think your gut is right :) | 21:18 |
| fungi | that's not a reputation i need. if people think i'm competent they'll give me more stuff to do | 21:19 |
| jeblair | fungi, anteaya: the graphite number includes all items, not just live ones. the value on the status page is calculated from json. | 21:19 |
| *** tqtran has joined #openstack-infra | 21:19 | |
| *** lakshmiS has quit IRC | 21:19 | |
| jeblair | the number we want is not in graphite; zuul needs a patch to start reporting that (i would just change the calculation for the existing value) | 21:20 |
| jeblair | (because i don't think the value reported is useful) | 21:20 |
| *** EricGonczer_ has joined #openstack-infra | 21:20 | |
| * jeblair owes fungi's gut a beer | 21:20 | |
| fungi | fair point, reporting the number of inactive changes in a pipeline is pretty worthless, so no real benefit to having both and doing math on the other end | 21:20 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on nova refresh failure https://review.openstack.org/308024 | 21:20 |
| mordred | Shrews: great point | 21:20 |
| fungi | jeblair: oh! though we do want inactive changes outside teh window in dependent pipelines, just not inactive dependency changes in independent pipelines | 21:21 |
| fungi | presumably that's easy enough to differentiate on zuul's end around where statsd is getting spammed | 21:21 |
| jeblair | fungi: those are actually different states -- the things outside the window are 'inactive'. the ones that are dependency-only changes not being tested are not 'live' | 21:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 21:23 |
| jeblair | fungi: (it's easy to remember, an live, inactive change may become active. a non-live change will not.) | 21:23 |
| mordred | clarkb: I'm gonna run with that now and see if it prevents the rest of our leaked fips | 21:23 |
| jeblair | fungi: (* there is room for future expansion for zombie changes) | 21:23 |
| *** fawadkhaliq has quit IRC | 21:23 | |
| clarkb | mordred: ok is that in place of the other retry stuff or on top of? | 21:23 |
| mordred | clarkb: on top of | 21:23 |
| fungi | jeblair: oh, much simpler then. for some reason i thought they were conflated, but the json also shows them as separate states | 21:23 |
| mordred | clarkb: I still think retrying is a good plan | 21:24 |
| *** andymaier has joined #openstack-infra | 21:24 | |
| Shrews | ++ | 21:24 |
| mordred | clarkb: although the previous logging showed that we did not have to retry at all | 21:24 |
| *** fawadkhaliq has joined #openstack-infra | 21:24 | |
| clarkb | mordred: so the changes to review are the retry, the cleanup on attach "fail", and the periodic cleanups? | 21:24 |
| mordred | yup | 21:24 |
| jeblair | fungi: yep, both separately exposed. i think we just re-used the dot color. | 21:24 |
| clarkb | mordred: ok reviewing those now | 21:24 |
| fungi | see. all my confusion is because we painted two bike sheds the same color | 21:24 |
| mordred | clarkb: Ihave set them all to have topic shade-nodepool | 21:24 |
| mordred | fungi: can I store recumbant bikes in the shed? | 21:25 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: Run tempest tests with periodic jobs https://review.openstack.org/297038 | 21:25 |
| fungi | mordred: yes, but no unicycles or tricycles | 21:25 |
| *** cody-somerville has joined #openstack-infra | 21:25 | |
| *** flwang1 has joined #openstack-infra | 21:25 | |
| *** hdd has quit IRC | 21:25 | |
| mordred | fungi: you hate people from harvey mudd? | 21:26 |
| *** EricGonczer_ has quit IRC | 21:26 | |
| *** matt-borland has quit IRC | 21:26 | |
| *** ok___delta has quit IRC | 21:26 | |
| *** ok_delta has quit IRC | 21:26 | |
| mtreinish1 | fungi: what about quadracycles or velocipedes? | 21:26 |
| fungi | mordred: one of my friends is actually tenured there | 21:26 |
| fungi | she's way better at math than me | 21:26 |
| mordred | armax, mestery: so - we've tracked down at least one of the neutron fip problems to be that the fip is created properly but never shows up on the nova record | 21:27 |
| mordred | armax, mestery: this is when we create the fip with a port_id argument so that it creates it attached to the given port | 21:27 |
| mordred | armax, mestery: this method of creating fips fails 100% of the time on older clouds - now it seems like it's down to 1 in 100 or so | 21:28 |
| *** infra-red has quit IRC | 21:28 | |
| armax | mordred: interesting | 21:28 |
| *** thorst_ has joined #openstack-infra | 21:28 | |
| armax | mordred: how much older? | 21:28 |
| mordred | armax: yah. so - also, I have not waited for forever to see if the nova record eventually refreshes | 21:28 |
| mordred | 180 seconds is our current wait timeout for that | 21:28 |
| *** fawadkhaliq has quit IRC | 21:28 | |
| mordred | armax: whatever the tripleo cloud is running | 21:29 |
| mordred | I think that's kilo? | 21:29 |
| *** brucet has joined #openstack-infra | 21:29 | |
| *** tonytan_brb is now known as tonytan4ever | 21:29 | |
| armax | I’ll fish through launchpad to see if this is a known issue | 21:29 |
| armax | mordred: unless your searched already? | 21:30 |
| mordred | I have not | 21:30 |
| *** Latha has quit IRC | 21:30 | |
| armax | mordred: ok, bear with me | 21:30 |
| mordred | I've been just beating my head against trying to work through it so far | 21:30 |
| mordred | armax: no rush - I believe we have a good workaround now | 21:30 |
| *** sdake_ has joined #openstack-infra | 21:30 | |
| dstufft | Just a FYI in case it effects y'all in a way that's noticable, I'm migrating all of the file URLs on PyPI right now (it will take some time). Once it's done your bandersnatch mirrors will either pick them up over time (basically, whenever it decides it needs to sync say "requests", it'll get all the new URLs for requests) or you can kick off a full sync to get all the new URLs immediately. Either should be fine though (and I suspect | 21:30 |
| dstufft | this migration to run for many hours) | 21:30 |
| armax | mordred: but this happens on newer clouds too? | 21:30 |
| mordred | mostly wanted to let you know cause it's one of those sorts of things I imagine we hit more than other people | 21:30 |
| mordred | armax: yes - but only sometimes | 21:30 |
| Shrews | mordred: the code on the retry delete stuff looks, good. i think we should add a comment though explaining the need to verify the delete | 21:31 |
| mordred | armax: we also have many successful stuff | 21:31 |
| *** thorst_ has quit IRC | 21:31 | |
| *** thorst_ has joined #openstack-infra | 21:31 | |
| mordred | Shrews: good call | 21:31 |
| armax | mordred: ack, so to make sure I understood the repro steps | 21:31 |
| Shrews | mordred: only b/c when i looked at it again i was like "why the heck are we... oh, right. neutron." | 21:31 |
| *** amitgandhinz has joined #openstack-infra | 21:32 | |
| mordred | armax: once I've got this stack cleared, I'll make you a shorter set of steps that are what would trigger it | 21:32 |
| *** amitgandhinz has quit IRC | 21:32 | |
| mordred | I mean, of course, it's racey, so 'fun' | 21:32 |
| *** mylu has quit IRC | 21:32 | |
| armax | mordred: a) nutron floatingip-create with --port-id results in the FIP not showing up when listing the FIPs with the nova api? | 21:33 |
| armax | mordred: ok, that’d be great | 21:33 |
| *** sdake has quit IRC | 21:33 | |
| armax | mordred: I can only assume that http://lists.openstack.org/pipermail/openstack-dev/2016-April/092613.html is pertitent to this very issue | 21:34 |
| *** brucet has quit IRC | 21:34 | |
| *** Yarboa has quit IRC | 21:34 | |
| mordred | armax: yes. that is exactly right | 21:37 |
| mordred | armax: oh, wait - no | 21:37 |
| mordred | not quite | 21:37 |
| clarkb | mordred: in https://review.openstack.org/#/c/307959/3/shade/openstackcloud.py we are invalidating the cache based on cached info, is that going to function properly? | 21:37 |
| clarkb | pabelanger: this error is consistent, looking more closely it is happening when we cache the bindep fallback packages | 21:37 |
| mordred | armax: a) nutron floatingip-create with --port-id results in the FIP not showing up in the addresses dict in the nova server record | 21:37 |
| pabelanger | clarkb: log? For centos we shouldn't be caching bindep | 21:38 |
| pabelanger | since we don't validate it in the gate | 21:38 |
| mordred | clarkb: yah - it should work - the point there is that we've deleted the server, so if the server had volumes, we need to invalidate the volume list becaue their attachment info will have changed | 21:38 |
| armax | mordred: right | 21:39 |
| clarkb | pabelanger: I probably have my element list wrong. It isn't an issue on nodepool | 21:39 |
| *** mylu has joined #openstack-infra | 21:39 | |
| mordred | armax: cool. just making sure | 21:39 |
| armax | mordred: I don’t think there’s a floating-ip-show in the nova api, is there? | 21:39 |
| pabelanger | clarkb: ok | 21:39 |
| armax | mordred: I assume that even nova floating-ip-list fails? | 21:39 |
| clarkb | pabelanger: centos-7 has cache-bindep in its element list | 21:40 |
| mordred | armax: honestly no clue- the nova server record is how we make decisions about whether a server has ips or not | 21:40 |
| mordred | armax: but I can try to reproduce again and see | 21:40 |
| mtreinish1 | armax: there is a show floating ips api in nova | 21:40 |
| pabelanger | clarkb: right, there is a check in the install script to simple skip caching. We do need bindep to be install. | 21:40 |
| mtreinish1 | armax: don't know if it's in novaclient though | 21:40 |
| pabelanger | clarkb: let me find review for that | 21:40 |
| *** ddieterly has joined #openstack-infra | 21:41 | |
| *** ddieterly is now known as ddieterly[away] | 21:41 | |
| armax | mtreinish1: I can’t see it in the client | 21:41 |
| mtreinish1 | it's just GET os-floating-ips/$floating_ip_id | 21:41 |
| *** ddieterly[away] is now known as ddieterly | 21:41 | |
| armax | mtreinish1: thanks for the tip | 21:41 |
| pabelanger | clarkb: https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/cache-bindep/install.d/50-cache-pkgs#n29 | 21:41 |
| pabelanger | clarkb: that should stop the caching for bindep | 21:42 |
| clarkb | pabelanger: it is failing on line 52 | 21:42 |
| pabelanger | clarkb: broken package I guess? | 21:43 |
| pabelanger | clarkb: it should only be using bindep-fallback.txt at that point | 21:43 |
| pabelanger | let me check nodepool.o.o | 21:43 |
| *** brucet has joined #openstack-infra | 21:43 | |
| pabelanger | clarkb: for now, you should be safe to remove it | 21:44 |
| anteaya | fungi jeblair thank you | 21:45 |
| pabelanger | clarkb: Ya, looks to be working on nodepool.o.o for todays image | 21:45 |
| clarkb | pabelanger: well I want to figure out why it isn't working | 21:45 |
| clarkb | also the du really make the logs impossible to read :( | 21:45 |
| *** |-paul-| has joined #openstack-infra | 21:46 | |
| ianw | clarkb: i'd love to fix that :) | 21:46 |
| mordred | armax: the floating ip created by neutron does show up in floating-ip-list | 21:46 |
| armax | mordred: but no association with the server id? | 21:46 |
| mordred | armax: however, our automation currently deletes the server when we hit the timeout associated with this - so I have no good way to tell you at the moment if it shows up in onva floating-ip-list at the time | 21:46 |
| armax | mordred: ok no worries | 21:47 |
| clarkb | pabelanger: I am going to compare the yum lines to see if the pckages lists are differne,t also mine complates about python34 but nodepool's doesn't | 21:47 |
| clarkb | pabelanger: I wonder if I have a stale repo somewhere | 21:47 |
| pabelanger | clarkb: Oh, wait | 21:47 |
| pabelanger | I know that error | 21:47 |
| pabelanger | Umm | 21:47 |
| pabelanger | yes, stale mirror | 21:47 |
| pabelanger | I've seem that a few times too | 21:47 |
| pabelanger | so, might be worth exporting a specific mirror for your build | 21:48 |
| clarkb | pabelanger: http://paste.openstack.org/show/494716/ | 21:48 |
| *** brucet has quit IRC | 21:49 | |
| clarkb | those lists look different? | 21:49 |
| clarkb | one has swig and unzip the other doesn't | 21:49 |
| *** _amrith_ is now known as amrith | 21:50 | |
| *** cloudtrainme has quit IRC | 21:51 | |
| *** eric_lopez has joined #openstack-infra | 21:51 | |
| *** mriedem has quit IRC | 21:52 | |
| *** mylu has quit IRC | 21:53 | |
| clarkb | I think something is wrong with my build env possibly | 21:53 |
| *** vgridnev has quit IRC | 21:53 | |
| *** elo has quit IRC | 21:54 | |
| clarkb | project-config is up to date in my git cache though and that is where the fallback list comes from | 21:54 |
| clarkb | oh wait | 21:54 |
| clarkb | did we add a bunch of stuff to the fallback list since this last ran? | 21:54 |
| *** rhallisey has quit IRC | 21:54 | |
| clarkb | er last ran on nodepool? | 21:54 |
| clarkb | last run on nodepool is 2 days ago | 21:54 |
| *** rhallisey has joined #openstack-infra | 21:54 | |
| *** kengo_sakai has joined #openstack-infra | 21:55 | |
| *** tchaypo has quit IRC | 21:56 | |
| clarkb | or maybe newer bindep changed behavior? | 21:56 |
| sdake_ | mordred test of the gate shows private fields not being set on internap | 21:56 |
| sdake_ | http://logs.openstack.org/91/306791/5/check/gate-kolla-dsvm-build-ubuntu-source/4500d6d/console.html#_2016-04-18_18_22_28_537 | 21:56 |
| sdake_ | this is today about 1 hour ago | 21:56 |
| *** Vivek_ has quit IRC | 21:56 | |
| clarkb | I don't see any changes to either so now I am really confused | 21:56 |
| *** thorst_ has quit IRC | 21:56 | |
| clarkb | pabelanger: are you able to try and reproduce? or maybe ianw? | 21:57 |
| *** Vivek has joined #openstack-infra | 21:57 | |
| clarkb | I am going to make a build without cache-bindep for now, but I worry that we are broken in there in a very subtle way | 21:57 |
| *** Vivek is now known as Guest32495 | 21:58 | |
| *** mtreinish1 has quit IRC | 21:58 | |
| *** Sukhdev has joined #openstack-infra | 21:58 | |
| pabelanger | clarkb: sure, I can get a build going here after my walk. | 21:58 |
| clarkb | pabelanger: ty ty | 21:58 |
| *** kengo_sakai has quit IRC | 21:59 | |
| *** gokrokve has quit IRC | 21:59 | |
| pabelanger | clarkb: so, you are just adding rpm-distro right? | 21:59 |
| clarkb | pabelanger: adding rpm-distro and removing cache-bindep everything else is the same as what is in nodepool.yaml for centos-7 | 22:00 |
| *** amrith is now known as _amrith_ | 22:00 | |
| *** xyang1 has quit IRC | 22:01 | |
| mordred | sdake_: awesome | 22:02 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:02 | |
| sdake_ | mordred our gate ehas been blocked for 4 days, not relaly sure what to do at this point :) | 22:02 |
| *** Sukhdev has quit IRC | 22:03 | |
| mordred | sdake_: yah - so - sorry about that - tidying up nodepool things like this is the only thing I'm working on - totally thought we had you fixed | 22:03 |
| mordred | sdake_: I'll re-look at it in about 10 minutes | 22:03 |
| sdake_ | would another option be for me to use the non-private interfaces instead in the short term? | 22:04 |
| *** Sukhdev has joined #openstack-infra | 22:04 | |
| clarkb | sdake_: only if the private interfaces do not exist | 22:05 |
| clarkb | otherwise you will likely run into nat issues | 22:05 |
| jhesketh | Morning | 22:05 |
| * sdake_ knew someone was going to say that.. | 22:05 | |
| *** tchaypo has joined #openstack-infra | 22:05 | |
| craige | o/ | 22:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Delete floating IP on nova refresh failure https://review.openstack.org/308024 | 22:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Don't look for invalid image keys for volume invalidation https://review.openstack.org/307959 | 22:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: DNM - Debugging lines for tracking down FIP leak https://review.openstack.org/307913 | 22:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Retry floating ip deletion before deleting server https://review.openstack.org/307439 | 22:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Have delete_server use the timed server list cache https://review.openstack.org/307887 | 22:05 |
| craige | someone just got off a plane... | 22:06 |
| *** ddieterly is now known as ddieterly[away] | 22:06 | |
| sdake_ | snakes on a plane!! | 22:06 |
| anteaya | jhesketh: morning | 22:07 |
| *** mylu has joined #openstack-infra | 22:07 | |
| *** tonytan4ever has quit IRC | 22:07 | |
| mordred | clarkb, Shrews: ^^ k. I think I addressed your comments. also, I fixed tests and even added one | 22:07 |
| *** tonytan4ever has joined #openstack-infra | 22:08 | |
| ianw | clarkb: i can pull everything and kick off a build. i'm a few days behind in my local env | 22:08 |
| Shrews | mordred: adding tests? that's unlike you | 22:08 |
| mordred | Shrews: it's getting late in the day | 22:09 |
| clarkb | ianw: mostly just want to figure out if htis is happeneing everywhere and nodepool will be broken for a bit of if it is just me | 22:09 |
| armax | mordred: looks like the issue went undetected for a while, I can’t seem to find any trace in launchpad | 22:10 |
| mordred | armax: neat! | 22:11 |
| armax | mordred: not sure that’s the adjective I would use, but still :) | 22:11 |
| mordred | armax: it might be worthwhile to try to see if the servers would _eventually_ find the IP if we let them go | 22:11 |
| mordred | armax: I'll see if I can learn if that is the case or not | 22:11 |
| *** tonytan4ever has quit IRC | 22:12 | |
| *** brucet has joined #openstack-infra | 22:12 | |
| *** tonytan4ever has joined #openstack-infra | 22:12 | |
| *** tonytan4ever has quit IRC | 22:12 | |
| *** xarses has quit IRC | 22:12 | |
| armax | mordred: if my memory doesn’t fail me, nova’s cache is refreshed periodically | 22:12 |
| armax | mordred: so I would imagine that the ip would show up at some point | 22:13 |
| mordred | so it's possible that part of this is us having our timeout set lower | 22:13 |
| *** Adri2000 has quit IRC | 22:13 | |
| *** Goneri has joined #openstack-infra | 22:14 | |
| fungi | stepping out to grab some dinner but may be around some later in the evening. if not, night all! | 22:14 |
| *** fawadkhaliq has joined #openstack-infra | 22:14 | |
| mordred | armax: oh - while I'm poking you about things | 22:14 |
| mordred | armax: actually, nevermind. I'm not sure I've verified the next thing yet | 22:14 |
| *** gildub has joined #openstack-infra | 22:14 | |
| armax | mordred: ok | 22:14 |
| mordred | armax: I'll poke you again if/when I have | 22:15 |
| armax | mordred: I am pok-able anytime | 22:15 |
| mordred | \o/ | 22:15 |
| *** mtreinish has joined #openstack-infra | 22:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: send ZMQ start/complete events https://review.openstack.org/308053 | 22:15 |
| jeblair | mordred, clarkb, fungi: ^ there's the zmq stuff for zuulv2.5. next up -- run jobs, then after that, copy logs. | 22:16 |
| *** ayoung has quit IRC | 22:16 | |
| *** Sukhdev has quit IRC | 22:16 | |
| *** alex_xu has quit IRC | 22:16 | |
| armax | mordred: so to be clear, you use neutron directly to allocate fip etc and then use nova to fetch the networking info? | 22:16 |
| *** tpsilva has quit IRC | 22:17 | |
| *** Sukhdev has joined #openstack-infra | 22:17 | |
| armax | mordred: do you happen to have a link to the script that triggers the failure? | 22:17 |
| mordred | jeblair: you are a machine | 22:17 |
| *** brucet has quit IRC | 22:18 | |
| *** alex_xu has joined #openstack-infra | 22:18 | |
| *** brucet has joined #openstack-infra | 22:18 | |
| *** furlongm has quit IRC | 22:18 | |
| clarkb | pabelanger: ianw I wonder if the || true in PACKAGELIST=`echo "$PACKAGES" | /usr/bindep-env/bin/bindep -b -f - || true` is hiding some error from us? | 22:19 |
| anteaya | jeblair mordred one of the best dressed machines I've ever seen | 22:19 |
| *** ybathia has joined #openstack-infra | 22:20 | |
| anteaya | I mean how many other machines even know what I tie clip is | 22:20 |
| clarkb | pabelanger: ianw ok error just happened on a different yum run without cache-bindep I think that does mean I am tlaking to a bad mirror | 22:21 |
| *** furlongm has joined #openstack-infra | 22:21 | |
| *** ianw has quit IRC | 22:22 | |
| clarkb | pabelanger: ianw: doesn't look like centos-minimal supports DIB_DISTRIBUTION_MIRROR? | 22:22 |
| * mordred likes tlaking | 22:23 | |
| *** piet has quit IRC | 22:23 | |
| prometheanfire | dunno if anyone has time to review a bindep change, but if you do... https://review.openstack.org/#/c/307493/ | 22:23 |
| *** ianw has joined #openstack-infra | 22:24 | |
| *** tonytan4ever has joined #openstack-infra | 22:25 | |
| *** salv-orlando has quit IRC | 22:26 | |
| *** piet has joined #openstack-infra | 22:26 | |
| *** sdake_ is now known as sake | 22:26 | |
| *** tonytan4ever has quit IRC | 22:27 | |
| *** avarner has quit IRC | 22:27 | |
| *** brucet has quit IRC | 22:30 | |
| *** brucet has joined #openstack-infra | 22:30 | |
| *** thorst has joined #openstack-infra | 22:30 | |
| clarkb | I am trying it with my dib tmp dir cleaned up | 22:31 |
| clarkb | going towork on adding dib_distribution_mirror support to yum minimal while I wait for that to run | 22:31 |
| *** IlyaG has joined #openstack-infra | 22:32 | |
| *** nwkarsten has quit IRC | 22:33 | |
| greghaynes | clarkb: yep, fedora-minimal too IIRC | 22:34 |
| greghaynes | clarkb: our CI *really* needs that fixed, ETIME | 22:34 |
| *** rfolco has joined #openstack-infra | 22:35 | |
| *** nwkarsten has joined #openstack-infra | 22:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Temporarily disable OVH GRA, as we are seeing failures" https://review.openstack.org/307792 | 22:36 |
| clarkb | greghaynes: I am just copy pastaing the pre-install script from centos7 to centos-minimal to start | 22:36 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Temporarily disable ovh-bhs provider" https://review.openstack.org/307791 | 22:36 |
| clarkb | but I need working image builds in order to test anything, so may not push that naytime soon | 22:37 |
| greghaynes | clarkb: hrm, ok. I remember looking at that a while ago and I am pretty sure I hit some larger issue | 22:37 |
| *** pfallenop has quit IRC | 22:37 | |
| greghaynes | clarkb: why does that break you? | 22:37 |
| clarkb | greghaynes: yum keeps breaking saying there are missing deps | 22:38 |
| greghaynes | That should just cause you to hit upstream mirrors as is, which in theory should work (most of the time) | 22:38 |
| greghaynes | huh | 22:38 |
| clarkb | theory is bad mirror upstream that is lacking those files for some rreason | 22:38 |
| *** ianw has quit IRC | 22:38 | |
| greghaynes | ah, sounds plausible | 22:38 |
| *** xarses has joined #openstack-infra | 22:38 | |
| *** asettle has joined #openstack-infra | 22:39 | |
| *** vhosakot has quit IRC | 22:40 | |
| *** nwkarsten has quit IRC | 22:40 | |
| mordred | woot! patches from this morning landed! | 22:41 |
| anteaya | yay landed patches | 22:42 |
| mordred | clarkb: I ahve restarted nodepool with the latest shade patch series applied | 22:43 |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Ignore uncategorized failures for puppet jobs https://review.openstack.org/307827 | 22:43 |
| mordred | clarkb: also, fwiw, I'm doing this: tail -f /var/log/nodepool/debug.log | grep "DEBUG shade" | grep -v "Waiting ..0 seconds" | grep -v "Endpoint not found" | 22:44 |
| clarkb | mordred: should I hold off on approvin things? | 22:44 |
| mordred | nope | 22:44 |
| mordred | I just updated the local install so that we can track how it's doing with leaks, and see if there are any additional leak vectors we didn't catch yet | 22:45 |
| *** persia has quit IRC | 22:46 | |
| JayF | ...is OpenStackId different than what I use to login to everything else openstack? | 22:46 |
| JayF | went to login for the conference and this is the first time I can ever recall seeing this login screen, and it's on a new domain, smells phishy | 22:46 |
| clarkb | JayF: it is the openid provider used by www.openstack.org and groups.openstack.org and translate.openstack.org | 22:47 |
| maestro1 | with nodepool can i manage windows images, ples help me | 22:47 |
| clarkb | JayF: transition is hard when we hvae to figure out user mappings | 22:47 |
| clarkb | maestro1: my initial guess is no, snapshot image builds rely on ssh and dib image builds rely on dib | 22:47 |
| clarkb | maestro1: I suppose if you could get ssh working on windows then snapshot builds would work | 22:48 |
| JayF | I'm not complaining there's a second login so much as making sure that's a valid thing before feeding it any credentials :) | 22:48 |
| clarkb | maestro1: (maybe) | 22:48 |
| openstackgerrit | Merged openstack-infra/irc-meetings: Re-schedule RefStack weekly team meeting from Monday to Tuesday. https://review.openstack.org/307450 | 22:48 |
| clarkb | JayF: it is a valid thing | 22:48 |
| JayF | I gather that now, thanks :D | 22:48 |
| clarkb | maestro1: as an alternative you could write your own nodepool image builder daemon that speaks the gearman protocol and have it do whatever is necessary for windows | 22:48 |
| *** dalgaaf has quit IRC | 22:49 | |
| clarkb | mordred: help me with https://review.openstack.org/#/c/307959/4 that relies on cached data to know if the cached data is wrong to invalidate? | 22:50 |
| maestro1 | clarkb : Thnks for suggestion, i think if prepare with ssh based then make sense with dib ? or any better way pls | 22:50 |
| clarkb | mordred: is it possible we won't have the volumes cached and thus not clear the cache to pick up the volumes | 22:50 |
| clarkb | maestro1: dib doesn't require ssh, the built in snapshot build process does | 22:50 |
| clarkb | maestro1: basically nodepool assumes you will be able to use ssh or dib to build an image | 22:51 |
| clarkb | maestro1: I do not think that either one is really straightforward with windows | 22:51 |
| *** Goneri has quit IRC | 22:51 | |
| *** dingyichen has joined #openstack-infra | 22:51 | |
| clarkb | maestro1: and I do not have any way to test that either | 22:51 |
| *** |-paul-| has quit IRC | 22:52 | |
| mordred | clarkb: it's not relying on the cached volumes data | 22:53 |
| mordred | clarkb: it's purely saying that if the server it just deleted had volumes | 22:53 |
| mordred | clarkb: that it should invalidate the volume cache | 22:53 |
| clarkb | mordred: ya but the if the server had volumes is relying on the cache | 22:53 |
| *** mriedem has joined #openstack-infra | 22:54 | |
| mordred | sure. it is possible that the server from the cache may have stale info that the server had volumes and therefore we'll invalidate one time that we don't need to | 22:54 |
| mordred | however | 22:54 |
| mordred | we do know that in the case the server has volumes and we don't invalidate that that is definitely wrong | 22:54 |
| maestro1 | clarkb : what work around can start for windwos images | 22:54 |
| mordred | since we know that the volume list will change after the server delete | 22:54 |
| maestro1 | clarkb: disk image create will support windows creation ? | 22:55 |
| clarkb | mordred: I see so it won't make the inconsistency worse it may just continue to propogate until some other thing flushes the cache | 22:56 |
| *** ddieterly[away] is now known as ddieterly | 22:56 | |
| clarkb | mordred: re https://review.openstack.org/#/c/308024/6/shade/openstackcloud.py can we add a test for that case? | 22:56 |
| mordred | clarkb: I was afraid you were going to ask for that | 22:56 |
| clarkb | maestro1: no I don't think disk image builder supports windows | 22:56 |
| mordred | clarkb: I don't wanna write that test. seems gross :) | 22:56 |
| pabelanger | back from walk, firing up local nodepool | 22:56 |
| clarkb | mordred: you have to mock a raise for that error then assert called with on the delete | 22:57 |
| maestro1 | clarkb: yes through disk image builder bin not supports windows platforms :( | 22:57 |
| *** thorst has quit IRC | 22:58 | |
| clarkb | mordred: sorry I -1'd for the test | 23:00 |
| mordred | yah. that's fine | 23:00 |
| mordred | I'll get it in a sec | 23:00 |
| mordred | I'm being whiny | 23:00 |
| *** ddieterly is now known as ddieterly[away] | 23:02 | |
| *** ZZelle_ has quit IRC | 23:02 | |
| *** ams_ has quit IRC | 23:03 | |
| sake | hey folks, we are in our execution phase of remvoing stable/liberty and creating stable/liberty as outllined in phsae 2 of the maiing list | 23:03 |
| *** ddieterly[away] is now known as ddieterly | 23:03 | |
| sake | dhellmann indicates the next step is for stable/liberty to be deleted by the infrastructure team for the kolla repo | 23:04 |
| *** claudiub has quit IRC | 23:04 | |
| sake | should I send an email to the ml for that or request here | 23:04 |
| clarkb | sake: I thought the release team can do that | 23:04 |
| *** persia has joined #openstack-infra | 23:04 | |
| sake | clarkb i'll check with dhellmann he said you ahve to do so | 23:04 |
| *** alex_xu has quit IRC | 23:05 | |
| dhellmann | clarkb : I've always had to ask to have a branch deleted. Do I have permission? | 23:05 |
| dhellmann | I can create them, but I thought the acls were different or something | 23:05 |
| clarkb | dhellmann: lets find out | 23:05 |
| clarkb | dhellmann: at https://review.openstack.org/#/admin/projects/openstack/kolla,branches if you check the box next to liberty can you hit the delete button? | 23:06 |
| *** alex_xu has joined #openstack-infra | 23:06 | |
| dhellmann | no, there's nothing to check and no delete button | 23:06 |
| *** andymaier has quit IRC | 23:07 | |
| clarkb | huh I must be mistaken then | 23:07 |
| clarkb | dhellmann: sake so that commit has been tagged and the tag pushed? | 23:07 |
| *** sdague has quit IRC | 23:07 | |
| dhellmann | clarkb : yep, you could verify by looking for liberty-early-demise in that repo | 23:07 |
| clarkb | https://git.openstack.org/cgit/openstack/kolla/tag/?h=liberty-early-demise there it is | 23:07 |
| dhellmann | http://git.openstack.org/cgit/openstack/kolla/tag/?h=liberty-early-demise | 23:07 |
| dhellmann | heh, yeah | 23:07 |
| clarkb | I will delete the branch now | 23:07 |
| dhellmann | thanks, clarkb | 23:08 |
| clarkb | it has open changes, sake will need to abandon those | 23:08 |
| dhellmann | sake : if you give me the sha now I'll go ahead and create the branch while I'm sitting here | 23:08 |
| sake | thanks a bunch folks, inc0 ready to strangle me for this work not being done :) | 23:08 |
| sake | clarkb will do - do you need that done first? | 23:08 |
| clarkb | sake: yes you cannot delete a branch iwth open changes | 23:08 |
| pabelanger | clarkb: is it just a matter of clicking delete and nothing else? | 23:08 |
| sake | clarkb ok moment | 23:09 |
| clarkb | pabelanger: I like to double check the branch has been tagged and the shas match (which I did with cgit url above) | 23:09 |
| sake | dhellmann the tag to branch from is f10a625a7210c39e7435b77e2cb4d89f14a6b96f | 23:09 |
| clarkb | pabelanger: then click the check box and delete | 23:09 |
| pabelanger | clarkb: ack | 23:09 |
| *** Qiming has joined #openstack-infra | 23:10 | |
| *** ddieterly is now known as ddieterly[away] | 23:10 | |
| clarkb | greghaynes: pabelanger and now I can't build anything with cnetos-minimal yum is super unhappy | 23:10 |
| *** kzaitsev_mb has quit IRC | 23:10 | |
| dhellmann | sake : I have that as tagged 2.0.0.0rc4 correct? | 23:10 |
| pabelanger | clarkb: I should know more in 20mins | 23:10 |
| clarkb | I think I am going to call it quits on centos, this is broken in 3 different ways now | 23:10 |
| dhellmann | sake : let's move back to #openstack-release | 23:10 |
| clarkb | and instead of fixing any of them it just keeps getting worse :( | 23:10 |
| *** tqtran has quit IRC | 23:11 | |
| pabelanger | clarkb: motivation for getting our mirror online I guess | 23:11 |
| clarkb | eh, you can't use it yet anyways | 23:11 |
| pabelanger | that is true | 23:11 |
| clarkb | but I can't test my change to use mirrors because yum is broken | 23:11 |
| clarkb | CRITICAL:yum.verbose.cli.yumdownloader:Yum Error: Error: rpmdb open failed | 23:11 |
| clarkb | which means I can't test if the thing that should fix selinux works | 23:12 |
| *** dimtruck is now known as zz_dimtruck | 23:13 | |
| clarkb | sake: let me know when I should try to delete the branch again | 23:13 |
| sake | clarkb all liberty outstanding reviews just abaonded | 23:13 |
| *** ianw has joined #openstack-infra | 23:13 | |
| clarkb | sake: it is deleted | 23:13 |
| dhellmann | ok, creating the new one | 23:13 |
| sake | clarkb ok thanks | 23:13 |
| *** ianw_ has joined #openstack-infra | 23:14 | |
| *** mylu has quit IRC | 23:14 | |
| ianw_ | clarkb: chown: invalid user: 'jenkins:jenkins' <- this is a new one | 23:14 |
| ianw_ | but, cache-bindep seemed to work | 23:14 |
| *** david-lyle_ has joined #openstack-infra | 23:14 | |
| *** david-lyle has quit IRC | 23:14 | |
| sake | dhellmann ok should be good to go with the sha f10a625a7210c39e7435b77e2cb4d89f14a6b96f | 23:14 |
| dhellmann | sake : done, please approve https://review.openstack.org/308064 | 23:14 |
| *** rbrndt_ has quit IRC | 23:15 | |
| *** mylu has joined #openstack-infra | 23:15 | |
| clarkb | ianw_: ya I ran into same issue with other yum install bits of the build. I think it is related to hwatever mirror I am talking to. But now yum is completely hosted so I dunno what to do | 23:15 |
| clarkb | maybe reinstall yum on my build host | 23:15 |
| *** ddieterly[away] is now known as ddieterly | 23:15 | |
| MarkAtwood | is there an ics or webcal feed for the Austin summit? i knew how to do it when it used sched.org | 23:15 |
| ianw_ | clarkb: hmm, you might have an old lockfile & rpm hanging around, did you bail out during the initial chroot at some point? | 23:16 |
| sake | dhellmann ok tis in progress, we require 2 ack approval of all changes even if they are obviouslly correct:) | 23:16 |
| dhellmann | MarkAtwood : https://gist.github.com/dstanek/83d536af9461eb26d7a19ff73a7391c2 | 23:16 |
| clarkb | ianw_: I did | 23:16 |
| clarkb | does it not clean that up using the exit handler? | 23:16 |
| MarkAtwood | ty dhellmann | 23:17 |
| ianw_ | clarkb: check as root for ~/.rpmmacros & lock file | 23:17 |
| clarkb | ianw_: ya those exist, delete them? | 23:18 |
| ianw_ | clarkb: yep. you're right there should be a cleanup in the exit handler | 23:18 |
| *** fawadkhaliq has quit IRC | 23:19 | |
| *** Sukhdev has quit IRC | 23:19 | |
| clarkb | that seems to make it happy, I am retrying with the osuosl mirror | 23:19 |
| *** fawadkhaliq has joined #openstack-infra | 23:19 | |
| clarkb | (assuming my copy pasta of DIB_DISTRIBUTION_MIRROR functions) | 23:20 |
| *** brucet has quit IRC | 23:20 | |
| *** Sukhdev has joined #openstack-infra | 23:21 | |
| *** sake is now known as sdake | 23:22 | |
| *** Adri2000 has joined #openstack-infra | 23:22 | |
| *** fawadkhaliq has quit IRC | 23:24 | |
| *** Qiming has quit IRC | 23:25 | |
| maestro1 | how to start nodepool in devstack,?, ran nodepoold -d -c /tools/fake.yaml got failed | 23:25 |
| *** kzaitsev_mb has joined #openstack-infra | 23:25 | |
| clarkb | maestro1: nodepool has a devstack plugin in repo | 23:25 |
| clarkb | maestro1: you can just enable to plugin and it should install everyting properly | 23:25 |
| *** brucet has joined #openstack-infra | 23:25 | |
| *** asettle has quit IRC | 23:26 | |
| openstackgerrit | Merged openstack-infra/storyboard: Add offset/limit pagination to subscription events https://review.openstack.org/307847 | 23:26 |
| openstackgerrit | Mike Perez proposed openstack-infra/project-config: Allow the CP spec liaisons to vote on OS-specs https://review.openstack.org/308069 | 23:27 |
| *** asettle has joined #openstack-infra | 23:28 | |
| *** austin81 has left #openstack-infra | 23:29 | |
| *** brucet has quit IRC | 23:29 | |
| *** tqtran has joined #openstack-infra | 23:29 | |
| greghaynes | ianw_: clarkb root.d might also be succeptable to https://review.openstack.org/#/c/259734/ | 23:30 |
| *** jrmyjohn has quit IRC | 23:30 | |
| greghaynes | or have a similar bug | 23:30 |
| greghaynes | (I think I have noticed cleanup fail during that phase before) | 23:30 |
| clarkb | greghaynes: I think it was root.d/ that I hit here | 23:31 |
| greghaynes | Yea, that phase has a similarly odd runner IIRC | 23:31 |
| openstackgerrit | João Vale proposed openstack-infra/jenkins-job-builder: Add Promoted Build parameter. https://review.openstack.org/308072 | 23:31 |
| maestro1 | Thanks Clarkb !! have enabled plugin | 23:31 |
| ianw_ | greghaynes: in this case, just a bad cleanup effort in a tricky bit of code. just testing something that cleans it up | 23:32 |
| *** mylu has quit IRC | 23:32 | |
| *** Swami has quit IRC | 23:33 | |
| greghaynes | ianw_: ok | 23:33 |
| *** mylu has joined #openstack-infra | 23:33 | |
| ianw_ | why am i ianw_? damn linode and their constant ddos attacks (where i have my irc bouncer) | 23:34 |
| *** ybathia has quit IRC | 23:35 | |
| *** ddieterly is now known as ddieterly[away] | 23:35 | |
| *** mylu has quit IRC | 23:36 | |
| *** mylu has joined #openstack-infra | 23:36 | |
| *** maestro2 has joined #openstack-infra | 23:38 | |
| *** elo has joined #openstack-infra | 23:39 | |
| *** eric_lopez has quit IRC | 23:39 | |
| *** maestro1 has quit IRC | 23:40 | |
| *** cburgess has quit IRC | 23:40 | |
| *** kengo_sakai has joined #openstack-infra | 23:41 | |
| openstackgerrit | Merged openstack-infra/elastic-recheck: Add query for testresources release pbr conflict on kilo https://review.openstack.org/307866 | 23:41 |
| *** Clint has quit IRC | 23:42 | |
| *** pleia2 has quit IRC | 23:43 | |
| *** pleia2 has joined #openstack-infra | 23:43 | |
| *** Clint has joined #openstack-infra | 23:43 | |
| *** Clint has quit IRC | 23:43 | |
| *** Clint has joined #openstack-infra | 23:43 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder: yum-minimal : better cleanup of initial yum failure https://review.openstack.org/308077 | 23:43 |
| ianw_ | greghaynes / clarkb: ^ should be a bit more robust | 23:43 |
| clarkb | ok my local build is still failing even after my attepts to use a different mirror | 23:44 |
| clarkb | oh I wonder if it is because I have epel enabled? | 23:45 |
| clarkb | which is required for centos-minimal | 23:45 |
| *** HenryG has quit IRC | 23:45 | |
| clarkb | I thought I removed it earlier :( | 23:45 |
| *** cburgess has joined #openstack-infra | 23:45 | |
| *** niska has quit IRC | 23:45 | |
| *** trey has quit IRC | 23:46 | |
| *** kuromagi^ has quit IRC | 23:47 | |
| *** kuromagi has joined #openstack-infra | 23:48 | |
| *** woodster_ has quit IRC | 23:48 | |
| *** niska has joined #openstack-infra | 23:48 | |
| *** HenryG has joined #openstack-infra | 23:48 | |
| *** maratoid has quit IRC | 23:50 | |
| *** trey has joined #openstack-infra | 23:51 | |
| pabelanger | okay, have nodepool running again. After yakshaving with apscheduler :( https://github.com/agronholm/apscheduler/issues/130 | 23:54 |
| clarkb | pabelanger: I highly recommend virtualenvs :) | 23:55 |
| *** tqtran has quit IRC | 23:56 | |
| pabelanger | clarkb: indeed | 23:56 |
| pabelanger | using stock setuptools on centos7, so that is the issue | 23:57 |
| pabelanger | mind you, I should have do my test with ubuntu-trusty | 23:57 |
| *** david-lyle_ is now known as david-lyle | 23:57 | |
| pabelanger | brb, need a beer | 23:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Ansible launcher: run ansible https://review.openstack.org/308082 | 23:58 |
| *** thorst has joined #openstack-infra | 23:59 | |
| jeblair | this change outputs cows ^ | 23:59 |
| jeblair | and i got it up to gerrit just in time ;) | 23:59 |
| *** lakshmiS has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!