Friday, 2016-09-02

jeblairso i think 'internet. lol.' is the best explanation we have for this issue right now.  i think sdague's change to bump the connect timeout to 30s is worth a shot, but i think that's experimental -- we need to follow up and see if it changes anything.00:00
*** markvoelker has quit IRC00:00
jeblairregardless of whether that improves things, i think the eventual real fix for this may lie in mordred's work for the launcher in zuulv300:00
*** markvoelker has joined #openstack-infra00:01
mordredjeblair: ++00:02
mordredjeblair: I'm +2 on 'internet. lol.'00:02
clarkbwhat changes in the launcher in v3 to avoid ssh problems?00:03
clarkbalso running git-review during db backups is like watching paint dry00:04
* clarkb is trying to start back up again on newton-xenial changes00:04
openstackgerritClark Boylan proposed openstack-infra/project-config: Run multinode jobs on xenial and trusty
clarkbso that we have some ready to go as soon as dhellmann is done with milestone 300:04
*** fguillot has quit IRC00:04
clarkbit turns out I can just troll the zuul stats for getting a good idea of things that need to be converted00:04
clarkblike nova functional tests00:04
*** ijw has joined #openstack-infra00:05
jeblairclarkb: mordred is looking into some fancy pants ansible plugins that will (a) let us do the things we said we were going to do in v3, and (b) give us more control over the part of the process that's handled by the async module now (which has been the source of a lot of our problems, including this one)00:07
clarkbapparently today is the first day of football being back00:08
clarkbI think I need to go plant myself in front of tv now00:08
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run
mordredclarkb: football!00:09
*** ijw has quit IRC00:10
*** vinaypotluri has quit IRC00:12
*** sputnik13_ has quit IRC00:13
*** fguillot has joined #openstack-infra00:16
*** pvaneck has quit IRC00:17
openstackgerritPaul Belanger proposed openstack-infra/nodepool: Retry ssh connect when EOFError is received
*** Genek has joined #openstack-infra00:21
*** xarses has quit IRC00:22
*** thorst has joined #openstack-infra00:24
*** claudiub has quit IRC00:24
*** Genek has quit IRC00:24
*** SumitNaiksatam has quit IRC00:24
*** ijw has joined #openstack-infra00:25
*** ijw has quit IRC00:26
*** annegentle has quit IRC00:31
*** annegentle has joined #openstack-infra00:32
*** ijw has joined #openstack-infra00:35
*** rfolco has quit IRC00:35
*** markvoelker has quit IRC00:35
*** tonytan4ever has joined #openstack-infra00:38
*** xarses has joined #openstack-infra00:40
openstackgerritMerged openstack-infra/project-config: Revert "tripleo-ui: add missing jobs for release management"
*** rfolco has joined #openstack-infra00:40
*** rfolco has quit IRC00:40
*** Julien-zte has joined #openstack-infra00:41
*** fguillot has quit IRC00:41
*** markvoelker has joined #openstack-infra00:44
*** zz_dimtruck is now known as dimtruck00:45
openstackgerritMerged openstack-infra/project-config: Add tripleo-ci-centos-7-ovb-ha-ipv6 into tripleo-ci jobs
*** Jeffrey4l has joined #openstack-infra00:46
*** fguillot has joined #openstack-infra00:47
*** Swami__ has joined #openstack-infra00:49
*** shashank_hegde has joined #openstack-infra00:49
*** shashank_hegde has quit IRC00:49
*** shashank_hegde has joined #openstack-infra00:50
*** kaisers_ has joined #openstack-infra00:51
*** Swami has quit IRC00:52
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Refactor stackviz run
*** kaisers_ has quit IRC00:55
pabelangerianw: mind a review on 36363900:57
ianwpabelanger: ok, didn't we fix that?  must have been a similar one01:00
pabelangerianw: ya, same issue, just different jobs01:01
*** ddieterly has joined #openstack-infra01:02
*** ddieterly is now known as ddieterly[away]01:02
openstackgerritMerged openstack-infra/project-config: tripleo-ui: use nodejs4-publish-to-npm template
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: pingtest: run 'openstack stack failures list' when failure
timothyb89ianw: ah, sorry, I guess I wasn't very clear, it will need 2 fully separate output directories right now01:08
ianwtimothyb89: yep, i think latest gets it01:10
timothyb89ianw: but I could change stackviz-export to write into an existing directory properly01:10
*** ijw has quit IRC01:10
ianwfor grenade, i've just made a copy in old & new01:11
*** ijw has joined #openstack-infra01:11
ianwi'm not sure if we really on want results for "new" ... i think that is what is sort of tacitly happening atm01:11
*** ijw has quit IRC01:11
ianwtimothyb89: also, what's with the zip options?  can we use that to compress everything?01:12
ianwthat might be a good follow-on01:12
timothyb89ianw: right now zipping is broken because the log server sends files with the wrong type01:12
openstackgerritMerged openstack-infra/project-config: Run ansible-role-ubuntu-trusty jobs on master
timothyb89ianw: so we can't decompress them in the browser01:12
timothyb89ianw: working around that is on the agenda but there's some things that need to happen first01:13
ianwtimothyb89: oh, is that something anyone is working on?  i guess it's os-loganalyzers fault?01:13
timothyb89ianw: at this point I'm planning to move all of the decompress+parsing to the client side01:14
*** zshuo has joined #openstack-infra01:15
timothyb89ianw: we'll have ~500KB of content and it will just directly read from the raw log files, compressed or not01:15
*** zhurong has joined #openstack-infra01:16
*** signed8bit_Zzz is now known as signed8bit01:17
pabelangerclarkb: image build done, starting upload to osic-cloud101:18
*** salv-orlando has joined #openstack-infra01:18
*** ijw has joined #openstack-infra01:20
*** spzala has joined #openstack-infra01:20
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Make builder workers proper threads
*** waht has joined #openstack-infra01:21
*** ddieterly[away] is now known as ddieterly01:23
*** ijw has quit IRC01:24
*** spzala has quit IRC01:25
*** ddieterly has quit IRC01:27
*** salv-orlando has quit IRC01:27
*** markvoelker has quit IRC01:28
*** shashank_hegde has quit IRC01:29
*** hsammic has quit IRC01:29
*** Julien-zte has quit IRC01:29
*** Genek has joined #openstack-infra01:31
*** ilyashakhat_mobi has joined #openstack-infra01:31
*** akshai has joined #openstack-infra01:31
*** Genek has quit IRC01:33
*** markvoelker has joined #openstack-infra01:35
*** ilyashakhat_mobi has quit IRC01:35
*** mtanino has quit IRC01:36
*** annegentle has quit IRC01:37
*** akshai has quit IRC01:37
*** aeng has quit IRC01:37
*** Genek has joined #openstack-infra01:38
*** yamamoto_ has joined #openstack-infra01:38
*** mriedem has quit IRC01:38
*** mlavalle has quit IRC01:38
*** SumitNaiksatam has joined #openstack-infra01:39
*** Julien-zte has joined #openstack-infra01:40
*** mriedem has joined #openstack-infra01:40
mriedemthis is approved but looks stuck
*** markvoelker has quit IRC01:40
mriedemdoes that need to be re-approved or something?01:40
*** gyee has quit IRC01:47
*** kzaitsev_mb has quit IRC01:48
openstackgerritIan Wienand proposed openstack-infra/project-config: Run bashate test over devstack-gate too
*** caowei has joined #openstack-infra01:48
*** spzala has joined #openstack-infra01:49
*** signed8bit is now known as signed8bit_Zzz01:49
*** yanyanhu has joined #openstack-infra01:50
*** Thelo_ has joined #openstack-infra01:50
*** mriedem has quit IRC01:51
*** Thelo_ has quit IRC01:52
*** SumitNaiksatam has quit IRC01:53
*** esberglu has joined #openstack-infra01:58
*** zhurong_ has joined #openstack-infra01:59
*** zhurong has quit IRC01:59
*** shashank_hegde has joined #openstack-infra02:01
pabelangerclarkb: ubuntu-xenial online is osic-cloud102:02
pabelangerclarkb: and working properly02:03
*** markvoelker has joined #openstack-infra02:04
*** chlong has joined #openstack-infra02:06
*** tphummel has quit IRC02:06
clarkbpabelanger: cool thanks02:07
*** esberglu has quit IRC02:08
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers
*** tqtran has quit IRC02:15
*** roxanaghe has joined #openstack-infra02:15
fungiif mridem comes back, i added a second approval on 36393702:16
fungiit was initially approved before one of its depends-on merged02:16
*** signed8bit_Zzz is now known as signed8bit02:18
*** rhallisey has quit IRC02:18
*** sdake has quit IRC02:21
*** markvoelker has quit IRC02:23
*** Swami has joined #openstack-infra02:24
openstackgerritMerged openstack-infra/project-config: Run with cells v2 in placement and neutron grenade jobs
*** tonytan4ever has quit IRC02:25
fungiand there it is02:26
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers
fungialso, while i'll be around some tomorrow, my availability for the maintenance later in the day will be determined in part by which direction this hurricane decides to steer itself02:27
*** Swami__ has quit IRC02:27
*** roxanaghe has quit IRC02:28
tonybwhat OS is  running?  Xenial?02:28
tonybfungi: Thanks.02:28
cloudnullpabelanger: w00t!02:28
fungitonyb: spotted an issue with it?02:28
*** tonytan4ever has joined #openstack-infra02:29
*** salv-orlando has joined #openstack-infra02:29
tonybfungi: I don't have a root cause but the symptom is that the automatic u-c bumps are being generated against master as opposed the stable/$branch02:29
*** Genek has quit IRC02:29
tonybfungi: see:
cloudnullevenings btw02:29
*** spzala has quit IRC02:30
tonybfungi: but that commands "work" for me so I was takign a wild swing that it migh have a different version of git02:30
*** Apoorva has quit IRC02:30
fungitonyb: odds are it doesn't have extra magic to map a git tag back to the correct branch02:30
fungilooking now02:30
tonybfungi: oh?  I look forwatf to seeing that magic02:31
fungitonyb: oh, this is an older run...02:31
fungitonyb: i think dhellmann has a change up to fix this02:31
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers
tonybOh  I'll go look02:31
*** salv-orlando has quit IRC02:32
tonybfungi: "older" is in the last 24hours so it's not *that* old :)02:32
fungii'm hunting too, because i think i got busy fixing something else and neglected to review it yet02:32
fungitonyb: yeah, just a few hours ago02:32
*** yamahata has quit IRC02:32
*** ilyashakhat_mobi has joined #openstack-infra02:33
* tonyb sees the fix02:33
fungitonyb: i think?02:33
tonybfungi: thanks02:33
openstackgerritIan Wienand proposed openstack-infra/devstack-gate: Update bashate to 0.5.0
fungitonyb: i've approved it just now--looked correct to me though i didn't run it to confirm02:34
tonybfungi: Thanks.02:35
fungiwe'll just see on the next tag whether it proposes the constraints change against the correct reqs branch02:35
*** Swami__ has joined #openstack-infra02:35
ianwtimothyb89: ok, is it's final form if you wouldn't mind checking the results wrt stackviz.  i'm working on the bashate failures ... it's a silly thing and i've got an upgrade patch in the works02:36
*** Swami has quit IRC02:38
*** Swami_ has quit IRC02:38
*** ilyashakhat_mobi has quit IRC02:38
*** Swami has joined #openstack-infra02:38
tonybfungi: Do you have a few mins to talk about project renames?02:38
*** kaisers_ has joined #openstack-infra02:40
*** kzaitsev_mb has joined #openstack-infra02:44
*** kaisers_ has quit IRC02:45
*** thorst has quit IRC02:45
*** thorst has joined #openstack-infra02:46
openstackgerritMerged openstack-infra/irc-meetings: Add diskimage-builder meeting agenda
openstackgerritMerged openstack-infra/project-config: fix branch handling in clone_repo
*** piet has joined #openstack-infra02:51
*** thorst has quit IRC02:54
*** rlandy|bbl is now known as rlandy02:57
*** dimtruck is now known as zz_dimtruck02:58
*** asettle has joined #openstack-infra03:05
*** fguillot is now known as fguillot_afk03:05
*** signed8b_ has joined #openstack-infra03:06
*** Genek has joined #openstack-infra03:06
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: NPM DSVM jobs are now voting.
*** fguillot_afk is now known as fguillot03:07
*** asettle has quit IRC03:10
*** signed8bit has quit IRC03:10
*** sflanigan is now known as sflanigan_afk03:10
*** cody-somerville has joined #openstack-infra03:11
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Make builder workers proper threads
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic thread
*** aeng has joined #openstack-infra03:15
*** mtanino has joined #openstack-infra03:16
*** piet has quit IRC03:17
*** kzaitsev_mb has quit IRC03:19
*** armax has quit IRC03:19
openstackgerritIan Wienand proposed openstack-infra/release-tools: Change to checked-out directory
*** shashank_hegde has quit IRC03:21
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Make builder workers proper threads
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Add ZooKeeper connection to the BuildWorkers
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Replace watch thread with periodic worker threads
*** signed8b_ has quit IRC03:32
*** vinaypotluri has joined #openstack-infra03:33
*** shashank_hegde has joined #openstack-infra03:37
*** zz_dimtruck is now known as dimtruck03:37
*** salv-orlando has joined #openstack-infra03:39
*** sflanigan_afk is now known as sflanigan03:41
*** cody-somerville has quit IRC03:41
*** coolsvap has joined #openstack-infra03:42
*** gouthamr_ has quit IRC03:45
*** ianw has quit IRC03:46
*** links has joined #openstack-infra03:46
*** tonytan_brb has joined #openstack-infra03:47
*** salv-orlando has quit IRC03:47
*** ijw has joined #openstack-infra03:47
*** Genek has quit IRC03:48
*** rfolco has joined #openstack-infra03:49
*** tonytan4ever has quit IRC03:49
*** ijw has quit IRC03:52
*** thorst has joined #openstack-infra03:52
*** kushal has joined #openstack-infra03:53
*** yfried has joined #openstack-infra03:54
*** fguillot has quit IRC03:57
*** thorst has quit IRC03:59
*** chlong has quit IRC04:01
*** dimtruck is now known as zz_dimtruck04:02
*** yfried has quit IRC04:03
*** Julien-zte has quit IRC04:05
*** cody-somerville has joined #openstack-infra04:05
*** senk has joined #openstack-infra04:15
*** kzaitsev_mb has joined #openstack-infra04:15
*** chlong has joined #openstack-infra04:18
*** kzaitsev_mb has quit IRC04:20
*** ianw has joined #openstack-infra04:21
*** cody-somerville has quit IRC04:21
*** spzala has joined #openstack-infra04:30
*** psachin has joined #openstack-infra04:32
stevemarboom goes the dynamite!04:34
stevemarpost_failures all over the gate04:34
*** spzala has quit IRC04:35
*** csmart has quit IRC04:35
*** Swami__ has quit IRC04:35
*** Swami has quit IRC04:35
*** ilyashakhat_mobi has joined #openstack-infra04:35
stevemarevery job is failing, on the last day of newton-3, damn you murphy's law04:35
*** d34dh0r53 has quit IRC04:36
*** cloudnull has quit IRC04:36
*** rlandy has quit IRC04:36
*** kzaitsev_mb has joined #openstack-infra04:36
*** sigmavirus has quit IRC04:37
*** bhagyashri has quit IRC04:38
*** ilyashakhat_mobi has quit IRC04:39
*** rfolco has quit IRC04:41
*** tonytan_brb has quit IRC04:44
*** _sigmavirus24 has joined #openstack-infra04:44
*** mtanino has quit IRC04:45
*** cloudnull has joined #openstack-infra04:45
*** dkehn_ has quit IRC04:47
*** dkehn has quit IRC04:47
*** d34dh0r53 has joined #openstack-infra04:47
*** yfried has joined #openstack-infra04:47
*** salv-orlando has joined #openstack-infra04:50
*** gongysh has quit IRC04:54
*** caowei has quit IRC04:54
*** yfried has quit IRC04:54
*** nstolyarenko has joined #openstack-infra04:55
*** jerryz has joined #openstack-infra04:56
*** thorst has joined #openstack-infra04:56
*** nstolyarenko has quit IRC04:57
*** senk has quit IRC04:58
*** yamamoto_ has quit IRC04:58
*** salv-orlando has quit IRC04:59
AJaegerinfra-root ^05:00
AJaegertonyb: we do the rename tonight - anything I can help with?05:01
jheskethhmm, that's not good05:01
* jhesketh looks05:01
jheskethlog server is full05:03
*** kzaitsev_mb has quit IRC05:03
*** thorst has quit IRC05:04
*** rajinir has quit IRC05:05
*** caowei has joined #openstack-infra05:06
jhesketh#status alert The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified.05:06
openstackstatusjhesketh: sending alert05:06
*** dolphm has quit IRC05:07
*** netsin has quit IRC05:08
*** fungi has quit IRC05:08
jheskethianw: are you around?05:08
-openstackstatus- NOTICE: The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified.05:09
*** lbragstad has quit IRC05:09
*** ChanServ changes topic to "The logs volume is full causing jobs to fail with POST_FAILURE. This is being worked on, please do not recheck until notified."05:09
AJaegerthanks, jhesketh - and thanks stevemar for notifying05:09
*** gongysh has joined #openstack-infra05:10
*** claudiub has joined #openstack-infra05:11
openstackstatusjhesketh: finished sending alert05:12
*** dolphm has joined #openstack-infra05:13
*** timsim has quit IRC05:14
*** jtomasek|afk is now known as jtomasek05:15
*** yfried has joined #openstack-infra05:16
*** lbragstad has joined #openstack-infra05:18
*** waht has quit IRC05:20
*** coolsvap is now known as coolsvap_05:21
*** dkehn has joined #openstack-infra05:21
*** fungi has joined #openstack-infra05:22
fungiwhat an ideal time for my irc client to lose access to freenode05:23
AJaegermorning, fungi!05:24
fungijhesketh: so merged a week ago but i've just noticed it hasn't taken effect05:24
fungibecause that's not the file we're using any longer. looks like it's been superseded by one in the puppet-openstackci repo but never cleaned up in system-config afterward?05:24
AJaegerthanks a lot for your deep digging into the pip/virtualenv etc. situation!05:24
jheskethfungi: that's the run running according to crontab05:25
*** dkehn_ has joined #openstack-infra05:25
fungijhesketh: right, but you want a lower retention05:25
*** jerryz has quit IRC05:25
fungijhesketh: anyway, my _strong_ recommendation is to run a copy of the log rotation script leaving out the compression bit so it gets to deleting stuff sooner, and dial down the retention05:26
jheskethfungi: ah right, I see what you mean05:26
jheskethfungi: I was just running it, but I didn't take out the compression... I'll do that now05:26
fungijhesketh: there's one in my homedir already set to -mtime 6005:26
jheskethoh even easier05:27
*** chlong has quit IRC05:27
fungii discovered you were already running the script because when i tried to run it the flock immediately bailed on me05:27
jheskethfungi: I've restarted with your script05:27
*** adriant has quit IRC05:27
fungithen some minutes after that my irc client told me it lost contact with freenode, so the ~10 minutes i had been talking to you in here about it, i was really just talking to myself05:27
AJaegerwhat fun, fungi ;)05:28
jheskethfungi: I was also attaching a new volume to grow it, but that'll require taking the partition offline and rsyncing the mess created while it reszes05:28
jheskethfungi: woops :-s05:28
fungijhesketh: i'd be curious how you expect to attach a new volume to it05:28
jheskethfungi: following
fungiafaik 16 xvd devices is the max for the version of xen rackspace is using05:29
*** yfried has quit IRC05:29
jheskethfungi: ah, I was wondering if I was running into a limit because it hasn't attached yet05:29
fungiwe have one for the rootfs, one for the ephemeral disk, and 14x 1tb (the maximum size rackspace will let us create them) cinder devices attached05:29
jhesketh(and had vague memories of that being why we didn't have more alrady)05:29
fungithere is no way i know of to add more local storage to that server05:30
jheskethoh well, the lower retention is working05:31
fungientirely separate from this, ipv6 in rax-iad seems to have gone sideways (which is why i dropped off irc)05:31
fungithough i think that won't impact our ci for a couple of reasons, one of which being that we still have iad completely disabled in nodepool05:32
*** adriant has joined #openstack-infra05:32
jhesketh#status ok Space has been freed up on the log server. If you have POST_FAILURE results it is now safe to issue a 'recheck'05:32
openstackstatusjhesketh: sending ok05:32
*** Julien-zte has joined #openstack-infra05:32
jheskethfungi: there's been some funky networking issues in rax the last few days it would seem05:33
*** javeriak has joined #openstack-infra05:33
jheskethif only we stored our logs in some kind of cloud storage solution... ;-p05:33
fungijhesketh: also, i should have mentioned it, but you probably want to run that log deletion script under screen... took >24 hours to complete the last few times i ran it manually05:33
jheskethfungi: yep, it's on a screen under my user if you wanted to join05:34
fungioh, cool05:34
fungianyway, i'm going to bed. i'm glad i peeked at irc one last time first :/05:34
fungithanks for taking care of this!05:34
jheskethit has all the gzip junk from earlier, but other than that it's working05:34
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"05:35
jheskethfungi: thanks for your help, sorry to have disturbed your night05:35
-openstackstatus- NOTICE: Space has been freed up on the log server. If you have POST_FAILURE results it is now safe to issue a 'recheck'05:35
fungijhesketh: you disturbed nothing--i looked at the computer by choice! ;)05:35
funginight all05:35
jheskeththanks nonetheless :-)05:36
*** javeriak_ has joined #openstack-infra05:37
*** senk has joined #openstack-infra05:37
*** ilyashakhat_mobi has joined #openstack-infra05:37
*** javeriak has quit IRC05:38
openstackstatusjhesketh: finished sending ok05:38
*** timsim has joined #openstack-infra05:43
*** pgadiya has joined #openstack-infra05:46
*** chlong has joined #openstack-infra05:48
*** _nadya_ has joined #openstack-infra05:50
*** _nadya_ has quit IRC05:50
tonybAJaeger: I was just wondering what I need to do in the requirements repo for projects that have been renamed?  I auumes that for a period of time we need to list both old and new packages or the proposal-bot will be confused05:51
*** yamamoto_ has joined #openstack-infra05:55
*** nstolyarenko has joined #openstack-infra05:57
AJaegerFor projects.txt: rename them directly05:59
AJaegerfor global-requirements: As soon as a new release is out: ADd a new entry06:00
AJaegerOnce all projects have imported the sync, remove the old one.06:00
AJaegerThat's what I would do - would that work?06:00
*** csmart_ has joined #openstack-infra06:00
AJaegerremember to backport the projects.txt change.06:01
*** martinkopec has joined #openstack-infra06:01
AJaegerI would ignore global-requiremens in stable branches for this06:01
*** nstolyarenko has quit IRC06:02
*** csmart_ is now known as csmart06:02
*** thorst has joined #openstack-infra06:02
*** bhagyashri has joined #openstack-infra06:02
*** hichihara has joined #openstack-infra06:05
*** aeng has quit IRC06:06
*** hashar has joined #openstack-infra06:07
*** thorst has quit IRC06:09
*** netsin has joined #openstack-infra06:12
*** salv-orlando has joined #openstack-infra06:16
*** kaisers_ has joined #openstack-infra06:17
*** adriant has quit IRC06:19
*** Thelo_ has joined #openstack-infra06:20
*** Thelo_ has quit IRC06:22
*** kaisers_ has quit IRC06:23
*** kzaitsev_mb has joined #openstack-infra06:29
*** woodster_ has quit IRC06:29
*** Julien-zte has quit IRC06:30
*** Julien-zte has joined #openstack-infra06:36
*** thorst has joined #openstack-infra06:36
*** ilyashakhat_mobi has quit IRC06:37
*** rcernin has joined #openstack-infra06:40
*** shashank_hegde has quit IRC06:41
*** Ravikiran_K has joined #openstack-infra06:41
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
*** thorst has quit IRC06:44
*** andreas_s has joined #openstack-infra06:48
*** pcaruana has joined #openstack-infra06:48
*** chlong has quit IRC06:49
*** ilyashakhat_mobi has joined #openstack-infra06:50
*** berendt has joined #openstack-infra06:50
*** gongysh has quit IRC06:51
*** florianf has joined #openstack-infra06:51
*** florianf has quit IRC06:52
*** kzaitsev_mb has quit IRC06:53
*** Julien-zte has quit IRC06:53
*** drifterza has joined #openstack-infra06:55
*** ilyashakhat_mobi has quit IRC06:56
*** shashank_hegde has joined #openstack-infra06:57
*** maishsk_ has joined #openstack-infra07:01
*** tesseract- has joined #openstack-infra07:02
*** signed8bit has joined #openstack-infra07:02
*** maishsk has quit IRC07:03
*** maishsk_ is now known as maishsk07:03
*** florianf has joined #openstack-infra07:03
*** javeriak_ has quit IRC07:05
*** javeriak has joined #openstack-infra07:05
*** signed8bit has quit IRC07:06
*** shashank_hegde has quit IRC07:07
*** kushal has quit IRC07:08
*** HeOS has joined #openstack-infra07:09
*** Thelo_ has joined #openstack-infra07:11
*** sflanigan has quit IRC07:14
*** salv-orlando has quit IRC07:14
*** kaisers_ has joined #openstack-infra07:15
*** mpaolino has joined #openstack-infra07:16
*** javeriak has quit IRC07:19
*** jpich has joined #openstack-infra07:19
*** mikelk has joined #openstack-infra07:20
*** Hal1 has joined #openstack-infra07:21
*** shashank_hegde has joined #openstack-infra07:22
*** shardy_afk has joined #openstack-infra07:22
*** askb has joined #openstack-infra07:23
*** Thelo_ has quit IRC07:26
*** javeriak has joined #openstack-infra07:26
*** spzala has joined #openstack-infra07:30
*** Na3iL has joined #openstack-infra07:30
*** javeriak has quit IRC07:31
AJaegertonyb: still around?07:32
openstackgerritPeter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare.
*** Thelo_ has joined #openstack-infra07:34
*** kaisers_ has quit IRC07:34
*** esikachev has joined #openstack-infra07:34
*** spzala has quit IRC07:35
tonybAJaeger: On and off ... trying to make dinner but also trying to make sure I don't cause probelms for people with -2's07:35
*** ifarkas_afk is now known as ifarkas07:36
AJaeger;) Enjoy dinner!07:36
ttxAJaeger: o/ We have a small problem with the tag-release job: since merged it tries to use zuul-cloner but that is apparently unavailable
ttxI'm tempted to revert the patch while we find a solution07:39
ttxunless you can find out why zuul-cloner is not available07:39
ttxI suspect that's because the signing node doesn't have it installed07:39
AJaegerianw, jhesketh: Here're a few reviews for project-config that should go in IMHO - for newton release work: - could you help landing these, please?07:40
AJaegerttx, mmh, fungi mentioned earlier thta the signing node should have zuul-cloner. Let's tell him so that he can double check later ^07:40
*** matrohon has joined #openstack-infra07:42
*** thorst has joined #openstack-infra07:42
AJaegerttx, zuul-cloner lives on other systems in /usr/zuul-env/bin/zuul-cloner07:42
AJaegerDo you want to send a patch to change the path - and then test?07:42
*** matthewbodkin has joined #openstack-infra07:42
*** nstolyarenko has joined #openstack-infra07:43
ttxAJaeger: if we are not 99% sure of the fix I'd rather just revert the change and unblock master releases07:44
*** nstolyarenko has quit IRC07:44
ttxsince it takes a bit of time to propagate07:44
*** amotoki has joined #openstack-infra07:44
ttxI can propose the revert if you're ok to approve it07:45
AJaegerttx, we need an infra-root to log into the signing node and confirm the path.07:45
AJaegerttx, sure I'm fine to approve to unblock you...07:45
AJaegerttx, reading backscroll: Update th epath - and I change it.07:46
*** oanson has joined #openstack-infra07:46
AJaegerLet me show you what I mean:07:46
*** Ravikiran_K has quit IRC07:46
openstackgerritThierry Carrez proposed openstack-infra/project-config: Revert "fix branch handling in clone_repo"
AJaeger"dhellmann: separate (but related) note... we did eventually get /usr/zuul-env/bin/zuul-cloner onto the signing node so that script can likely be simplified again when you're ready to hack on that"07:47
AJaegerSo, dhellmann simplified too much ;)07:47
ttxah, hmm, that might qualify as 99% sure07:47
*** shardy has joined #openstack-infra07:47
ttxAJaeger: let me propose the path change instead07:48
AJaegerttx, ok. Will wiat for it and approve directly if it's fine...07:48
*** thorst has quit IRC07:48
*** yaume has joined #openstack-infra07:49
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: Implement undercloud upgrade job - Mitaka -> Newton
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton
*** kzaitsev_mb has joined #openstack-infra07:50
*** shardy_afk has quit IRC07:51
*** Julien-zte has joined #openstack-infra07:52
openstackgerritThierry Carrez proposed openstack-infra/project-config: Update zuul-cloner path
AJaegerttx, +2A07:54
ttxit can hardly be worse anyway :)07:54
AJaegerI trust fungi, it will work ;)07:55
*** shashank_hegde has quit IRC07:57
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-infra08:00
*** oanson has quit IRC08:00
openstackgerritMasayuki Igawa proposed openstack/gertty: Add number of changes to the change list view
*** shashank_hegde has joined #openstack-infra08:01
*** hichihara has quit IRC08:03
*** Julien-zte has quit IRC08:05
*** lucas-dinner is now known as lucasagomes08:05
*** asettle has joined #openstack-infra08:07
*** derekh has joined #openstack-infra08:08
*** bauzas is now known as bauwser08:10
*** ggnel_t has joined #openstack-infra08:11
openstackgerritMerged openstack-infra/project-config: Update zuul-cloner path
*** kaisers_ has joined #openstack-infra08:12
*** kzaitsev_mb has quit IRC08:21
*** gongysh has joined #openstack-infra08:23
*** salv-orlando has joined #openstack-infra08:25
openstackgerritlijunli proposed openstack-dev/pbr: When generating local console scripts, should give user a choice to decide whether add -O to ‘executable’, such as '/usr/bin/python -O'
*** maishsk has quit IRC08:25
*** javeriak has joined #openstack-infra08:26
*** maishsk has joined #openstack-infra08:26
*** berendt has quit IRC08:32
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton
*** oanson has joined #openstack-infra08:35
*** yolanda has joined #openstack-infra08:37
*** r-mibu has quit IRC08:38
*** dkehn_ has quit IRC08:38
*** r-mibu has joined #openstack-infra08:38
*** dkehn has quit IRC08:39
*** shashank_hegde has quit IRC08:42
*** berendt has joined #openstack-infra08:44
*** dkehn has joined #openstack-infra08:45
*** thorst has joined #openstack-infra08:46
*** _degorenko|afk is now known as degorenko08:47
openstackgerritPeter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare.
*** shardy has quit IRC08:48
*** gongysh has quit IRC08:48
*** berendt has quit IRC08:50
*** notnownikki has joined #openstack-infra08:50
*** thorst has quit IRC08:54
*** javeriak has quit IRC08:57
*** dtantsur|afk is now known as dtantsur08:57
openstackgerritOpenStack Proposal Bot proposed openstack-infra/project-config: Normalize projects.yaml
*** dkehn_ has joined #openstack-infra08:57
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** Ravikiran_K has joined #openstack-infra08:59
AJaegerinfra-root, is not setup correctly, there's no content. The repo was created yesterday. Could you set permissions correctly, please?09:02
*** Genek has joined #openstack-infra09:03
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Enable release-notes for magnum
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP - Implement overcloud upgrade job - Mitaka -> Newton
*** jaosorior has joined #openstack-infra09:11
openstackgerritPeter Zhurba proposed openstack-infra/project-config: Add repo for openstack/puppet-glare.
*** mptacekx has joined #openstack-infra09:19
andreykurilinAJaeger: hi! Does it possible(if yes - how?) to get privileged access to #openstack-rally chat? it looks like it is managed by openstack-infra09:20
*** Hal1 has quit IRC09:22
*** Na3iL has quit IRC09:25
*** sambetts|afk is now known as sambetts09:28
AJaegerandreykurilin: one of the IRC admins can do this for you - I'm not one. If it's not urgent, better wait until later today09:30
AJaegerandreykurilin: see also
*** _nadya_ has joined #openstack-infra09:31
AJaegerandreykurilin: and yes, our accessbot has privileged access to #openstack-rally so that our bots can send messages and admins have access09:32
andreykurilinAJaeger: it is not urgent. I just want to modify topic of the chat, so I can wait :) whom should I ping later?09:33
AJaegerIf you issue "/msg chanserv access #openstack-rally list", you see who has access.09:33
AJaegerSo, boris-42 and SergeyLukjanov should be able to help you09:34
andreykurilinok, thanks09:34
AJaegerOr one of the others later09:34
andreykurilinthanks a lot09:34
*** yaume has quit IRC09:35
*** javeriak has joined #openstack-infra09:35
*** yanyanhu has quit IRC09:37
*** yamamoto_ has quit IRC09:38
*** yaume has joined #openstack-infra09:39
*** Na3iL has joined #openstack-infra09:39
*** zshuo has quit IRC09:40
andreykurilinAJaeger: btw, where I can look at all available job environment variables? for example I want to access to name of the job from execution script. Does it possible?09:41
AJaegerlook at the log file, see the ansible logs for details09:46
AJaegerandreykurilin: but accessing the name of the job is a really bad idea, that might change.09:46
AJaegerandreykurilin: why do you want to do that?09:47
*** berendt has joined #openstack-infra09:48
andreykurilinAJaeger: actually, I'm not interesting on the name of job, I just want to find some variable which can help to determine where test was launched(at local-pc or at upstream-job). Such information will not affect how tests are launched and how how results are processed(I do not want volkswagen-ci), but it can help to build different formats of logs/reports09:49
andreykurilin*interesting in09:50
AJaegerSo, check the ansible log for which are set09:51
*** thorst has joined #openstack-infra09:52
AJaegerZUUL_* are pretty unique to our CI ;)09:52
*** javeriak has quit IRC09:52
lifelessfungi: quite seriously, noone *needs* editables, though they often desire it.09:54
*** e0ne has joined #openstack-infra09:55
lifelessianw: not sure what you mean by querying how you were installed...09:56
andreykurilinAJaeger: :( okay, I'll parse logs:)09:58
*** bhagyashri has quit IRC09:58
AJaegerlifeless: even with usedevelop=False it fails for me ;(09:58
*** thorst has quit IRC09:58
lifelessAJaeger: :(09:59
AJaegerthe only solution that worked is constraints ;)09:59
lifelesswho woulda thought10:00
*** zhurong_ has quit IRC10:03
openstackgerritPawel Koniszewski proposed openstack-infra/project-config: Add new job to test live migration with grenade
*** markvoelker has joined #openstack-infra10:10
*** rtheis has joined #openstack-infra10:11
*** kevinbenton has quit IRC10:15
*** drifterza has quit IRC10:17
*** kevinbenton has joined #openstack-infra10:17
*** tkelsey has joined #openstack-infra10:19
*** kzaitsev_mb has joined #openstack-infra10:19
*** rods has quit IRC10:22
*** markvoelker has quit IRC10:22
*** tkelsey has quit IRC10:23
*** kzaitsev_mb has quit IRC10:25
*** rods has joined #openstack-infra10:25
*** pblaho has joined #openstack-infra10:26
*** markvoelker has joined #openstack-infra10:30
*** caowei has quit IRC10:31
*** mpaolino has quit IRC10:33
*** kzaitsev_mb has joined #openstack-infra10:37
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Add images profile for openstackci oscc clouds and add it to infracloud
*** markvoelker has quit IRC10:40
rcarrillocruzmordred , pabelanger : automating mirror images bootstrapping on our new clouds ^10:41
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Update local.conf for ironic-multinode case
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Preparing multinode networking for Ironic
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: Setup ssh-key on subnodes for Ironic
AJaegerproject-config cores: Here're a few reviews for project-config that should go in IMHO - for newton release work: - could you help landing these, please?10:49
*** matbu is now known as matbu|lch10:50
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: [Rally] Add new experimental jobs
*** thorst has joined #openstack-infra10:56
*** berendt has quit IRC10:56
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
*** yolanda has quit IRC10:57
*** zul has quit IRC10:58
*** slaweq has quit IRC10:58
*** yolanda has joined #openstack-infra11:01
*** thorst has quit IRC11:04
*** sarob has joined #openstack-infra11:05
*** kzaitsev_mb has quit IRC11:07
*** _nadya_ has quit IRC11:07
*** mptacekx has quit IRC11:07
*** Genek has quit IRC11:07
*** claudiub has quit IRC11:07
*** jgrassler has quit IRC11:07
*** dulek has quit IRC11:07
*** afred312 has quit IRC11:07
*** M-docaedo_vector has quit IRC11:07
*** degorenko has quit IRC11:07
*** lennyb has quit IRC11:07
*** spiffxp has quit IRC11:07
*** wfoster has quit IRC11:07
*** darrenc has quit IRC11:07
*** SotK has quit IRC11:07
*** anteaya has quit IRC11:07
*** sambetts has quit IRC11:07
*** stevebaker has quit IRC11:07
*** eliqiao has quit IRC11:07
*** aderyugin has quit IRC11:07
*** _nadya_ has joined #openstack-infra11:09
*** mptacekx has joined #openstack-infra11:09
*** claudiub has joined #openstack-infra11:09
*** dulek has joined #openstack-infra11:09
*** afred312 has joined #openstack-infra11:09
*** M-docaedo_vector has joined #openstack-infra11:09
*** degorenko has joined #openstack-infra11:09
*** lennyb has joined #openstack-infra11:09
*** spiffxp has joined #openstack-infra11:09
*** wfoster has joined #openstack-infra11:09
*** darrenc has joined #openstack-infra11:09
*** SotK has joined #openstack-infra11:09
*** anteaya has joined #openstack-infra11:09
*** sambetts has joined #openstack-infra11:09
*** stevebaker has joined #openstack-infra11:09
*** eliqiao has joined #openstack-infra11:09
*** aderyugin has joined #openstack-infra11:09
*** sarob has quit IRC11:09
*** rhallisey has joined #openstack-infra11:11
*** rhallisey has quit IRC11:11
*** rhallisey has joined #openstack-infra11:13
*** jgrassler has joined #openstack-infra11:14
*** dingyichen has quit IRC11:15
openstackgerritZara proposed openstack-infra/python-storyboardclient: Add basic docs for python client
*** jkilpatr has joined #openstack-infra11:20
*** javeriak has joined #openstack-infra11:20
*** ramishra has quit IRC11:21
*** ramishra has joined #openstack-infra11:23
*** flepied has quit IRC11:25
*** flepied has joined #openstack-infra11:27
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** matbu|lch is now known as matbu11:28
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
*** yolanda has quit IRC11:29
*** matthewbodkin has quit IRC11:29
*** zul has joined #openstack-infra11:30
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Send notifications to subscribers for worklists
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Make it possible to get worklist/board timeline events via the API
*** florianf has quit IRC11:31
*** spzala has joined #openstack-infra11:31
*** ldnunes has joined #openstack-infra11:32
*** Thelo_ has quit IRC11:33
*** spzala has quit IRC11:35
*** apetrich has quit IRC11:37
*** Thelo_ has joined #openstack-infra11:38
*** baoli has joined #openstack-infra11:39
*** Thelo_ has quit IRC11:40
*** lucasagomes is now known as lucas-hungry11:40
*** Thelo_ has joined #openstack-infra11:43
*** Hal1 has joined #openstack-infra11:43
*** florianf has joined #openstack-infra11:45
*** dprince has joined #openstack-infra11:47
*** gouthamr has joined #openstack-infra11:49
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
*** baoli_ has joined #openstack-infra11:54
*** EricGonczer_ has joined #openstack-infra11:56
*** EricGonczer_ has quit IRC11:56
*** thorst has joined #openstack-infra11:57
*** baoli has quit IRC11:57
*** EricGonczer_ has joined #openstack-infra11:59
*** derekh has quit IRC12:02
*** kzaitsev_mb has joined #openstack-infra12:03
*** pilgrimstack has quit IRC12:03
*** gongysh has joined #openstack-infra12:03
*** tpsilva has joined #openstack-infra12:04
openstackgerritMerged openstack-infra/nodepool: Unsubvert image and flavor caching
*** _sigmavirus24 is now known as sigmavirus12:05
*** sigmavirus has joined #openstack-infra12:05
*** ekarlso_ has joined #openstack-infra12:05
*** pilgrimstack has joined #openstack-infra12:05
*** EricGonczer_ has quit IRC12:06
*** raildo has joined #openstack-infra12:06
*** baoli has joined #openstack-infra12:08
*** baoli__ has joined #openstack-infra12:10
*** ekarlso_ has quit IRC12:11
*** salv-orlando has quit IRC12:11
*** flaper87 has quit IRC12:11
*** baoli_ has quit IRC12:12
*** matthewbodkin has joined #openstack-infra12:12
*** ilyashakhat has joined #openstack-infra12:12
*** asettle has quit IRC12:13
*** baoli has quit IRC12:13
*** zul_ has joined #openstack-infra12:15
*** trown|outtypewww is now known as trown12:15
*** gordc has joined #openstack-infra12:15
*** askb has quit IRC12:15
*** zul_ has quit IRC12:16
*** zul_ has joined #openstack-infra12:17
*** esikachev has quit IRC12:17
*** zul has quit IRC12:17
openstackgerritVasyl Saienko proposed openstack-infra/devstack-gate: DO NOT REVIEW
rcarrillocruzclarkb: , as discussed the other day, to just run the launcher against a particular clouds from our clouds_layouts.yaml12:21
*** javeriak has quit IRC12:23
*** berendt has joined #openstack-infra12:24
*** rlandy has joined #openstack-infra12:24
mordred#status log restarted nodepool with the latest shade and nodepool changes. all looks well - floating-ips, images and flavors are not being hammered12:24
openstackstatusmordred: finished logging12:24
mordredinfra-root: ^^12:24
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
*** andreas_s has quit IRC12:26
*** rodrigods has quit IRC12:27
*** rodrigods has joined #openstack-infra12:27
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Move the clouds_layouts.yaml file
*** flaper87 has joined #openstack-infra12:29
*** flaper87 has quit IRC12:29
*** flaper87 has joined #openstack-infra12:29
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add release jobs for TripleO Quickstart
*** rfolco has joined #openstack-infra12:30
AJaegermordred, rcarrillocruz : Here're a few reviews for project-config that should go in IMHO - for newton release work: - could you help landing these, please?12:34
*** electrofelix has joined #openstack-infra12:34
*** javeriak has joined #openstack-infra12:38
*** annegentle has joined #openstack-infra12:38
*** lucas-hungry is now known as lucasagomes12:38
*** javeriak has quit IRC12:38
openstackgerritMerged openstack-infra/project-config: Add cinder-stable-maint to os-brick
*** zul_ has quit IRC12:42
openstackgerritMerged openstack-infra/project-config: Add nova-stable-maint to os-vif gerrit ACL
*** jaosorior has quit IRC12:43
*** jaosorior has joined #openstack-infra12:43
AJaegerthanks, rcarrillocruz12:43
*** zul has joined #openstack-infra12:43
*** zul has quit IRC12:45
*** zul has joined #openstack-infra12:45
*** zhurong has joined #openstack-infra12:46
*** zynzel has joined #openstack-infra12:46
zigomordred: <--- As per our discussion yesterday with fungi.12:47
rcarrillocruzpabelanger: you know what, following what fungi said yesterday... how about  deploying chocolate with Mitaka?12:47
rcarrillocruzi mean12:47
openstackgerritMerged openstack-infra/project-config: fix networking-hyperv release acls
rcarrillocruzwe could start provisioning12:47
rcarrillocruzdeploying the cloud12:48
rcarrillocruzfinish up things during mid cycle12:48
rcarrillocruzand will be a great exercise to redeploy a whole cloud when newton is out the door and we decide it's ok to go with it12:48
rcarrillocruzi see no point on having all this HW idle for now12:48
amrithrcarrillocruz, thanks for your help yesterday. got to the bottom of the problem, merged a fix, all humming along again.12:48
mordredzigo: awesome12:48
rcarrillocruzand putting RC1 or whatever it's risky12:49
rcarrillocruzamrith: you're very welcome!12:49
*** alaski is now known as lascii12:49
rcarrillocruzRC1 Newton that is12:49
zigomordred: Cheers!12:49
amriththx rcarrillocruz12:49
amrithmordred, g'morning12:49
mordredmorning amrith !12:49
amrithis today better?12:49
zigoAJaeger: Could you +2 workflow this one too, so we can move ahead and release Newton? :)
jheskethAJaeger: sorry I was out this evening... looks like rcarrillocruz beat me to it on those project-config patches needed for newton12:49
AJaegerzigo: not right now. Could you please look at and clean up those problems?12:50
zigoAJaeger: I did have a look, but I'm not sure how. I did remember cleaning-up the upstream repo though.12:51
*** gouthamr_ has joined #openstack-infra12:51
openstackgerritMerged openstack-infra/project-config: do not error if a branch already exists
zigoAJaeger: Do I still need to approve the removal of deb-websockify if the repo was cleaned?12:52
* zigo checks again with git fsck12:52
*** annegentle has quit IRC12:52
zigoAJaeger: git fsck says "dangling blob 9e758039948eef26684cd776824a2accf08a295f"12:52
fungiAJaeger: ttx: oh, yep when i reviewed 364497 i missed that it didn't specify the path to zuul-cloner. it was indeed supposed to use /usr/zuul-env/bin/zuul-cloner like on our other nodes12:53
zigoThat's not a big issue, it's should be fine no?12:53
zigoOr should I still fix it?12:53
skraynevrcarrillocruz: hi. do you have rights to add me to group of reviewers for new (recently created) project?12:53
rcarrillocruzi do12:53
rcarrillocruzwhich group is12:53
fungiAJaeger: ttx: also reverting wouldn't have been much better since prior to that change it was checking out the wrong branch for some tags12:53
ttxfungi: yes, we got it all fixed12:54
ttxWorks almost perfectly now12:54
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add release jobs for TripleO Quickstart
ttxJust needs
*** kgiusti has joined #openstack-infra12:54
ttxfungi: then life will be perfect12:54
*** jcoufal has joined #openstack-infra12:55
fungiAJaeger: is replicating successfully now. same as usual, manage-projects got an http error response back from the github project creation api call, so thought it failed and didn't bother attempting to set permissions for gerrit replication into it12:55
*** gouthamr has quit IRC12:55
skraynevrcarrillocruz: groups,members,members,members according patch
*** woodster_ has joined #openstack-infra12:56
rcarrillocruzskraynev: done12:57
fungiandreykurilin: i just made sure that boris-42 (as current ptl) has sufficient permissions to add other admins to #openstack-rally through chanserv now12:58
*** kushal has joined #openstack-infra12:58
*** Goneri has joined #openstack-infra12:58
zigofungi: Can you +2 workflow so we can go ahead and release Newton ?
*** Thelo_ has quit IRC12:59
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
*** dtardivel has joined #openstack-infra12:59
*** piet has joined #openstack-infra13:00
*** andymaier has joined #openstack-infra13:00
AJaegerzigo: I don't know what needs to be done - right now that repo is in broken state. Please talk with clarkb and figure it out.13:03
zigoAJaeger: Will do.13:03
*** berendt has quit IRC13:04
*** psachin has quit IRC13:04
AJaegerfungi, thanks for fixing github - then the gerrit/projects upstream cleanup change is fine...13:04
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove experimental bandit job from anchor
dhellmannAJaeger : thanks for your help with that zuul-cloner issue earlier.13:05
* dhellmann makes a note to pay closer attention to paths when fungi says things to him13:05
*** salv-orlando has joined #openstack-infra13:06
fungiAJaeger: so usedevelop=False works for me _if_ i don't add skipsdist=True like you did in 364671,213:07
*** xyang1 has joined #openstack-infra13:08
*** jaosorior has quit IRC13:09
*** tongli has joined #openstack-infra13:09
*** signed8bit has joined #openstack-infra13:10
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Add python-db-jobs to Glare
*** yuval has joined #openstack-infra13:12
openstackgerritSergey Skripnick proposed openstack-infra/project-config: Add python-db-jobs to Glare
*** asettle has joined #openstack-infra13:13
*** spzala has joined #openstack-infra13:16
*** mriedem has joined #openstack-infra13:16
*** maishsk_ has joined #openstack-infra13:17
openstackgerritMerged openstack-infra/project-config: Normalize projects.yaml
*** rossella_s has quit IRC13:18
fungiAJaeger: from the tox logs, it looks like it never bothered to directly install openstackdocstheme from source in a separate step (perhaps because it saw it already installed by one of the dependencies?)13:18
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: NPM DSVM jobs are now voting.
*** rossella_s has joined #openstack-infra13:19
*** asettle has quit IRC13:19
*** maishsk has quit IRC13:19
*** maishsk_ is now known as maishsk13:19
*** piet has quit IRC13:19
skraynevrcarrillocruz: great! thank you :)13:21
*** yuval has quit IRC13:21
*** eharney has joined #openstack-infra13:22
*** zhurong has quit IRC13:23
fungittx: dhellmann: i've just approved 36441713:23
*** mmedvede has quit IRC13:23
Zaraquestion-- I want to test my rudimentary storyboard commandline interface against storyboard-dev, but I get 'certificate verify failed' errors. I'm guessing it's because the certificates are self-signed. Is there anything we can do to make it play nicely?13:25
openstackgerritPawel Koniszewski proposed openstack-infra/project-config: Add new job to test live migration with grenade
fungii need to step away again for more storm prep, but i hope to still be around to work on our rename maintenance planning (and around for the maintenance itself)13:25
*** dstanek has quit IRC13:25
openstackgerritAleksey Zvyagintsev proposed openstack-infra/project-config: murano-pkg-check: add coverage job
fungiZara: depends a bit on how your tool is implemented, but there should be ways to make it okay with self-signed certs13:26
pabelangerrcarrillocruz: yes, if we can make sure the hardware in infracloud-chocolate is happy but deploying a cloud, I say we actually redeploy at the midcycle to newton RC113:26
fungiZara: is teh source code public somewhere i can skim?13:26
dhellmannfungi : thanks!13:26
*** zz_dimtruck has quit IRC13:26
Zarafungi: sure, it's pretty much a wrapper for the python client,
Zarawell, the python client would be more useful to link13:27
fungidhellmann: i assume the outer bands of this storm have already reached you--hope you're on high ground!13:27
*** madorn has quit IRC13:27
*** wendar has quit IRC13:27
*** kencjohnston has quit IRC13:27
openstackgerritMerged openstack-infra/project-config: Add merge commit ACL for packaging-deb
fungiZara: got it, so storyboardclient is raising an exception?13:27
AJaegerfungi, ah - thanks! New test up...13:28
AJaegerfungi, all the best for the storm! Hope it's not too stormy13:28
dhellmannfungi : I think the worst of it is supposed to pass southeast of us, but we're supposed to see some from the edges. I hope you weather it ok.13:28
Zarafungi: yeah, paste here:
*** mptacekx has quit IRC13:29
rcarrillocruzbkero: the gotty thing to share terminals over a browser is superawesome13:29
rcarrillocruzthx for the suggestion13:29
mrhillsmanmorning/evening - just checking if things in cloud8 are ok13:29
AJaegerfungi, indeed: THANKS a lot!13:29
dhellmannfungi : it looks like the leading edge has already reached your area13:29
fungiAJaeger: we've already moved most stuff inside (was up at sunrise carrying deck furniture around), so not much else to worry about unless the waters crest our ground floor in which case i need to bring a bunch of things upstairs13:29
*** pgadiya has quit IRC13:29
*** wendar has joined #openstack-infra13:29
*** jcoufal has quit IRC13:30
fungialready have all the storm shutters closed13:30
AJaegerfungi, hope it's not that high...13:30
fungire saying the storm surge could be up to a meter above ground level, but i think that's only likely on the ocean side of the island13:31
dhellmannfungi : what do you do with your car?13:31
fungiunless we end up with winds pushing water into the inlet13:31
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Tweak 'about' page
*** kencjohnston has joined #openstack-infra13:32
fungidhellmann: throw bikes in the trunk, drive cars to a store parking lot about a mile waay that's on higher ground up on the dunes, then bike home and hang the bikes up13:32
dhellmannfungi : smart13:32
* fungi remembers he needs to check tire pressure on the bikes :/13:32
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
fungialso, it's only a mile, so easy enough walk if the weather hasn't gotten terrible yet13:33
*** zz_dimtruck has joined #openstack-infra13:33
fungiheck, the entire island is only a mile across at the widest point anyway13:33
*** zz_dimtruck is now known as dimtruck13:33
*** madorn has joined #openstack-infra13:34
*** dstanek has joined #openstack-infra13:34
* Zara wondered 'if one side of the island is the ocean side, what's on the other side?' then figured it was an island near the mainland and there was less sea on the other side...13:35
openstackgerritMerged openstack-infra/project-config: do not fail if there are no constraint updates to push
fungimost of our flood risk on this side is likely to be over the weekend after the storm passes once all the water it dumps on the mainland flows down into the sound, if it flows in faster than it can flow out the inlet into the ocean13:35
fungiZara: barrier island, so it separates a modest sized body of water from the atlantic13:37
*** senk has quit IRC13:38
fungibasically a very narrow strip of land (quite literally a sandbar) that stretches for a couple hundred miles13:38
openstackgerritDoug Hellmann proposed openstack/os-client-config: Update reno for stable/newton
fungiwith a few inlets at various points13:38
fungiyep, that13:39
openstackgerritLucas Alvares Gomes proposed openstack-infra/elastic-recheck: Add query for bug 1548652
openstackbug 1548652 in VirtualBMC "calls to vbmc via ipmitool sometimes time out" [High,Confirmed]
fungii'm there at the end of teh albemarle sound where it says "kill devil hills"13:39
Zara:0 that sounds awesome13:40
Zarawell, except for the storms13:40
*** jcoufal has joined #openstack-infra13:40
fungiyou get used to them13:40
fungii'm less worried about this one since it's not coming at us directly off the atlantic13:40
*** jcoufal has quit IRC13:41
*** links has quit IRC13:41
fungiactually storm weekends can be kinda nice since they call for mandatory tourist evacuations on most of the islands here ;)13:42
*** jcoufal has joined #openstack-infra13:42
*** Na3iL has quit IRC13:43
fungiespecially some of the islands to the south of us reachable only by boat, because the ferries have to shut down and they don't want to have a ton of tourists stranded there becoming a nuisance13:43
Zaraso a good time for a swim?13:43
fungican be sometimes, but usually you don't want to wander out into the ocean for a while after one of these passes through13:44
fungidebris, jellyfish, possibly lingering dangerous currents13:45
fungigreat for shell hunting though13:45
*** akshai has joined #openstack-infra13:45
*** deadnull has joined #openstack-infra13:46
*** kevinbenton has quit IRC13:47
*** hsammic has joined #openstack-infra13:47
Zaragah, now I want to live on the beach13:48
*** hongbin has joined #openstack-infra13:48
fungiZara: so looking at your traceback, there are a few options. complicated a bit by the old oslo-incubator code in storyboardclient though13:48
Zarayeah, we're looking to remove that anyway in the future13:49
lucasagomeshi all, if you have time minda taking a quick look at ? It just changes a wildcard in the function that copies the logs in the Ironic jobs to also include the logs from the deployment ramdisk13:49
lucasagomesthank you13:49
fungii need to look a little closer at the _get_all() method in there to see how hard it would be to pass extra requests parameters into openstack.common.apiclient.client13:50
Zara(got to the 'make a story' step!/story/2000707)13:50
*** kevinbenton has joined #openstack-infra13:50
fungiZara: so anyway, the "nice" solution to this would be to add a parameter passed through down to the requests.get() call to add a cert parameter, then plumb that up through into your cli as an option like --cert-path13:51
*** mmedvede has joined #openstack-infra13:52
*** matbu is now known as matbu|brb13:52
fungioptionally (or additionally) make it possible to do a verify=False in the requests.get() call with a sufficiently scary sounding option name like --insecure13:53
fungihowever a workaround would be to set the CA_BUNDLE envvar in the calling environment to a file with a copy of the server cert (i don't recall the exact dance steps there)13:54
*** gouthamr_ has quit IRC13:54
*** zhurong has joined #openstack-infra13:54
*** cardeois has joined #openstack-infra13:54
*** e0ne has quit IRC13:55
*** bhunter71 has joined #openstack-infra13:55
*** e0ne has joined #openstack-infra13:55
*** gouthamr has joined #openstack-infra13:55
fungioh, ignore what i said about the cert option to requests.get(), that's for specifying a client certificate. the verify option does double-duty either to disable verification or to specify the path to the trusted cert bundle13:56
bhunter71a question;  in gerrit, can you add a user (email address) to a group _before this user has ever logged in_?    I've historically required that users log in one time before being granted extra privs,13:56
fungibhunter71: depends on how you create users in gerrit. if you're talking about, new users are created on initial login and don't exist before then13:57
Zarafungi: okay, thank you! :) I'll have a poke around.13:57
bhunter71fungi: I'm not particularly talking about, but my gerrit acts the same as you describe.13:58
fungibhunter71: yeah, if you're using automatic account creation with openid authentication then i don't know of any way to pre-add them to a group13:58
openstackgerritEmilien Macchi proposed openstack-infra/project-config: Add release jobs for TripleO Quickstart
bhunter71fungi; is there a way you know of to pre-create the account , yet use openid auth?13:59
*** senk has joined #openstack-infra13:59
*** Julien-zte has joined #openstack-infra13:59
bhunter71assuming I know all the details - name, email, etc.14:00
*** ccarmack has joined #openstack-infra14:00
rcarrillocruzbhunter71: i'm afraid not14:00
fungibhunter71: only if you also know what their openid url will be... then you could precreate the account and add the openid as an account_external_ids row14:00
rcarrillocruzgerrit does not have a bootstrap mechanism14:00
rcarrillocruzyou would need to inject it to the DB14:00
rcarrillocruzas fungi said14:00
rcarrillocruzthere was a project around that had that code14:00
rcarrillocruzlet me search14:00
fungibhunter71: the way openid auto account creation works, if the user logs in with an openid url gerrit hasn't previously seen, gerrit will create a completely new account for it (even if there is already an existing account with the same name, e-mail addresses, et cetera)14:01
fungifor similar reasons, we end up with duplicate accounts somewhat frequently because users do things that cause their openid url to change14:01
bhunter71fungi/rcarrillocruz.   Thanks so much, this is very helpful!14:02
andreykurilin fungi: thanks. will ask him14:02
*** mat128 is now known as mat128|away14:02
fungibhunter71: so in short, _if_ you control the openid service you're using such that you can predict a user's openid, then you _could_ add it to gerrit in advance14:03
*** tesseract- has quit IRC14:03
fungibhunter71: or put a different way, if you want to manually associate a known openid with an account, you can simply add it to the account_external_ids table in the database before they log in for the first time14:04
bhunter71you guys are the freaking best.   I <3 #openstack-infra14:04
fungior you can use an update query later to move that openid to a different account_id14:04
bhunter71thanks fungi14:04
anteayafungi: did the release team get everything finished that they need to get finished?14:05
*** bin_ has joined #openstack-infra14:05
*** kaisers_ has quit IRC14:05
anteayaor have we checked with them yet?14:06
Zarafungi: thanks, that worked \o/14:06
fungianteaya: yeah, as you've discovered, it's release meeting time right now14:07
anteayafungi: thanks14:07
*** vhosakot has joined #openstack-infra14:08
*** jamesdenton has joined #openstack-infra14:09
*** mtanino has joined #openstack-infra14:10
*** tesseract- has joined #openstack-infra14:13
anteayamorning zaro14:14
*** deadnull_ has joined #openstack-infra14:14
openstackgerritDmitry Tantsur proposed openstack-infra/project-config: Fix ironic-inspector-client tempest job to actually use the client from git
*** rbrndt has joined #openstack-infra14:16
*** javeriak has joined #openstack-infra14:16
*** deadnull has quit IRC14:18
*** Na3iL has joined #openstack-infra14:19
*** kaisers_ has joined #openstack-infra14:19
*** tkelsey has joined #openstack-infra14:22
*** Julien-zte has quit IRC14:22
*** matbu|brb is now known as matbu14:23
*** Julien-zte has joined #openstack-infra14:24
*** fguillot has joined #openstack-infra14:25
*** asettle has joined #openstack-infra14:25
*** annegentle has joined #openstack-infra14:26
*** tkelsey has quit IRC14:26
*** matt-borland has joined #openstack-infra14:29
*** senk has quit IRC14:29
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
*** martinkopec has quit IRC14:31
*** andymaier has quit IRC14:33
*** _nadya_ has quit IRC14:34
pabelangerrcarrillocruz: are any of the 3 held nodes in nodepool from debugging failures from yesterday?  If so, can they be deleted?14:34
rcarrillocruzthey can be deleted, yeah14:35
pabelangerwill do that now14:35
*** esberglu has joined #openstack-infra14:35
*** sputnik13_ has joined #openstack-infra14:35
rcarrillocruzi held one for checking the location of the cache/files/cirros-blah14:35
*** inc0 has joined #openstack-infra14:35
inc0pabelanger, did you try to start gatejobs on our little cloud?14:36
*** javeriak_ has joined #openstack-infra14:36
krotscheckOh gerrit, how I love thee, let me count the days.14:36
krotscheck...of waiting.14:36
*** e0ne has quit IRC14:37
*** javeriak has quit IRC14:37
inc0krotscheck, gertty helps.14:37
pabelangerinc0: not yet. I think we are waiting for releases to be finished for the day14:37
*** e0ne has joined #openstack-infra14:37
krotscheckinc0: tty is for crusty old ops people.14:37
inc0ok, fair enough, let me know if you need any help14:38
inc0krotscheck, crusty old ops people know what works.14:38
*** eharney has quit IRC14:38
krotscheckinc0: Yes, but it takes more than one lick to get to the toffee center.14:38
*** martinkopec has joined #openstack-infra14:38
* krotscheck will leave you with that mental image. 14:39
fungithanks for the url reminder, anteaya!14:39
fungialso i haven't checked yet, but someone posted a late entry for renames to the infra ml14:40
anteayaoh, I will look14:40
fungiif we get a chance to squeeze that one into the mix14:40
openstackgerritMatthew Bodkin proposed openstack-infra/storyboard-webclient: Make sidebar submenu the same length as sidebar
anteayaalso gerrit is slow if that isn't already a known thing14:40
inc0ok...Smaug...we (Kolla) competed with them for Koala mascot. Because apperantly they were renaming and new name was something connected to I ask the hell Karbor is connected in any shape or form to Koalas?14:40
anteayado we have to swear?14:41
*** matbu is now known as matbu|mtg14:41
anteayacan we not just ask a question?14:41
fungibackwards it's "robrak"14:42
*** jcoufal has quit IRC14:42
fungistill no connection14:42
openstackgerritAndrey Kurilin proposed openstack-infra/project-config: [Rally] Add new experimental jobs
inc0swearing is a form of empowering emotion, it is a poetic tool in our disposals.14:42
inc0let's use it proudly14:42
anteayafungi: did you have a chance to chat with tonyb at all about the requirements rename patch?
*** jcoufal has joined #openstack-infra14:42
anteayainc0: really, that is an interesting stance14:43
* fungi would prefer to keep discussions here less "emotionally empowered" as well ;)14:43
krotscheckI have a +2A patch with a finally-merged depends-on thats not getting into the gare. Can someone toggle the +A for me?
*** kushal has quit IRC14:43
fungikrotscheck: done14:43
krotscheckfungi: thanks14:44
openstackgerritMerged openstack-infra/project-config: wait to publish releases.o.o until after tagging
fungidhellmann: ttx: ^ merged14:44
bauwserhey -infra, scrolling backwards doesn't show me people reporting having Gerrit issues, but I just got a 502 after 20mins of high latency14:45
anteayabauwser: thank you, gerrt is slow for me as well14:45
anteayaI don't think any of us have gotten around to looking at yet, you are welcome to if you wish14:46
inc0anteaya, look, there is also a greek word for it! and if there is ancient greek word for something it has to be wise and true14:46
anteayainc0: do you really want to waste my time with this?14:46
anteayayou can't see I am busy?14:46
anteayahow's that for emotionally empowered?14:47
*** esikachev has joined #openstack-infra14:47
*** links has joined #openstack-infra14:47
openstackgerritRicardo Carrillo Cruz proposed openstack-infra/system-config: Update infracloud documentation to remove old geo information
HeOSHello, infra-team! If this request has +2, could you please merge it into repository?
inc0anteaya, well, it's your time;) anyway good point, I have handful of reviews to make, ah glorious rc season14:48
HeOSI'll really appreciate your help.14:48
anteayainfra-root, can someone check javamelody for the gc rate please? cpu usage appears to be up:
*** e0ne has quit IRC14:49
*** Julien-zte has quit IRC14:51
anteayafungi: let me know once you have decided your rename workflow of choice14:52
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
jeblairfungi, anteaya: gerrit gc time is high14:53
anteayajeblair: thank you14:53
fungiyep, i just started pulling that up14:53
jeblairstarted maybe 30 mins ago and is continuing14:53
fungialso show-caches --show-threads reports we're around 66-67 httpd threads (we set our max to 100) so not responsible for the 5xx errors14:54
fungii saw a few very small gc time spikes late yesterday but was hoping it would hold off until the maintenance14:54
fungilooks like we made it about a week since the last restart14:55
fungilast restarted thursday, august 2514:56
*** kushal has joined #openstack-infra14:56
jeblairfungi: yeah, *maybe* the spike around 10:00 was a predictor?  <shrug>14:56
*** piet has joined #openstack-infra14:56
irtermite@all where do I find the schedule of infra jobs that run? I ask, as there is a request to upgrade the switches the Rackspace RPC OSIC cluster is running on. There is a request for Central Time next week Wednesday 00:00 – 06:00 and Thursday 00:00 – 06:0014:56
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types
irtermiteHow impactful would this be to your efforts?14:56
irtermitecloudnull: ^^ for awareness14:57
cloudnullcool, Nexus upgrade?14:57
cloudnullbetter RA support hopfully14:57
fungiirtermite: those times should be relatively quiet for us. especially since the feature freeze rush ends this week14:57
openstackgerritMerged openstack-infra/project-config: Added npm-docs draft build for eslint-config-openstack.
irtermitethanks fungi... how did the weather go?14:58
fungiwe could preemptively dial our quota there down to 0 just to make things nicer14:58
*** rbrndt has quit IRC14:58
anteayairtermite: have you seen before? if you look at the bottom you can see the graphs of the job rates14:58
fungiirtermite: the bulk of the storm hasn't reached us yet, and the flood risk is overnight tonight through to tomorrow night14:58
dhellmannfungi : ack, thanks14:58
*** maishsk has quit IRC14:59
irtermitecloudnull: yes... I just CCed you on the email. Fixes: Unexpected switch reloads, VPC and port-channel disruptions, IPv6 HSRP issue.14:59
*** mikelk has quit IRC14:59
*** sputnik13_ has quit IRC14:59
*** kzaitsev_mb has quit IRC14:59
anteayairtermite: this is the configuration file of all the jobs that can be called upon to run:
anteayabut that doesn't give you information on when they are run14:59
anteayafungi: do you have to move furniture today?15:00
fungianteaya: i got up at sunrise to do that. already taken care of15:00
irtermiteoh no... best of luck to you fungi!15:00
AJaegerproject-config cores, could you review a change to add releasenotes for magnum, please?
fungiirtermite: thanks, this doesn't look like it will be a major impact for us since the eye is coming at us over the mainland rather than over the water15:01
anteayafungi: awesome, glad you are all set15:01
irtermitethank you anteaya15:01
anteayairtermite: you're welcome15:01
*** rcernin has quit IRC15:01
*** maishsk has joined #openstack-infra15:02
irtermitewell keep us updated fungi!15:02
fungijeblair: i'm considering restarting gerrit... there are things on the cusp of merging though15:02
*** jcoufal has quit IRC15:02
openstackgerritMonty Taylor proposed openstack-infra/shade: Use list_servers for polling rather than get_server_by_id
anteaya+1 restart gerrit15:03
*** links has quit IRC15:03
mordredShrews, jeblair: if'n you don't mind ^^ I noticed the GetServer poll in the nodepool logs this morning15:03
irtermitehow would you prefer to proceed next week? drop quota to 0 or just roll with it and see? We are doing reboots of odd numbers devices first then even after. So... node access will still be up. Honestly, I don't foresee that being a problem with the current quota at all.15:04
anteayaAJaeger: +215:04
fungiirtermite: oh, then i'd just leave things running15:04
irtermitecloudnull: +2?15:04
cloudnullI have no +215:04
fungiirtermite: generally if our ci loses access to a node it just reschedules the job running there and deletes the old node15:04
irtermiteyou do to this decision cloudnull   ;)15:05
irtermiteyou hold the keys, if you willl15:05
*** Ravikiran_K has quit IRC15:05
irtermitefungi: perfect15:05
cloudnullhow long will the upgrade take ?15:05
fungidhellmann: ttx: dims: we're looking at a quick gerrit restart to relieve some resource pressure... are you at a lull on release approvals for the moment?15:05
irtermiteThe plan is (Odds Wednesday 00:00 – 06:00 and Evens Thursday 00:00 – 06:00)15:05
*** weshay is now known as weshay_mtg15:06
fungiodds bodkins15:06
*** salv-orlando has quit IRC15:06
*** e0ne has joined #openstack-infra15:06
cloudnulldo we really think it'll take 6 hours?15:06
anteayahongbin: we are renaming higgins to zun, do you still want the gerrit groups to be called higgens-core and higgens-release? or do you want to update
irtermitepft, doubtful15:06
dimsfungi : nothing is ready to push a release for. so sure15:06
anteayadims: thank you15:07
cloudnullpersonally, I'd say Bill O’Reilly it and "DO IT LIVE!"15:07
hongbinanteaya: ack, will update that15:07
anteayahongbin: thank you15:07
*** jordanP has joined #openstack-infra15:07
anteayahongbin: also gerrit is about to be restarted, so wait a minute or two15:07
fungii'm waiting for one tempest job to complete so zuul can report a couple of merges into gerrit and then we should be at a safe window for a quick gerrit restart15:07
anteayafungi: ah okay, thanks15:07
dhellmannfungi : a quick restart is ok with me15:08
dhellmannfungi : I'm still reviewing branch results15:08
irtermiteACK cloudnull and fungi moving forward live this wed and thur15:08
irtermiteon it15:08
openstackgerritBrad P. Crochet proposed openstack-infra/project-config: Add zaqar to scenario002 for tripleo ci
*** maishsk has quit IRC15:09
*** kushal has quit IRC15:10
cloudnullwe have other users on this cloud so if the impact is misserable it'll be felt everwhere and there's really no getting around it. we also have the controll plain split out in different cabs so "in theory" as you take devices down and upgrade them the cloud "should" stay up even w/ degrated connectivity.15:10
pleia2fungi: o/ what do we have for prep re: renames?15:10
pleia2(whenever you have a moment, no rush)15:10
*** dprince has quit IRC15:10
cloudnullso yes, irtermite "Do it LIVE!"15:10
anteayapleia2: and good morning15:10
pleia2anteaya: thanks15:10
fungiokay, zuul's processing these merges now15:11
anteayapleia2: currently I'm just reviewing the patches15:11
pleia2anteaya: nods, I'll do the same15:11
anteayapleia2: 3 rename patches and an oopsie fix15:11
* pleia2 nods15:11
pleia2I recall the oppsie x_x15:12
*** matbu|mtg is now known as matbu15:12
anteayayeah :(15:12
*** martinkopec has quit IRC15:13
* fungi learned his lesson. no restarting puppet early even if it looks like the timing will work out. because sometimes... it doesn't :/15:13
*** javeriak_ has quit IRC15:13
anteayawe all have our lessons to learn15:13
anteayasome of them are hard15:14
fungiluckily the correction on that one is to just delete a couple directories while gerrit is offline15:14
anteayaawesome I wondered what the fix was15:14
anteayaare you going to put it in the etherpad at some point so we can see it crossed off?15:15
*** yaume has quit IRC15:15
*** pcaruana has quit IRC15:15
anteayahistory and all15:15
*** apetrich has joined #openstack-infra15:15
*** bnemec is now known as beekneemech15:16
*** caowei has joined #openstack-infra15:16
*** salv-orlando has joined #openstack-infra15:16
*** rkukura has quit IRC15:16
*** sdake has joined #openstack-infra15:16
*** thcipriani|afk is now known as thcipriani15:17
fungii think the gerrit slowness is causing zuul to take even longer to handle the fallout from these merges, so keeps getting new merge events piling up15:18
pabelangerlooks like we are having some issues deleting nodes in ovh15:18
*** matrohon has quit IRC15:18
pabelangerseems limited to bhs115:18
fungipabelanger: yeah, from the graph it seems like that just started15:18
pabelangerfungi: Ya, checking now15:19
odyssey4meproject-config cores - I'd appreciate a second review on asap15:19
*** dansmith is now known as superdan15:19
fungiluckily the correction on that one is to just delete a couple directories while gerrit is offline15:20
fungier, wrong line from my client history15:20
anteayawell if we are having our cascading failures _after_ feature freeze I still count that as the best feature freeze week ever15:20
fungi#status notice The Gerrit service on is restarting quickly to relieve resource pressure and restore normal performance15:20
openstackstatusfungi: sending notice15:20
anteayafungi: ha ha ha15:20
anteayafungi the reruns15:20
anteayafungi's greatest hits15:20
*** esikachev has quit IRC15:21
*** zhurong has quit IRC15:21
-openstackstatus- NOTICE: The Gerrit service on is restarting quickly to relieve resource pressure and restore normal performance15:21
fungiit's on its way back up now15:21
anteayapleia2: I'm picture fungi as The Beaver talking with Wally and eating cookies15:22
pleia2anteaya: hahaha15:22
EmilienMthx for the restart, it looks better now15:22
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002
fungiEmilienM: sure, sorry for the delay... wanted to find a moment when zuul wasn't in the middle of trying to merge a change15:23
openstackstatusfungi: finished sending notice15:23
*** kushal has joined #openstack-infra15:24
fungianteaya: tonyb talked through the requirements rename bits with AJaeger earlier, so should be all set15:25
fungi(just catching back up on conversation from before the gerrit restart)15:25
fungigonna step away from a couple minutes now that nothing's on fire... will be back shortly15:25
*** esikachev has joined #openstack-infra15:26
*** spzala has quit IRC15:26
anteayafungi: awesome thank you15:26
anteayafungi: thanks for the restart15:26
odyssey4methanks anteaya15:27
anteayaodyssey4me: welcome15:27
openstackgerritHongbin Lu proposed openstack-infra/project-config: Zun: rename higgins to zun (2)
anteayahongbin: thank you15:29
*** spzala has joined #openstack-infra15:29
*** spzala has quit IRC15:29
*** sputnik13_ has joined #openstack-infra15:29
zigopabelanger: <--- Yet another instance of httpredir.d.o mirror not working. Do you think I could work on doing a "proper" Debian mirroring thing? Where would that be?15:30
anteayapleia2: can you check and make sure I have the gerrit group info correct on the etherpad, please?15:30
pleia2anteaya: sure thing15:31
anteayapleia2: thank you15:31
pabelangerzigo: aside from unsigned packages in our AFS mirror, what is the issue with using them today?15:31
zigopabelanger: Well, the unsigned thing prevents using it during the sbuild-create.15:31
pabelangerzigo: and no way to work around that?15:31
zigosbuild-createchroot, that is.15:32
zigopabelanger: None that I know at least.15:32
*** esikache1 has joined #openstack-infra15:33
pabelangerzigo: it is just a time issue right now, I need to spend some time on the apt-ftp script you linked before15:33
zigopabelanger: I was asking here if there was a way so I could do it...15:33
zigopabelanger: I'm sure you're super busy! :)15:33
*** mrjk has joined #openstack-infra15:34
*** jcoufal has joined #openstack-infra15:34
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added documentation publish jobs for the JavaScript SDK
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: npm-docs job is now voting
pleia2anteaya: thumbs up15:35
pabelangerzigo: you could write the patch, but we'd need an infra-root to run / watch the script. We'd need to update our mirror_update.pp manifest to use the rsync process, in 2 stages.  I think the script you linked from debian will do that15:35
*** esikachev has quit IRC15:36
*** kaisers_ has quit IRC15:36
zigopabelanger: Where should I write such patch? Any path I should look into in project-config?15:36
zigoOr something new?15:37
mordredthe biggest 'feature' we may need to write15:37
mordredis that we do _not_ want to delete files when rsync thinks they shoudl be deleted15:37
mordredwe want to register those files $somewhere15:37
mordredand delete them on a 2-3 hour delay15:37
mordredso every time we run the rsync, we need to collect the list of things that rsync wants to delete15:37
zigomordred: That's already what the mirror thing does.15:37
mordredas long as it does that, then we're set15:38
pabelangerI think ftpsync does this, I just haven't run it15:38
zigoIn fact, it's the master FTP that does that.15:38
zigoIt first pushes the update.15:38
zigoThen updates the indexes.15:38
zigoThen after 2 or 3 dak run only, it deletes the old stuff.15:38
openstackgerritMerged openstack-infra/project-config: Add non-voting multi-region functional test to OSA os_swift
*** esikache1 has quit IRC15:38
*** esikachev has joined #openstack-infra15:39
zigoIf we use InRelease (I'm not sure Jessie does, but Xenial should be), then it's completely atomic.15:39
zigoThe Release.gpg stuff broke the atomicity.15:39
dhellmannfungi , anteaya : I've reviewed the logs for the branches I created, and they all look good. It should be fine to go ahead with the gerrit maintenance later today.15:39
zigoOnly bad FTP mirrors not using the correct ftp-sync scripts have issues.15:40
anteayadhellmann: thank you!15:40
zigoUnfortunately, many mirrors are like that. :(15:40
anteayapleia2: ^^15:40
anteayaAJaeger: are you around today for the rename?15:40
persiaFor Debian, ftpsync is likely the best.  For Ubuntu, ubumirror may be preferable.15:40
anteayapleia2: and thanks for checking the gerrit group change info15:41
pleia2dhellmann: thanks!15:41
*** martinkopec has joined #openstack-infra15:42
*** sputnik13_ has quit IRC15:42
*** andymaier has joined #openstack-infra15:42
anteayapleia2: so right now I'm waiting for fungi to tell me if he wants to use the documented new workflow using ansible playbooks, or the old manual way15:42
*** tesseract- has quit IRC15:42
anteayapleia2: plus the online reindex will be new as well15:42
pleia2anteaya: yeah, I was going to ask about that next15:42
* pleia2 nods15:42
anteayazaro: how did the online reindex test go?15:42
pleia2I do have to hop out at 19:30 for a lunch meeting I nearly forgot about15:43
anteayazaro: does it look good to use it for today?15:43
pleia2but hopefully we'll know one way or the other re: online/offline, and will be either waiting or doing by then15:43
anteayapleia2: understood, helping with prep is wonderful, thank you15:43
anteayapleia2: ack15:43
anteayapleia2: we might be done by the time you have to go15:43
zaroanteaya: the test went great. gerrit was slow during the index but nothing failed.15:43
pleia2hope so!15:43
pleia2I am going out for Ramen with one of the women who helped with book editing :d15:44
anteayazaro: awesome thank you15:44
pleia2book comes out next week :D15:44
zaroso yeah, i think it's go to go15:44
anteayapleia2: oh that sounds wonderful15:44
*** pelix has joined #openstack-infra15:44
anteayapleia2: yay book \o/15:44
AJaegeranteaya: no, won't be around - I'm not feeling too well, and will leave soon15:44
*** xarses has quit IRC15:44
pleia2pretty exciting, I hope it doesn't suck15:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote os_nova/lxd and os_neutron/ovs jobs to voting
anteayapleia2: I of course want an autographed copy15:44
pleia2anteaya: of course :)15:44
anteayaAJaeger: oh my, take care of yourself15:44
anteayaAJaeger: I hope you start feeling better soon15:45
anteayapleia2: thank you :)15:45
anteayazaro: wonderful, I hope it works15:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack-infra/project-config: Promote os_nova/lxd and os_neutron/ovs jobs to voting
mordredpersia: what's the difference?15:46
clarkbgood morning15:47
*** electrofelix has quit IRC15:47
anteayamorning clarkb15:48
AJaegerthanks, anteaya15:48
anteayaclarkb: if you care, renames are a go and the etherpad is here:
anteayaAJaeger: :)15:48
anteayaAJaeger: hope you have an enjoyable weekend15:48
anteayaclarkb: waiting on fungi to say what workflow he wants to use, he'll be back in a bit15:49
pabelangerzigo: mordred: running ftpsync locally to see how it works.15:50
* AJaeger hopes that the rename works just fine15:50
pleia2thanks AJaeger, feel better15:50
*** martinkopec has quit IRC15:51
AJaegerthanks , pleia215:51
persiamordred: Each is the official mirroring script for the relevant distribution, and so they invoke rsync subtly differently, encode different metadata about the distro in source, etc.15:51
mordredpersia: lovely15:52
persiaBut it's been a few years since I was a mirror operator, so I don't remember the details15:52
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
AJaegeranteaya: Do you want to stack the rename changes on top of each other?15:52
*** eharney has joined #openstack-infra15:52
* AJaeger proposes to soft-freeze project-config so that we don't have any merge conflicts15:52
mordredpabelanger: I'm hoping that the on-disk results of ftp-sync isn't massively different than our current reprepro output dir ... I mena, I'd expect the pool to be largely the same I hope?15:52
AJaegerSo, let's only merge critical changes for the release right now...15:52
anteayaAJaeger: I'm just going to wait to hear back from fungi before I do anything else yet15:53
anteayaAJaeger: I don't know as there are any release critical patches outstanding15:53
AJaegeranteaya: We merged everything release critical that I was aware off.15:53
anteayawe got the green light from dhellmann so I take that to mean the release team is clear and isn't waiting on anything15:53
pabelangermordred: Ya, I'm looking to see if we can only mirror jessie. But expect everything but indexes to change15:54
AJaegerdhellmann, ttx, dims, are there still any changes you like to have merged? Otherwise I propose to soft-freeze project-config15:54
anteayaAJaeger: yeah, so I'm on board with project-config patch freeze15:54
pabelangermordred: err, nothing to change but indexes15:54
anteayaI won't merge anything else15:54
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins
anteayaI'll let fungi inform the others15:54
dimsAJaeger : nothing from me15:55
persiapabelanger: maybe the stamp files will change? (not that this affects clients)15:55
anteayaAJaeger: realease team already said we are good to go15:55
*** senk has joined #openstack-infra15:55
anteayawe don't have to keep asking them15:55
mordredpabelanger: oh. phew15:55
mordredpabelanger: the first thign you said was concerning :)15:55
*** matrohon has joined #openstack-infra15:56
Daviey /win 4715:57
clarkbmy understanding of ftpsync was it does two passes and on the second it removes the packages no logner in the index15:57
clarkbwhich is what we don't want15:57
clarkbbut maybe I was reading debians "your tools must do this" wrongly and thinking that was also what their tools do15:57
*** pilgrimstack has quit IRC15:58
*** matrohon has quit IRC15:58
pabelanger#RSYNC_OPTIONS2="--max-delete=40000 --delay-updates --delete --delete-excluded"15:58
pabelangerthat is what happens on stage 215:58
pabelangerwe have the option to change that15:58
*** matrohon has joined #openstack-infra15:59
*** rkukura has joined #openstack-infra15:59
mordredclarkb: according to zigo - the debian archive already does the delayed delete in the archive itself15:59
pabelangerwe could remove the --delete flag, like we do today.15:59
clarkbmordred: greghaynes there is something about the ubuntu minimal images compared to the regular ubuntu cloud iamges that makes virsh shutdown work against regular images and not against dib ubuntu-minimal images. Maybe some missing acpi package?15:59
mordredclarkb: interesting16:00
zigoclarkb: AJaeger asked me to see with you what can be done for deb-websockify which was broken in the upstream repo (but I repaired it since).16:00
zigoclarkb: So, what can I do here?16:00
mordredpabelanger: yah. I mean, my original thought had be too pass -n to the delete command and just collect the output - then parse it and record the files into a little sqlite db16:00
mordredpabelanger: along with timestamp for having been added to the db16:01
clarkbzigo: you need to fix the repo then get a gerrit admin to push the fixed repo into gerrit16:01
mordredpabelanger: then doing the delete on anything wiht a timetamp older than like a day or somethig16:01
zigoclarkb: Well, it's fixed...16:01
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Merge builder.Builder and builder.Jenkins
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Move CacheStorage to its own module.
zigoclarkb: git fsck doesn't give the error there used to be.16:01
clarkbzigo: ok please push a revert of the change to remove it and note that a gerrit admin will have to force push the repo contents16:02
*** matthewbodkin has quit IRC16:02
AJaegeranteaya: project rename changes look fine - I didn't check pypi16:02
*** tongli has quit IRC16:02
anteayaAJaeger: thank you for reviewing the patches16:02
anteayaI think pleia2 checked pypi and launchpad16:03
anteayapleia2: was that for all of them or just one of the patches?16:03
clarkbfungi: jhesketh have we done any investigating into what caused our log disk usage to skyrocket? seems like we had been well under capacity for a long time16:03
AJaegeranteaya: they don't seem to conflict, so no need to base them on top of each other - on the other hand, stacking them should be trivial ;)16:03
anteayaAJaeger: agreed16:03
clarkbmaybe we aren't compressing some class of file or some project is uploading massive qcow2s on every job run again?16:03
*** rbrndt has joined #openstack-infra16:03
*** sandanar has joined #openstack-infra16:05
pabelangermordred: Right, we have some options.16:07
openstackgerritThomas Goirand proposed openstack-infra/project-config: Revert "Remove deb-websockify from projects.yaml"
zigoclarkb: There you go.16:07
pabelangermordred: Issue I see right now, it looks to be hard to only mirror jessie, like we do today. Working to see exlude magic can help16:07
*** tphummel has joined #openstack-infra16:08
anteayapleia2: feel like posting a reminder to the channels about the gerrit maintenance?16:08
fungidhellmann: thanks for confirming!16:08
anteayaah here's fungi16:08
anteayaso either one16:08
fungianteaya: pleia2: i think i'm comfortable trying the documented playbook, though with a few edits. i'll get patches up shortly to document the revision16:09
anteayafungi: sounds good16:09
fungiclarkb: ^16:09
zigomordred: os-client-config fails to build with the sphinx doc calling git: :(16:09
pabelangerclarkb: So, I've pushed ubuntu-xenial to almost all regions, which includes new version of glean.  It looks like we might have an issue in rackspace however16:09
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added documentation publish jobs for the JavaScript SDK
pabelangerclarkb: still waiting for a ubuntu-xenial image to come online16:10
* fungi is still catching up on scrollback16:10
clarkbpabelanger: fun fun. It is the odd one out when it comes to using the config drive network info16:10
*** roxanaghe has joined #openstack-infra16:10
*** weshay_mtg is now known as weshay16:10
zigomordred: Is this fixed in later release?16:10
fungiAJaeger: project-config soft freeze until maintenance (for non-maintenance-related/non-emergency) changes sounds great, thanks16:11
dstufftfungi: sorry, I wasn't feeling great yesterday. What's up16:11
pabelangerclarkb: Ya, I wonder if we are just running glean too soon now16:11
pabelangerclarkb: I'll see if I can get the console working in rackspace16:12
openstackgerritDean Troyer proposed openstack/os-client-config: Update reno for stable/newton
*** ilyashakhat_mobi has joined #openstack-infra16:12
fungiclarkb: yeah, i haven't dug into what may be consuming more log space over the past few months (besides that brief period with ungodly huge keystone logs)16:12
*** tonytan4ever has joined #openstack-infra16:13
pleia2anteaya: I think we'll do a notification like a half hour before16:13
fungidstufft: oh, yesterday was mostly me discovering that setuptools 25.0.0 broke the ability to successfully pip install -e something and have it take precedence in the import path after a packaged version of the same something is installed16:13
dstufftfungi: oh, is this the raw path thing?16:14
fungidstufft: yep16:14
pleia2anteaya: oh and yeah, checked lp for all (and pypi for the one that needed)16:14
anteayapleia2: oh I noticed nova folks planning their afternoon16:14
*** xarses has joined #openstack-infra16:14
anteayapleia2: and I bet they forgot about the maintenance16:15
persiapabelanger: You cannot exclude suites with EXCLUDE= in ftpsync.  I always just put up with them, because there aren't very many.16:15
anteayawas considering speaking up in channel but thought a channel wide broadcast would be better16:15
mordredzigo: os-client-config uses reno as part of its sphinx doc run - do my knowledge reno does not yet have git information cached in the source tarball ...16:15
fungidstufft: there seem to be a couple of alternative solutions (one being the long-standing pr to have pip -e uninstall existing versions before installing)16:15
pabelangerpersia: yup, that is what I am reading too. We might have to do the same16:15
mordreddhellmann: ^^ is that still correct?16:15
pleia2anteaya: not sure I want to send a really early alert just because one team didn't read my email x_x16:15
pleia2I can send out a reminder email16:15
anteayapleia2: okay that would work16:15
anteayaand I bet they read it and forgot16:16
anteayaand thanks for checking lp and pypi16:16
*** andymaier has quit IRC16:16
mordredzigo: how do you handle this in the other reno-using projects?16:16
pleia2ok, email sent16:16
mordredzigo: (it's also possible there is something I'm doing wrong with reno in os-client-config)16:17
*** Swami has joined #openstack-infra16:17
openstackgerritWayne Warren proposed openstack-infra/jenkins-job-builder: Simplify delete by removing unnecessary loop
fungidstufft: mostly just curious at this point as to whether or have much chance of appearing in a coming pip release to counter that, or whether it's something tox should be patched to work around as a known problem behavior?16:17
mordredzigo: oh! you know what ...16:18
anteayapleia2: thank you16:18
*** shashank_hegde has joined #openstack-infra16:18
mordredzigo: ok. I think I see the issue16:18
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002
zigomordred: I thought you would! :)16:18
dstufftfungi: looks like the first one is merged already?16:18
*** tonytan_brb has joined #openstack-infra16:19
dtroyermordred: was it this?
*** jordanP has quit IRC16:19
dtroyernevermind ^^^, that never merged...16:20
*** matrohon has quit IRC16:20
*** _nadya_ has joined #openstack-infra16:21
zigoI added git as build-depends-indep, but then it may fail with no git log.16:21
dhellmannmordred , zigo : the feature is in reno, but not pbr, so yes we are not caching release notes in the tarballs yet16:21
openstackgerritJohn Dickinson proposed openstack-infra/project-config: Add openstack-releasenotes-jobs to python-swiftclient
fungidstufft: oh, maybe. i'll admit i find gituhb's ui a bit confusing. i should just clone the repo and look with the cli16:21
dhellmannmordred , zigo : that's high on my ocata list16:21
*** asettle has quit IRC16:21
fungidstufft: aha, there it is... "xavfernandez merged commit 3417fac into pypa:master 15 days ago"16:21
openstackgerritMonty Taylor proposed openstack/os-client-config: Don't build releasenotes in normal docs build
mordredzigo, dhellmann: ^^16:22
fungidstufft: so i guess once we have new pip and virtualenv releases we should be safe to go back to usedevelop=true in tox again16:22
*** tonytan4ever has quit IRC16:22
zigomordred: Oh, indeed, I'll add it as a Debian patch, thanks a lot.16:22
mordreddtroyer: ^^ you too - I believe that will fix the problem zigo is having making packages16:22
*** _nadya_ has quit IRC16:22
*** zul has quit IRC16:23
*** dims has quit IRC16:23
dstufftfungi: theortically!16:23
fungidstufft: thanks for following up!16:23
dstufftI'm not really much of a fan of usedevelop=true16:23
dstufftI can probably see about cutting a release though16:24
openstackgerritChris Krelle proposed openstack-infra/glean: Adjust the way we wait for interfaces to become available
*** gyee has joined #openstack-infra16:24
fungidstufft: yea, we have some projects that take a while to install and the devs like local tox envs to have editable installs so they don't have to keep reinstalling... but i agree in our ci it's something we could consider overriding entirely16:24
openstackgerritMerged openstack-infra/shade: Use list_servers for polling rather than get_server_by_id
*** sambetts is now known as sambetts|afk16:25
dtroyermordred: +@… do we need that in stable/newton too?16:25
mordreddtroyer: just pushed the backported patch - yeah, I think we shoudl do16:25
clarkbpabelanger: I can put my local ubuntu-xenial image somewhere if it will help you debug quicker on rax16:25
dtroyerI see it now ;)16:25
clarkbpabelanger: the password is totally insecure though16:25
fungidstufft: though that leads to potential bad developer experiences if they didn't realize that locally they weren't testing their actual code changes, and then got a nasty surprise failure after pushing code up for ci testing16:26
clarkbI am going to have to pop out just around when the gerrit downtime starts to meet up with other foundation staff to talk about using gerrit (don't worry we can use review-dev)16:26
dstufftfungi: yea :/16:26
jeblairclarkb: that is a master stroke of scheduling :)16:26
fungiclarkb: or maybe you can just use review.o.o... ideally the outage will be brief and gerrit will just be sluggish for a few hours afterward until the online reindexing completes16:27
clarkbfungi: oh right online reindexing16:27
clarkblets hope16:27
dstufftI mostly just don't like it because develop installs are materially different than normal installs so it's not out of the realm of possibilities to have something pass in usedevelop=True that would otherwise fail16:27
fungidstufft: agreed. it's messy16:27
anteayarename etherpad for those who want to play along:
fungii've been known to just copy code from my source tree into the virtualenv rather than use editable installs16:28
*** tongli has joined #openstack-infra16:28
openstackgerritBrad P. Crochet proposed openstack-infra/tripleo-ci: Add Zaqar to scenario002
*** esberglu has quit IRC16:31
*** tongli has quit IRC16:32
*** dims has joined #openstack-infra16:32
*** degorenko is now known as _degorenko|afk16:33
dimsAJaeger : how long would the soft freeze on project-config be for?16:34
*** dprince has joined #openstack-infra16:34
*** hashar has quit IRC16:34
anteayadims: until after the gerrit downtime is complete16:35
anteayadims: what do you need?16:35
*** sandanar_ has joined #openstack-infra16:35
dimsanteaya : ah i thought it was related to release.. :)16:35
dimsno worries16:35
anteayanot related at all16:35
bkerorcarrillocruz: have you been able to use gotty ywt?16:35
*** zul has joined #openstack-infra16:36
anteayasolely so that the project config patches for the rename don't need to be rebased16:36
*** sputnik13 has joined #openstack-infra16:36
dimsanteaya : i was asking around and found one thing someone wanted ( but that can wait till next week for sure16:36
*** Apoorva has joined #openstack-infra16:36
*** bin_ has quit IRC16:37
zigomordred: It didn't fix it... :/16:37
*** jpich has quit IRC16:37
jeblairbkero: i find this approach simpler, and it uses only commonly available tools:
zigoI'll disable doc build for the moment.16:37
anteayadims: okay great, yeah freeze will be off by end of the day16:37
*** shashank_hegde has quit IRC16:37
*** hashar has joined #openstack-infra16:38
*** eranrom has quit IRC16:38
*** sandanar has quit IRC16:38
bkerojeblair: im not sure if tcpserver is a commonly installed binary16:38
bkerobut certainly more available than gotty. I'm assuming it's in some distro package.16:39
pabelangerclarkb: jeblair: The good news, is our EOFError rate is way down now16:39
*** hashar is now known as hasharAway16:39
*** hasharAway is now known as hasharAwayy16:39
clarkbpabelanger: awesome16:39
mordredzigo: it worked for me locally16:40
jeblairbkero: i said 'available' not 'installed' on purpose.  :)  it's ucspi-tcp in debuntu.16:40
*** jerryz has joined #openstack-infra16:41
mordredzigo: I made a source tarball with that commit, then untarred that into a location that has no git things anywhere16:41
mordredzigo: then ran the sphinx build16:41
jeblairbkero: it's all just an apt-get away16:41
mordredzigo: can you share the new failure?16:42
jeblairpabelanger: what reduced it?16:42
jeblairpabelanger, clarkb: want to land ?16:43
bkerojeblair: looks like a fine workable solution with distro tools. I just have a strong distaste for the 'script' binary. Too many uni classes being recorded with it and needing perfect execution to look proper in vim.16:43
clarkbjeblair: looking16:43
*** tqtran has joined #openstack-infra16:43
*** eranrom has joined #openstack-infra16:43
*** dtantsur is now known as dtantsur|afk16:43
*** eranrom has quit IRC16:43
clarkbbkero: but script replay is awesome16:43
clarkbbkero: ever need to do a live demo? thats teh way to do it16:43
pabelangerjeblair: we found openssh-server was getting restarted by glean, so early connection would drop16:44
jeblairpabelanger: ah, right. cool.16:44
mordredpabelanger: \o/16:44
jeblairpabelanger: shall we abandon the nodepool change then?16:44
openstackgerritmathieu bultel proposed openstack-infra/tripleo-ci: WIP -- clean full upgrade review
bkeroclarkb: I've found it to be more trouble than other methods. Mayne that's PTSD.16:44
pabelangerHow ever, it appears rackspace is having issues now16:44
jeblairbkero: it sounds like you were using it for something different.16:45
pabelangerjeblair: I think we can for now to avoid churn16:45
zigomordred: Oh, my bad, it looks like it didn't have my change somehow.16:45
* zigo rebuilds16:45
bkerojeblair: recording terminal sessions?16:45
mordredzigo: yah - I was about to say...16:46
*** annegentle has quit IRC16:47
clarkbjeblair: I have approved the timeout change16:47
openstackgerritArmando Migliaccio proposed openstack-infra/elastic-recheck: Expand query for bug 1617447
openstackbug 1617447 in neutron "InterfaceAttachFailed: Failed to attach network adapter device" [High,Fix released] - Assigned to Kevin Benton (kevinbenton)16:47
*** senk__ has joined #openstack-infra16:48
jeblairbkero: yes, recording terminal sessions requiring perfect execution in vim seems slightly different than broadcasting a session where you can always hit "^L" if there is a problem.16:48
*** chem is now known as chem|off16:48
*** senk has quit IRC16:49
jeblair(though i have not encountered such problems.  but then, i don't use vim.)16:49
*** tongli has joined #openstack-infra16:49
bkerojeblair: What I meant to say was that I was using script to record terminal sessions to prove I knew how to unix. The professor in question erroneously told us to use 'script' to record them so he could "watch" them in vim.16:51
*** rajinir has joined #openstack-infra16:51
bkeroThe shortness of my previous explanation was due to phone software keyboard :(16:51
jeblairbkero: that was indeed an error.  and i would say that reflects about as well on the suitability of 'script' for this as if the prof had asked you to use 'gotty' so they could watch it in vim.  :)16:51
bkeroI'd be impressed if he could open a HTTP session in vim16:52
*** hsammic has quit IRC16:52
clarkbbkero: you can pretty easily16:52
fungiyou can totally open streaming network sockets in vim16:53
fungias well as that16:53
clarkb(I do this a lot when dealing with our job logs)16:53
fungialso you can open network sockets in bash as if they were file descriptors too :/16:53
jeblairbkero: the script/tcpserver thing is really nice -- it's simple and very fast -- no buffering for the audience, and it looks like a real terminal session :)16:53
*** caowei has quit IRC16:53
openstackgerritMerged openstack-infra/zuul: bump timeout on ssh commands to 30s
*** eranrom has joined #openstack-infra16:54
*** tongli has quit IRC16:54
*** lucasagomes is now known as lucas-dinner16:54
bkerojeblair: It sounds elegant, although I'm curious how it supports different line length wraps between client and server16:54
jeblairbkero: it works best if everyone sticks to the same window size, fer shure.  :)16:54
fungiand terminal type, especially if there are going to be ansi escapes16:55
jeblair"film should be shown in its intended aspect ratio"16:55
bkerooof, true16:55
anteayajeblair: ++16:55
bkeroyeah, looks like different column lengths completely mangle it16:56
bkeroat least it recovers gracefully16:56
*** dprince has quit IRC16:57
*** jkilpatr has quit IRC16:57
fungiobvious caveats aside, it's a remarkably simple solution16:57
bkerojeblair: I'm trying it with 'sl' and it looks like there are some artifacts generated in the listening terminal -- know what that's about?16:57
*** ifarkas is now known as ifarkas_afk16:58
bkerolooks like I have 79 cols on server and client16:58
*** ZZelle has quit IRC17:00
*** ZZelle has joined #openstack-infra17:00
bkero server on left, client on right17:00
pabelangerclarkb: okay, I have the console working in rax. Did you want to upload your image to rax-ord?17:01
*** amotoki has quit IRC17:01
clarkbpabelanger: its up to you, wasn't sure if that would help or not17:01
pabelangerclarkb: Ya, will same me time rebuilding one so I can login via shell17:01
clarkbok the question though becomes how do I do the upload as not nodepool :/17:01
clarkbmordred: ^ is there an easy way or should I just cry17:02
jeblairbkero: i do not know.17:02
clarkboh I have to convert it to vhd too :/17:03
jeblairbkero: still happens with just a tail of the script file, so it's not the tcpserver or telnet part of things17:03
*** javeriak has joined #openstack-infra17:03
mordredclarkb: there should be a really  easy way17:04
clarkbpabelanger: I will copy it up to nodepool.o.o where it can be converted then figure out the upload from there17:04
*** oanson has quit IRC17:05
mordredclarkb: c=shade.openstack_cloud(cloud='openstackci-osic-cloud1') ; c.create_image(name='foo', filename='path/to/foo.qcow2', wait=True)17:05
bkerojeblair: I'm wondering if it's imperfections in the terminal type17:05
bkeroand how it handles control characters17:05
pabelangerclarkb: does recuse mode work with our images and rackspace17:05
*** yuval has joined #openstack-infra17:06
mordredclarkb: or:  c=shade.openstack_cloud(cloud='rax', region_name='ORD') ; c.create_image(name='foo', filename='path/to/foo.qcow2', wait=True)17:06
mordredclarkb: you can do the same with ansible - but since it's literally two lines in python, i'd probably do it in the repl :)17:06
*** trown is now known as trown|lunch17:06
*** _nadya_ has joined #openstack-infra17:07
mordredyou might also want to do shade.simple_logging(debug=True) before the shade.openstack_cloud call - you'll get _Some_ feedback about activity that way17:07
clarkbmordred: cool thanks17:07
*** _nadya_ has quit IRC17:07
clarkbpabelanger: no idea17:08
*** signed8bit is now known as signed8bit_Zzz17:08
*** roxanaghe has quit IRC17:10
*** jerryz has quit IRC17:10
*** signed8bit_Zzz is now known as signed8bit17:10
*** roxanaghe has joined #openstack-infra17:10
*** sandanar__ has joined #openstack-infra17:10
clarkbpabelanger: the image is going into /opt/test_images where I will convert to raw then back to vhd then do mordreds shade thing17:11
*** bryan_att has left #openstack-infra17:11
clarkbmordred: does create_image not require a container type argument?17:11
*** jerryz has joined #openstack-infra17:11
fungizaro: what's the command to initiate an online reindex in gerrit?17:11
mordredclarkb: nope. it infers from filename17:12
*** ddieterly has joined #openstack-infra17:12
*** shashank_hegde has joined #openstack-infra17:12
*** shashank_hegde has quit IRC17:12
zigomordred: It worked! :)17:13
mordredzigo: woot!17:13
fungiooh, here's another problem i just noticed with the rename playbook... it seems to assume that the account you're running this in on the puppetmaster has ssh credentials for the gerrit api17:13
pabelangerclarkb: ack17:13
fungiwhich will almost certainly not be the case17:13
mordredclarkb: it'll also look at what the cloud claims to want ... if you're doing something like uploading a qcow2 to ovh, which claims to want raw, then you might have to be explicit17:13
* zigo goes out to eat17:13
*** sandanar_ has quit IRC17:14
*** florianf has quit IRC17:15
*** HeOS has quit IRC17:15
mordredclarkb: you know - we _could_ have it use libmagic to figure out for sure what type of image you gave it, instead of having it look at extension17:15
openstackgerritJeremy Stanley proposed openstack-infra/system-config: Update project rename instructions
fungianteaya: pleia2: mordred: ^17:17
anteayafungi: thank you17:17
* pleia2 reviews17:17
fungispeaking of which, i'm going to go ahead and make sure our puppet/ansible cron is good and stopped now17:18
anteayafungi: good idea17:18
*** rossella_s has quit IRC17:18
*** bethwhite_ has quit IRC17:18
*** rossella_s has joined #openstack-infra17:19
fungicurrent run started at 17:15 utc17:19
*** ddieterly has quit IRC17:19
anteayahow long do you expect it to take?17:19
fungii've commented it out in the crontab but will check in on it again in a few minutes17:19
*** e0ne has quit IRC17:19
fungihopefully it'll be well and done in the next 20 minutes17:19
clarkbpabelanger: what region would be best to test in?17:19
fungii did not comment out the cron for infra-cloud hosts since they should be entirely unrelated to this work17:20
* clarkb does ord as I think we have most quota there17:20
* pleia2 nods17:20
pabelangerclarkb: ord should be good17:20
fungialso when i run the playbook, i'm going to take out the gerrit group renames step and perform those directly on my own17:21
anteayafungi: ack17:21
anteayafungi: i put the gerrit group chances in the etherpad and pleia2 reviewed them17:21
fungiat least until puppetmaster's root account has gerrit admin account credentials17:21
fungiwhich won't be som,ething i'll work on today17:22
fungispeaking of which, i mostly expect the api calls to perform group renames would not work without adding in some sort of delay17:22
fungigerrit is not immediately usable as soon as its initscript returns17:23
*** sarob has joined #openstack-infra17:24
*** ddieterly has joined #openstack-infra17:25
*** tongli has joined #openstack-infra17:25
*** sarob has quit IRC17:25
clarkbpabelanger: the upload is running in a screen owned by my user (yet another use of script is to make screen work for other users) you can follow along there. Also the readme in /opt/test_images has the user auth info17:25
fungioh, i guess this is also missing the command to start online reindexing. i need to look that up since zaro seems to not be around17:25
*** mwhahaha has quit IRC17:25
*** sarob has joined #openstack-infra17:25
clarkbmordred: I want this set of steps to be in the official rax docs17:26
clarkbmordred: their docs are basically here use the gui or consider making a snapshot because its complicated otherwise17:26
*** salv-orlando has quit IRC17:26
pabelangerclarkb: need to teaching, do you know if it is udev that runs to setup config-2 for blkid?17:26
openstackgerritEmilien Macchi proposed openstack-infra/devstack-gate: Add some missing projects
pabelangerclarkb: I'm thinking we just need to add another dependency for systemd17:27
clarkbpabelanger: that sounds right17:27
clarkbpabelanger: though in a systemd world its probably some new thing that has replaced udev :P17:28
pabelangergoing to check an image in osic and see17:28
zarofungi: ssh -p 29418 gerrit index start --force changes17:28
*** mwhahaha has joined #openstack-infra17:29
*** ilyashakhat_mobi has quit IRC17:29
*** markvoelker has joined #openstack-infra17:29
fungithanks zaro!17:29
zarowe we following an etherpad for this project name change?17:29
*** tongli has quit IRC17:29
openstackgerritMerged openstack-infra/elastic-recheck: Expand query for bug 1617447
openstackbug 1617447 in neutron "InterfaceAttachFailed: Failed to attach network adapter device" [High,Fix released] - Assigned to Kevin Benton (kevinbenton)17:30
*** mat128|away is now known as mat12817:30
*** ddieterly is now known as ddieterly[away]17:31
anteayafungi: do you want me to recheck those 3 patches or not bother?17:31
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove octavia bandit job
fungianteaya: what was wrong with them?17:32
fungior just want to make sure they're still passing jobs?17:32
anteayajust wanted to make sure they didn't need a rebase17:32
anteayayeah that17:32
anteayapleia2: you want to do the statusbot announcement?17:32
clarkbpabelanger: I have a systemd-udevd service on tumbleweed17:32
pleia2sure, let's see...17:32
AJaegerclarkb: is EmilienM's change 365071 fine? I thought we don't want to add more to PROJECTS...17:33
clarkbAJaeger: correct those should go in the tests that need those projects17:33
pabelangerclarkb: Ya, I am just reading up on it now17:34
*** markvoelker has quit IRC17:34
clarkbAJaeger: my eventual goal is to get the PROJECTS list down to what devstack runs by default. nova neutron glance cinder swift horizon maybe17:34
anteayafungi: all send them through check again17:34
*** markvoelker has joined #openstack-infra17:34
pabelangerclarkb: I think we want After=systemd-udevd.service17:34
clarkbAJaeger: beacuse such a large chunk of our job time is setting up repos we will never use17:34
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: Use time.sleep() when waiting for workers to run
pabelangerbut need to test17:34
clarkbpabelanger: ya that seems about right17:34
clarkbpabelanger: my iamge exists in ORD now17:34
AJaegerclarkb: gave a -1 ;)17:35
AJaegerclarkb: ok, then I remembered correctly17:35
pleia2anteaya, fungi - want to take a quick look at what I put in the etherpad for notice?17:35
clarkbpabelanger: and log in details are in /opt/test_images/README17:35
pabelangerclarkb: we might want to also include haveged too17:36
pabelangerclarkb: sure, will check shortly17:36
clarkbpabelanger: that image should have haveged17:36
anteayapleia2: looks good to me17:36
clarkboh you mean in the unit file ?17:36
EmilienMclarkb: ack17:36
pabelangerclarkb: actually, I don't know if we need the service running17:36
pabelangerI think jeblair said we don't17:36
clarkbpabelanger: I don't think its necessary since the init urandom script turns it off if it is17:36
pleia2#status notice The infrastructure team is taking Gerrit offline for maintenance, beginning shortly after 18:00 UTC for a potentially 4 hour maintenance window.17:36
openstackstatuspleia2: sending notice17:36
pabelangerclarkb: Ya, that sounds right17:37
clarkbpabelanger: we don't it invokes the process directly and retreives the data17:37
*** dtardivel has quit IRC17:37
*** sasi_ has joined #openstack-infra17:37
jeblairclarkb, pabelanger; the urandam init does not require a running haveged17:37
zarofungi:fyi, probably need to online reindex accounts as well as changes.17:37
zarodo accounts first17:37
anteayazaro: what is the command, is it different that the one you gave?17:38
jeblairclarkb, pabelanger: it's fine for the haveged service to start after urandom init runs (indeed, things will be slightly faster if it does :)17:38
-openstackstatus- NOTICE: The infrastructure team is taking Gerrit offline for maintenance, beginning shortly after 18:00 UTC for a potentially 4 hour maintenance window.17:38
*** eranrom has quit IRC17:38
zaroanteaya: it's on the review of that change17:38
fungizaro: thanks17:38
*** _nadya_ has joined #openstack-infra17:38
anteayazaro: ah thank you17:38
sasi_having problem in building dib image on nodepool --
pabelangerjeblair: thanks17:39
*** mriedem is now known as mriedem_afk17:39
*** waht has joined #openstack-infra17:39
clarkbsasi_: do you have a /var/log/nodepool/images? look inthere for the build logs to see why it failed17:39
openstackstatuspleia2: finished sending notice17:39
pleia2openstackstatus: you get a cookie!17:40
clarkbsasi_: but it looks like maybe it doesn't have file permissions to create that dir it is complaining about17:40
zarofungi: step 4.5 - stop gerrit?17:40
*** deadnull_ has quit IRC17:40
*** shashank_hegde has joined #openstack-infra17:40
fungizaro: that playbook already stops and starts gerrit17:40
AJaegerfungi, openstackdocstheme gets more interesting ;( I merged the one change and for the fun of it want to enable constraints on it - but now I get "Could not satisfy constraints for 'openstackdocstheme': installation from path or url cannot be constrained to a version" - see
fungithe ansible script on puppetmaster.o.o is no longer running and is still commented out in the crontab there, so should be all set17:42
AJaegerfungi: I have a working solution, so can ignore this one. No need to dig now into this...17:42
anteayafungi: thank you17:42
AJaegerfungi: more a FYI17:42
*** Na3iL has quit IRC17:42
*** chem|off has quit IRC17:43
sasi_Clarkb: no error on the image.log17:43
fungiAJaeger: i think that's usually due to lack of edit-constraints setting the entry in your requirements list appropriately, but in this case it could be complicated by your transitive dependency loop17:43
clarkbsasi_: ya its likely failing early then and its a permissions issue. make sure it can write to that directoy17:43
AJaegerfungi, yeah - I have edit-constraints. I think the transitive loop is the problem...17:43
fungiAJaeger: er, actually edit-constraints modifies the constraints file, which should make that work. so not sure what's happening there17:43
AJaegeryeah, after changing the usedevelop setting it suddenly fails...17:44
*** prometheanfire has quit IRC17:45
*** dprince has joined #openstack-infra17:45
sasi_clarkb: owner of the /opt/nodepool_dib is nodepool, and it can write into it17:45
sasi_drwxr-xr-x  2 nodepool nodepool 4096 Sep  2 10:44 nodepool_dib/17:45
*** ccarmack has quit IRC17:46
openstackgerritPaul Belanger proposed openstack-infra/glean: Add After=systemd-udev-settle.service for systemd
pabelangerclarkb: ^ WIP until I can test it17:48
*** adrian_otto has joined #openstack-infra17:49
*** tongli has joined #openstack-infra17:49
*** _nadya_ has quit IRC17:49
fungianteaya: pleia2: zaro: okay, i think the steps in the etherpad coupled with the proposed change will cover it17:49
anteayafungi: awesome, thank you17:49
fungii'm working on the ansible playbook parameter file now17:50
*** eranrom has joined #openstack-infra17:50
zarofungi: lgtm17:50
*** yamahata has joined #openstack-infra17:51
anteayafungi: patches finished recheck, no rebases required17:51
anteayafungi: I haven't stacked them17:52
* AJaeger signs off now and hopes that all playbooks play well ;)17:53
anteayaAJaeger: thank you17:53
*** tongli has quit IRC17:53
*** waht has quit IRC17:54
stevebakerhey, it looks like merging to heat-templates is wedged
anteayastevebaker: we are about to take gerrit offline to rename some projects17:55
anteayastevebaker: perhaps post maintenance we can come back to this?17:56
anteayastevebaker: thanks :)17:56
stevebakerthank you17:56
jesusaurclarkb: do your log_processor workers ever die with MemoryError? I'm seeing that happen a few times a month17:56
clarkbjesusaur: no, I had to give logstash 2gb of heap though17:56
*** eharney_ has joined #openstack-infra17:57
*** eharney has quit IRC17:57
*** ijw has joined #openstack-infra17:57
AJaegeranteaya: those two reviews by stevebaker really look strange and need investigation - but agreed, gerrit offline is important.17:57
AJaegerreally off now ;)17:57
openstackgerritEmilien Macchi proposed openstack-infra/project-config: tripleo: add missing PROJECTS in the list
*** eharney_ is now known as eharney17:58
EmilienMpabelanger, AJaeger ^17:58
anteayaI just figured not sense getting into and then gerrit going offline17:58
fungianteaya: pleia2: zaro: the playbook parameter file is drafted in the etherpad now17:58
anteayafungi: thank you17:58
*** piet has quit IRC17:58
*** AnarchyAo has joined #openstack-infra17:58
fungii'm getting the parameter file and an edited copy of the playbook into place in my homedir on puppetmaster.o.o now17:58
anteayafungi: looks right to me17:58
pabelangerclarkb: okay, have server online and network fixed17:58
pabelangerclarkb: testing patch now17:59
*** ijw has quit IRC17:59
pabelangerafter I installed vim17:59
clarkbpabelanger: ya its minimal :)17:59
*** piet has joined #openstack-infra17:59
jesusaurclarkb: hm, ok. I'm also limiting logstash to 2gb, but i also still have a local elasticsearch with a 1gb heap17:59
*** sandanar has joined #openstack-infra17:59
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci: DNM: hack projects until
zarofungi: lgtm18:00
* jesusaur will try to play with heap sizes or bump to 8gb instances18:00
pleia2woo, lmk when it's alert time :)18:00
stevebakerAJaeger, anteaya: it could be that heat-templates-buildimages is failing - that job can be deleted18:01
*** ijw has joined #openstack-infra18:01
fungianteaya: pleia2: zaro: i'm all set with the parameter file and updated playbook now18:01
anteayastevebaker: ah perhaps post maintenance we can begin there18:01
anteayafungi: awesome, can pleia2 alert folks now?18:01
fungiyep, alert away18:01
pleia2#status alert Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC.18:02
openstackstatuspleia2: sending alert18:02
fungiwho else in infra-root wants to grab steps in the plan at ?18:02
*** sandanar__ has quit IRC18:03
pabelangerfungi: As soon as I fix ubuntu-xenial in rackspace, I'll add my name to something18:03
pleia2I should probably not be the reindexer, since I need to step out in 90 minutes18:03
fungizaro: is it okay for accounts reindex and changes reindex to be running concurrently?18:03
*** ijw has quit IRC18:03
fungipleia2: i can handle that part. i'll be around18:03
zarodon't know. i have not tested that scenario so i wouldn't try it.18:04
zaroaccounts should be pretty fast18:04
pleia2oh, anyone handling zuul yet?18:04
-openstackstatus- NOTICE: Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC.18:04
*** ChanServ changes topic to "Gerrit is now going offline for maintenance, reserving a maintenance window through 22:00 UTC."18:04
fungizaro: fair enough. i'll just watch the processes list until accounts finishes18:04
pleia2I need to look up commands for that18:04
*** annegentle has joined #openstack-infra18:04
fungipleia2: in
fungiwhich is why i linked it18:05
fungiyou can probably pull up the docs-draft version now if you want18:05
zarofungi: watch gerrit show-queue18:05
fungizaro: oh, right, that one. thanks! ;)18:05
pleia2I can do zuul whenever we're ready18:05
anteayaI don't see a stop gerrit step, is there a stop gerrit step?18:05
zaroit's in the playbook18:06
anteayazaro: ah sorry18:06
anteayaI do believe we are ready to stop zuul, are we not?18:06
pabelangerclarkb: here is the failure:
pabelangerneed to figure out why18:06
fungiyep... pleia2: whenever you're ready18:06
*** trown|lunch is now known as trown18:07
pleia2on it18:07
jeblairi'm around; ping me if you need backup18:07
fungithanks jeblair!18:07
openstackstatuspleia2: finished sending alert18:07
* pleia2 waits for zuul to stop18:07
clarkbpabelanger: I think that meams there is no network device with that macaddr in sysfs18:08
pleia2zuul step all done18:08
anteayapleia2: wonderful18:08
fungipleia2: thanks. i'll run the playbook now18:09
* pleia2 fingers crossed18:09
pabelangerclarkb: Hmm, maybe this is a bug with the new release over systemd issue18:09
pabelangerbecause glean is running18:09
pabelangerjust failing18:09
fungiit's running now, but i'm going to carefully inspect its output and then go check behind it as per the plan18:10
* pleia2 nods18:10
anteayafungi: ack18:10
fungioof, a failure18:10
fungioh geez, i see a typo in the playbook18:11
anteayaoh no :(18:11
anteayawhat do you want to do?18:11
fungihowever, this thing probably isn't idempotent so i doubt i can just rerun it after i fix it18:11
pleia2fungi: is 6.1 Gerrit or GitHub?18:12
fungipleia2: gerrit18:12
jeblairfungi: can you paste the playbook output?18:13
fungii'm going to correct the typo and try to surgically remove the steps which have already run18:13
fungijeblair: yep, doing that now18:13
mordredfungi: there is an option with running ansible playbooks to re-run failed steps - as an fyi18:14
fungii think the nested dirs need trailing slashes18:14
jeblairheh, those plays need names :)18:15
mordredfungi: you can also use "--start-at" - but things need names :)18:15
mordredansible-playbook foo.yml --start-at-task=START_AT18:15
*** ijw has joined #openstack-infra18:16
fungihrm, actually no i don't think it's a trauiling slash problem18:17
mordred(I'd be helping more explicitly, but I've got a call in 10 minutes so I don't want to have to bail in the middle - so comments from the peanut gallery is best I've got today - sorry)18:18
jeblairfungi: it's missing a trailing '.git'18:18
fungijeblair: yep, i just identified the same :/18:18
fungitrying to tab-complete those dirnames18:18
*** kushal has quit IRC18:18
*** ijw has quit IRC18:19
*** notnownikki has quit IRC18:19
pabelangerclarkb: mordred: Okay, I see the issue. There is a bug in glean that we need to fix18:20
*** ddieterly[away] is now known as ddieterly18:20
fungithough there were also "mv: cannot stat ‘/opt/lib/gitopenstack/higgins" so the missing trailing / was also a problem18:20
*** spzala has joined #openstack-infra18:20
fungii'm correcting both, and also for the git0*.o.o step18:20
mordredpabelanger: neat18:20
fungiand rerunning with the steps prior to the failed mv commands removed18:21
anteayafungi: ack18:21
jeblairfungi: agreed18:21
jeblairmy updated copy of the playbook is this:
fungiit's running again18:21
pabelangermordred: clarkb: so using and trying to configure eth0, there is a KeyError because glean assumes the first ethernet_mac_address in links is eth0, when in fact, it is eth1 in this case18:22
fungisucceeded everywhere except the storyboard step this time18:22
fungianother paste on the way18:22
fungibut this is probably "these projects aren't in storyboard" nonissues18:22
clarkbpabelanger: neat18:22
clarkbpabelanger: so it needs to iterate to find the right one18:22
anteayafungi: ah, so yay mostly success18:23
pabelangerclarkb: Ya, I just need to see what is the right way to do that18:23
fungiCan't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock18:24
mordredpabelanger: oh stellar!18:24
*** pvaneck has joined #openstack-infra18:24
pleia2is it not finding the mysql creds?18:24
fungithat's what i'm checking18:25
mordredor hostname is maybe not set?18:25
*** spzala has quit IRC18:25
pleia2hostname should be with the rest of the creds18:25
*** e0ne has joined #openstack-infra18:25
*** gyee has quit IRC18:25
jeblairnone of these are in storyboard, though, right?  so it doesn't matter for this run?18:25
anteayajeblair: correct18:26
*** yuval has quit IRC18:26
fungii bet we didn't puppet /root/.my.cnf and then neglected to copy it when we replaced the sb server18:26
mordredroot@storyboard:~# mysql storyboard18:26
mordredERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)18:26
*** spzala has joined #openstack-infra18:26
jeblairmordred: reproduced!18:26
mordredI can confirm both the error, and the lack of .my.cnf18:26
fungiyeah, there's no /root/.my.cnf file18:26
fungianyway, we can take that as something to fix after. i'll remove up through the storyboard.o.o task and start the playbook again18:27
fungifatal: []: UNREACHABLE! => {"changed": false, "msg": "Failed18:28
fungi to connect to the host via ssh.", "unreachable": true}18:28
pleia2hm, wrong port?18:28
*** salv-orlando has joined #openstack-infra18:28
fungiremote_user: gerrit218:29
fungia downside of this playbook is that it assumes it can ssh as multiple accounts and into the gerrit api18:29
fungiit only actually has credentials to ssh as root18:29
mordredfungi: yah - we should change away from remote_user and to become_user - which uses sudo18:29
*** jkilpatr has joined #openstack-infra18:30
mordredturns out we're not the only ones to have encountered that problem18:30
fungimordred: so make remote_user: root and add become_user: gerrit2?18:30
mordredyah - or leave remote_user out18:30
*** spzala has quit IRC18:30
fungioh, and it has sudo: yes on that task for some strange reason18:31
mordredfungi: we want "become: true\nbecome_user: gerrit2"18:31
*** ccarmack has joined #openstack-infra18:31
*** pelix has quit IRC18:32
fungilooks like become: true must be assumed if you supply become_user?18:32
*** ccarmack has left #openstack-infra18:32
fungibecause it worked for me without become: true18:32
mordredmaybe that's because sudo: true was there there18:32
fungii removed the sudo line completely18:32
notmorganoh hi.18:33
pleia2o/ notmorgan18:33
fungiso that's the last of the playbook18:33
*** ogelbukh has quit IRC18:33
* notmorgan hails the wonderful infra folks from the california south.18:33
*** shashank_hegde has quit IRC18:33
notmorganpleia2: sorry i didn't bug ya when i was in the bay area, but was in a rush to get down here (plans changed). Will have more time on the trip back north.18:34
anteayafungi: yay playbook's done18:34
pleia2notmorgan: was sorry to miss you, but yes, next time :)18:34
fungiyep, i'll get to checking behind it now to make sure it did... stuff18:34
fungipreferably the stuff we wanted done18:34
*** ogelbukh has joined #openstack-infra18:34
*** jkilpatr has quit IRC18:35
fungiit does seem to have started gerrit there on the last run18:35
fungipeople can probably begin poking around to make sure it's running correctly18:35
fungii'm checking filesystem and database now18:35
anteayayes gerrit is back for me18:35
* notmorgan logs in at the right time to see gerrit return.18:35
notmorgantalk about timing18:35
*** shashank_hegde has joined #openstack-infra18:35
fungii'll note that smaug doesn't seem to have renamed openstack/smaug-dashboard18:35
fungiwas it supposed to be in the list?18:36
jeblairpython-smaugclient exists as well18:36
openstackgerritMorgan Fainberg proposed openstack-infra/shade: Prevent Negative Caching
*** salv-orlando has quit IRC18:36
fungialso openstack/python-smaugclient18:36
anteayafungi: it was in the patch:
*** roxanaghe has quit IRC18:37
anteayasorry I didn't realize it should have been in the playbook config file18:37
fungianteaya: ahh, just not in our list in the etherpad i guess18:37
anteayamy bad reviewing18:37
pleia2ugh, sorry18:37
jeblairGitCommandError: 'git fetch +refs/changes/21/363321/2:refs/changes/21/363321/2' returned with exit code 12818:37
jeblairstderr: 'fatal: Couldn't find remote ref refs/changes/21/363321/218:37
fungii was taking your list at the top there as a full list of project names we were going to rename, sorry18:37
anteayaalso I didn' tknow the ansible config file was not going to use the patch information18:37
*** tongli has joined #openstack-infra18:38
anteayafungi: yeah, sorry those were just the commit message titles of the patches18:38
anteayaor at least what was put into the wiki18:38
fungioops. i shouldn't have assumed18:38
anteayawhich I copy pasted from the wiki page and was input by the patch owners18:38
fungiokay, i'll see if i can get a working playbook assembled to do those other two18:39
anteayayeah, sorry i missed that this was info that should have been in the ansible file18:39
anteayafungi: thanks, sorry18:39
jeblairoh, maybe that just hasn't propogated yet because of the replication backlog on startup18:39
jeblairthat ref is in gerrit, so that looks good18:40
*** prometheanfire has joined #openstack-infra18:40
*** hasharAwayy is now known as hashar18:40
*** tongli has quit IRC18:42
*** e0ne has quit IRC18:42
*** craige has quit IRC18:42
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add nodeset as a top-level configuration item
jeblairpleia2: zuul appears to be running18:43
fungizuul's i don't see it running in ps18:44
pleia2I don't either18:44
jeblairhuh. i thought ! was not a command.18:44 shows activity18:44
anteayasomething is doing something18:44
jeblairyeah, the status page is extant18:45
pleia2was I search for the wrong thing in ps?18:45
jeblairis that an apache cache or something?18:45
*** e0ne has joined #openstack-infra18:45
pleia2I think it must be18:45
jeblairah yep18:45
fungiokay, panic complete18:45
jeblair returned data, until i shift-reloaded it18:46
jeblairi have not seen it do that before18:46
fungii'm about to rerun the playbook with the culmination of our planned edits for the two remaining repos18:46
anteayafungi: lets see what happens this time18:46
pleia2sounds good18:46
jeblairfungi: you get to test the updates! yay!18:46
fungifor the record, this is the updated playbook
jeblairfungi: wait a sec18:47
fungii am holding18:47
jeblairfungi: how did you fix the trailing / problem?18:47
jeblairoh i think i see it18:47
fungiinjecting a / between the parameters in the format string18:47
fungiwhere there was none previously18:48
fungidone for both review and git0*18:48
jeblairfungi: playbook lgtm.18:48
fungiit's running now18:49
fungino failures!18:49
fungithe output was
*** jkilpatr has joined #openstack-infra18:52
fungifilesystem changes on review.o.o lgtm now. checking db next18:52
anteayawhen is the best time to fix the oopsie from last rename window?18:55
fungii already did18:55
anteayaawesome, thank you18:55
*** dprince has quit IRC18:55
*** kushal has joined #openstack-infra18:55
fungidatabase looks good...
*** e0ne has quit IRC18:56
*** Thelo_ has joined #openstack-infra18:56
*** dprince has joined #openstack-infra18:56
fungichecking files on git servers now18:56
pleia2want to divide and conquer git servers?18:56
*** kushal has quit IRC18:56
fungii was just going to spot check a couple18:57
fungibut feel free to check some too18:57
fungii trust that if ansible says it ran something on each server, it did. i'm more interested that the thing it ran on at least one server was the thing we wanted18:57
*** e0ne has joined #openstack-infra18:57
fungilooks right to me!18:58
pleia2spot check on 6,7,9 lgtm18:58
fungii checked 2 and 518:59
pleia2time to bring zuul back?18:59
fungii think so, yes18:59
*** sandanar has quit IRC18:59
pleia2ok, doing so18:59
pleia2reenqueuing gate19:00
pleia2now check19:00
fungionce you rename gerrit groups, i'll start the account reindex19:00
fungii worry that doing those out of order might not work correctly19:00
pleia2ok, all should be reenqueued19:00
pleia2renaming now19:00
fungithough would likely be fine19:01
pleia2renames done19:01
fungisince both are being handled within gerrit anyway19:01
fungiokay, getting reindexing going19:01
*** sdake has quit IRC19:02
fungium, gerrit index start --force accounts seems to also start indexing changes19:02
anteayazaro: ^^19:03
zarowhat's the gerrrit queue say?19:03
fungiat least gerrit show-queue returns a ton of entries for "Index all changes of project openstack/..."19:03
openstackgerritPaul Belanger proposed openstack-infra/glean: Make sure interface_name is validate for network files
pabelangermordred: clarkb: ^ I think that is the correct approach19:03
*** yuanying has quit IRC19:03
zarohmm, well i didn't notice that when i ran account reindex but probably only because i only changed accounts and not any changes that go along with it.19:04
fungipresumably that's something we can try to recreate on review-dev19:04
fungianyway, 4379 tasks still in the queue when i checked again just now19:05
pleia2where did it start?19:05
fungi0b201ba1              19:02:09.162      Index all changes of project openstack-infra/project-config19:06
fungii think19:06
zaroit should list all repos19:06
fungiyeah, it does19:06
fungii mean, i didn't count them all, but the rowcount makes it pretty likely this is all repos19:07
*** Jeffrey4l has quit IRC19:07
zarois one of the name change ones in the queue?19:07
anteayado we have any idea how long reindexing is expected to take?19:07
fungizaro: i'll find out19:07
zaroi think same amount of time as offline index19:07
pleia2should we ok and do another alert about gerrit being slow?19:08
anteaya4-5 hours19:08
fungi9a2e9b97 waiting .... 19:02:09.170      Index all changes of project openstack/karbor19:08
fungizaro: so, yes19:08
anteayaare there any actions gerrit can't perform successfully until reindex is complete?19:08
zarook, once it's off the queue you can do a gerrit search for changes and it should appear.19:08
fungiand no matches for "smaug" in the output anywhere19:08
*** sasi_ has quit IRC19:09
*** tongli has joined #openstack-infra19:09
*** tongli has quit IRC19:09
fungiso anyway, i guess we'll count items 6.2 and 6.3 done19:09
zaroanteaya: no, should work as normal except probably slow19:09
anteayazaro: okay thank you19:09
*** tongli has joined #openstack-infra19:09
*** tongli has quit IRC19:09
zaroyou just can't work on the renamed projects until it's done indexing them19:09
anteayapleia2: I'm fine with okay and gerrit is slow for about 5 hours19:09
anteayazaro: that's what I was wondering19:09
pleia2do I need to OK before another alert?19:09
fungithere are still a few steps (>=7) with no assignees. any latecomers intending to pick any of those up or should i grab some?19:09
*** tongli has joined #openstack-infra19:09
*** yuval has joined #openstack-infra19:10
anteayapleia2: I think so19:10
fungipleia2: no need to okay before doing a second alert but i vote we just okay with that message instead19:10
openstackgerritJames E. Blair proposed openstack-infra/zuul: Use NodeSet objects in jobs and NodeRequests
*** ddieterly is now known as ddieterly[away]19:10
pleia2fungi: like: #status ok Gerrit is completing an online re-index, you may encounter slowness until it is complete19:10
pleia2#status ok Gerrit is completing an online re-index, you may encounter slowness until it is complete19:10
openstackstatuspleia2: sending ok19:10
fungithat will set everyone's channel topics back to normal, but still echo that message to the channels19:11
* pleia2 nods19:11
fungii think that's a good compromise for mild degradation but systems online and functioning19:11
anteayahongbin: best not to work on the renamed projects until online reindexing is complete, guessing at about 5 hours19:11
hongbinanteaya: ack19:11
anteayayuval: same for karbor ^^19:11
anteayahongbin: thank you19:11
pleia2if we're good to do the github renames, I can pop over there and learn how to do that19:11
jeblairzaro: what happens if you do?19:11
yuvalanteaya: :)19:12
anteayayuval: thank you19:12
yuvalanteaya: thank you for the hard work!19:12
mordredpleia2: tl;dr "clickety clickety click. click click. clicketty click."19:12
pleia2mordred: figured :)19:13
jeblairanteaya, zaro: if i'm understanding zaro correctly -- nothing *bad* will happen if people do 'work' on changes in the project, it's just that searches won't work correctly19:13
anteayayuval: I'm a bystander, fungi and pleia2 did the heavy lifting with cameos by jeblair and mordred :)19:13
*** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs | bugs | source | channel logs"19:13
-openstackstatus- NOTICE: Gerrit is completing an online re-index, you may encounter slowness until it is complete19:13
anteayajeblair: ah okay, thanks for clarifying19:13
zarojeblair: i'm guessing that it will work, just can't search19:13
anteayayeah searches not working is not something I would have worried about19:13
jeblair(so, i think it should be okay to, say, upload changes, perform reviews, etc -- just don't expect search queries, dashboards, etc to work as expected while running.  zaro should confirm though.  :)19:13
anteayazaro: ah okay thanks19:14
anteayahongbin: yuval ^^19:14
yuvalanteaya: ofcourse, I meant it in the plural context19:14
fungipleia2: yeah, github renaming should be fine now19:14
anteayayuval: thank you :)19:14
pleia2on it19:14
fungijeblair: zaro: i wonder if zuul will get confused query responses until reindexing is complete19:14
mordredfungi: that'll be fun to learn19:14
fungifor those repos, specifically19:15
zarofungi: queries for thos projects won't work so i'm assume zuul would get confused.19:15
fungii mean, chances are it would just fail to find a change and so wouldn't enqueue it19:15
anteayahongbin: yuval if you do see anything funny in gerrit with your repos, please do share, we are curious19:15
yuvalanteaya: anything specific I should be looking for?19:16
jeblairfungi, mordred: yep.  i expect it to not work but not break.19:16
fungiand might also fail to enqueue changes with a cross-repo depends-on to changes in one of the renamed repos19:16
zarofungi: curious, what's the cpu util?19:16
fungizaro: great q! checking now19:16
fungizaro: though you can also look in cacti.o.o19:16
anteayayuval: behaviour that doesn't respond as normal, patch urls, comments on patches, searches19:16
*** shashank_hegde has quit IRC19:16
openstackstatuspleia2: finished sending ok19:16
jeblairnot as high as a gc loop, apparently.19:17
fungidefinitely looks like it's doing stuff, but not slammed19:17
pleia2github renames done, probably :)19:17
anteayapleia2: yay19:17
pleia2(got the two other smaugs too)19:17
hongbinanteaya: search didn't work, patches are gone under the "Outgoing reviews" session. That is what I observed19:17
*** ddieterly[away] is now known as ddieterly19:17
*** Nakato has quit IRC19:17
anteayahongbin: thanks for sharing19:18
* pleia2 patpat former dragons19:18
*** spzala has joined #openstack-infra19:18
zarothat's surprising.  i saw very different behavior on review-dev19:18
zaroi guess tuning on review probably helped this19:18
anteayazaro: what did you see?19:18
*** Nakato has joined #openstack-infra19:19
zaroCPU utilization like at over 100%19:19
pleia2fungi: anyone on #7?19:19
fungipleia2: i haven't picked up any new tasks yet19:19
fungii was taking a minor break19:19
jeblairfungi: are there more re-index commands to run?19:20
fungii'm happy to approve those changes, but might be better if someone who actually read them did that19:20
fungijeblair: apparently not19:20
anteayawhen do we merge the project-config patches?19:20
jeblairfungi: oh, now i see the rest of the ones in the queue :)19:20
fungi|grep -c 'Index all changes of project' is reporting 395 to go19:21
zarorenamed projects off queue yet?19:21
jeblairzaro: did that 'sort projects by number of changes for reindexing' patch land?19:22
funginote the ones it says are currently active indexing: cinder, horizon, keystone, heat, nova, openstack-manuals, neutron... it's going to be a while19:22
fungijeblair: i don't think we had that change cherry-picked19:22
anteayawhen do the project-config patches merge?19:22
zarojeblair: it landed upstream but cannot be easily backported19:22
jeblairfungi, zaro: well, interestingly, nova ended up near the beginning for some reason19:22
jeblairso that's nice :)19:22
fungianteaya: when someone approves them. any time now is fine19:22
*** spzala has quit IRC19:23
anteayaokay, shall I go ahead?19:23
fungianteaya: be my guest!19:23
jeblairmaybe something about how '--accounts' ended up triggering them19:23
zarohmm, that's odd maybe i'm not remember correctly. lemme check again19:23
dhellmannfungi : is it safe to start processing releases again? we have one bug fix in a lib19:23
fungidhellmann: it should be safe as long as it's not for one of the projects we just renamed19:24
dhellmannos-vif, so I don't think so19:24
funginope, that one's fair game19:24
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Upgrade Laravel Version and ORM Framework
dhellmanngreat, thanks19:24
pleia2thanks anteaya19:24
anteayafungi: what should happen with the requirements change at this point?
fungianteaya: i'll see how much of it is safe19:25
anteayapleia2: I just noticed merge the patches wasn't on the list of items to do19:25
anteayafungi: thanks19:25
*** ilyashakhat_mobi has joined #openstack-infra19:25
zarojeblair: nope, it wasn't backported.  probably indexes them in random order?19:25
*** tkelsey has joined #openstack-infra19:25
pleia2anteaya: I assumed that was step 719:25
*** sarob has quit IRC19:25
fungianteaya: i believe 353371 can't be merged until there is a release of that lib19:26
anteayapleia2: oh19:26
pleia2but I could be mistaken :)19:26
anteayait says puppet configuration19:26
rcarrillocruzbkero: i've used it only locally, flawless...19:26
fungianteaya: note tonyb's inline comments19:26
rcarrillocruzi need to test it from internet, doing nat on my router19:26
rcarrillocruzone thing i see that doesn't work is splitting screens with gnu screen19:27
anteayafungi: okay thank you, I read that but am grateful you also looked19:27
rcarrillocruzthe splits are not seen on the browser19:27
rcarrillocruzi have to try with tmux (i'm a gnu screen user really)19:27
fungianteaya: basically that change only adds the new name, but won't work until there's a release of it on pypi19:27
anteayafungi: ah thank you, now I understand19:27
anteayashould step 7 have said project config patches, rather than puppet configuration?19:27
fungianteaya: then it will pass testing and be able to merge, then requirements updates propagate out with the new name, then a change can be submitted to remove the old name from those files19:27
anteayaah, okay thank you, sorry I'm slow19:28
fungianteaya: yeah, i think it's poorly named in our instructions (from back when this was all in openstack-infra/config probably). feel free to leave an inline comment on my process update19:28
fungionce i have a chance to digest this maintenance i'll have a bunch of other updates to make in that change anyway19:29
rcarrillocruzhow's gerrit rename going? been off all my afternoon/evening19:29
fungircarrillocruz: you missed a bunch of ansible playbook fun19:29
*** spzala has joined #openstack-infra19:29
rcarrillocruzwhich playbook19:29
*** tkelsey has quit IRC19:30
fungithe project renaming playbook19:30
rcarrillocruzyou ran the one that yolanda had in review or something19:30
anteayafungi: done19:30
fungithis was our first attempt to run it in production, yes19:30
rcarrillocruzwas it good?19:30
fungiit needs lots of edits, but i think it will be more useful in the future19:30
zaroluckily, the online index part wasn't as exciting19:30
pleia2we didn't destroy openstack19:30
pleia2so, success19:30
fungipleia2: there's always next week!19:30
*** vhosakot has quit IRC19:31
rcarrillocruzyolanda is awesome, wasn't  expecting less19:31
*** vhosakot has joined #openstack-infra19:31
anteayayolanda is awesome ++19:31
pleia2aaand I am not lunching because my lunch date is also in ops and she's stuck with an emergency19:31
pleia2(we may never actually meet)19:31
fungircarrillocruz: maybe star 365067 and pitch in after i make some updates in a new patchset based on our findings from today's maintenance19:32
rcarrillocruzpabelanger: mind reviewing and ?19:32
anteayapleia2: ha ha ha19:32
rcarrillocruzalso clarkb , we chatted about it ^19:32
rcarrillocruzfungi: sure, let me look19:32
anteayapleia2: glad to have your continued presence19:32
*** gouthamr has quit IRC19:33
*** dkehn_ has quit IRC19:34
anteayastevebaker: still want to chat?19:34
*** spzala has quit IRC19:34
mordred"pleia2 | we didn't destroy openstack" <-- that's really all I ever hope for on any given day19:35
*** dkehn has quit IRC19:35
anteayain other news I just learned that working groups have presidents19:35
*** ddieterly is now known as ddieterly[away]19:35
anteayaI did not know this19:35
mordredanteaya: do they have parliments or congresses?19:35
anteayamy eyes have been opened19:35
*** spzala has joined #openstack-infra19:36
anteayaI don't know but now I am curious19:36
*** thorst has quit IRC19:36
fungisokath, his eyes uncovered!19:36
*** dkehn has joined #openstack-infra19:36
* anteaya looks up the sokath reference19:37
fungisorry, terrible st:tng episode19:37
*** thorst has joined #openstack-infra19:37
*** markvoelker has quit IRC19:37
fungipopped into my head when you said "my eyes have been opened"19:37
mordredfungi: s/terrible/amazing/19:37
jeblairfungi when the disk filled19:37
mordredjeblair, his zuul full19:38
*** yuval has quit IRC19:38
fungilast night it was jhesketh when the disk filled19:38
jeblairfungi, and jhesketh, at tanagra19:38
openstackgerritMerged openstack-infra/project-config: Rename os-failures to os-faults
*** thorst_ has joined #openstack-infra19:39
pleia2jeblair: haha19:39
*** vhosakot has quit IRC19:39
*** roxanaghe has joined #openstack-infra19:39
*** ddieterly[away] is now known as ddieterly19:39
openstackgerritMerged openstack-infra/project-config: Zun: rename higgins to zun (2)
rcarrillocruzone dreams19:40
openstackgerritMerged openstack-infra/project-config: Rename Smaug to Karbor
rcarrillocruzwhen we get the finglonger thing19:40
rcarrillocruzwe can schedule this kind of things19:40
rcarrillocruzwe sit19:40
rcarrillocruzand get the results feeded into firehose19:40
mordreda bug in that theoretical language seems to be that it would be rather difficult to tell the story of darmok and jalad at tenagra in the first place if the langauge only works by literary references to other events19:40
*** spzala has quit IRC19:40
rcarrillocruzmtreinish: ^19:40
rcarrillocruzone dreams!19:40
fungianteaya: was that all of them merged just now?19:40
anteayafungi: I'm reading the webpage on it19:40
jeblairi'm going to stand on the 'amazing' side of the line for this episode too... my appreciation for it only increased when i learned there were societies on earth where you have to know the full shared history of proverbs in order to effectively communicate (at least, at a high level)19:41
anteayafungi: yes, 3 patches, those were them19:41
*** thorst has quit IRC19:41
mordredjeblair: ++19:41
jeblairmordred: well, i mean, that's what the tachyon langugae compensator is for19:41
pabelangermordred: clarkb: The reason we didn't catch with glean testing, is because systemd will run glean per interface, where testing does all.  We should fix that with a follow patch19:41
fungijeblair: yes, it's a good episode, i've just seen it too many times for it to be anything other than annoying at this point19:41
pleia2rcarrillocruz: yes, that's totally how it will be ;)19:41
mordredjeblair: oh. right. I forgot19:41
jeblairfungi: it does go on a bit19:42
mordredpleia2: ++19:42
*** dkehn has quit IRC19:42
mordredI mean - pleia2 ++ for sure - but I was going for pabelanger ++19:42
pleia2all the ++s19:42
fungianteaya: so it does look to me like they've all merged now19:43
fungii'm going ahead with reenabling puppet in that case19:43
rcarrillocruzmordred: one thing i've noticed with glean and the handling with vlans, which i believe it was not always the case, is that the raw interface gets configured the interface as 'inet manual' , i.e. it doesn't pull from dhcp19:43
*** asselin__ has joined #openstack-infra19:43
rcarrillocruzi believe it would be better if we had the raw interface also do a dhcp19:43
*** links has joined #openstack-infra19:43
*** thorst_ has quit IRC19:43
rcarrillocruzthat way if we ever get lose access to the public ip range, we can always ssh from the baremetal node to the private interfaces19:44
rcarrillocruzwithout the need to go thru ilo textcons, etc19:44
fungiwoah, we're down to just 4 projects still reindexing19:44
fungizaro: ^19:44
fungiunsurprisingly, they are: neutron, nova, openstack-manuals, tempest19:44
*** Na3iL has joined #openstack-infra19:44
zarook, so i assume you can check that querying the renamed project changes are working now19:44
*** asselin_ has quit IRC19:45
anteayafungi: yes, all merged19:45
zaroyou can check logs for 'INFO :' messages.19:45
zarofungi: when done you'll get something like 'INFO : Reindex to version 14 complete'19:46
*** dkehn_ has joined #openstack-infra19:46
*** jklare_ has quit IRC19:46
*** jklare has joined #openstack-infra19:46
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: [WIP] Make builder recognize manual build requests
zaroyep, i am able to query changes for openstack/zun19:47
fungizaro: indeed, right now all we have is:19:47
fungi[2016-09-02 19:02:09,155] INFO : Starting online reindex from schema version 14 to 1419:47
fungithat looks like when i sent the reindex api command19:47
*** amitgandhinz has quit IRC19:48
openstackgerritBen Nemec proposed openstack-infra/tripleo-ci: Don't send metrics to graphite in failed jobs
*** shardy has joined #openstack-infra19:49
openstackgerritDavid Shrewsbury proposed openstack-infra/nodepool: [WIP] Make builder recognize manual build requests
*** dkehn has joined #openstack-infra19:49
openstackgerritPaul Belanger proposed openstack-infra/glean: Make sure interface_name is validate for network files
fungiand now we're down to just neutron and nova19:51
pabelangermordred: clarkb: okay, I think that is everything^ until we review, release glean. We don't have ubuntu-xenial in rax19:51
fungiansible should kick off to run manage-projects in about 8 minutes19:51
*** ddieterly is now known as ddieterly[away]19:51
*** annegentle has quit IRC19:51
*** pblaho has quit IRC19:53
anteayaand it is labour day weekend19:53
mordredanteaya: not to be pedantic ...19:54
pleia2I know where this is going19:54
mordredanteaya: but in murica, it's labor day weekend ...19:54
anteayaso it is19:54
anteayaI was talking about Canada's labour day weekend19:54
mordredso you were19:54
pleia2is that this weekend too?19:54
anteayaindeed it is19:54
mordredI'm so used to thanksgivings being on different days19:54
anteayathey are so19:54
mordredit would be cool if thanksgiving had two different spellings19:55
fungiwow, a labour day weekend and a labor day weekend on the same weekend19:55
anteayano u in thanksgiving19:55
mordredanteaya: we could add one19:55
*** madhuvishy is now known as madhuvishy|food19:55
anteayafungi: who would have thought19:55
fungithanks gouging19:55
mordredanteaya: thauksgiving ?19:55
pabelangerthanksgiving in a month!19:55
anteayaha ha ha19:55
pleia2jeblair: still no u19:55
mordredof course - we have boxing day sales in the US now19:55
*** links has quit IRC19:55
mordredso the homogenization continues19:55
pabelangerleifmadsen: brigden fair! ^19:55
anteayadid you get boxing day from the uk via canada?19:56
*** spzala has joined #openstack-infra19:56
mordredwe don't have boxing day19:56
mordredjust sales on it19:56
mordredbecause selling things19:56
anteayaalways need to have a sale19:56
mordredwe'll adopt ANYTHING if it can be an excuse for a sale19:56
*** jcoufal has quit IRC19:56
*** gyee has joined #openstack-infra19:56
mordredheck - we'd adopt the metric system if someone could link it to black friday19:56
anteayaha ha ha19:57
anteayasomeone should tell NASA19:57
*** mriedem_afk is now known as mriedem19:57
anteayathat's what it takes19:57
* mordred points out that we _did_ adopt the metric system in two places ... coke and Coke19:57
anteayaI don't understand that reference19:57
mordredcoke is sold in grams and kilos. Coke is sold in liters19:57
anteayaha ha ha19:58
mordredanteaya: cocaine and coca-cola19:58
anteayaokay now I'm caught up19:58
mordreda 2-liter bottle is an extremely common thing in the US19:58
mordredand nobody seems to have a problem with it19:58
anteayathe druggies are always at the cutting edge of tech19:58
mordredanteaya: I'm guessing it's because cocaine is an import19:58
anteayamoar Coke19:58
mordredand doing the conversion would be a lot of work19:58
anteayaha ha ha19:58
anteayametric for the laxy19:59
mordredwhereas weed is almost always grown domestically, thus the continued selling in ounces and pounds19:59
openstackgerritMerged openstack/os-client-config: Don't build releasenotes in normal docs build
anteayaha ha ha19:59
mordredbut yeah - good for coca-cola for shifting extremely sugary drugs to metric19:59
mordredI'm guessing it makes global commerce easier for them19:59
anteayano way they did it as a public service19:59
*** jcoufal has joined #openstack-infra20:00
dstufftmedication is almost entirely done in metric too20:00
mordredI was just about to say that20:00
anteayathough I will say I'm hybrid20:01
anteayatempeture is either20:01
anteayadistances is km if longer than 50m and feet and inches for shorter distances20:01
anteayaif I am standing on a bathroom scale it is pounds20:01
dstufftI almost entirely use imperial, not because I think it's better, but because the relative sizes of things is stuck in my head as imperial20:02
anteayabaking can be done in ml/mg/g/kg or imperial20:02
anteayaliquid measure is cups or oz, rarely ml20:02
pabelangermordred: clarkb: looks like is what broke rackspace, we must be using an old version of glean in the images that are still working20:03
anteaya12 ft is 12 ft not 4m20:03
anteayamy height is feet and inches not cm and m20:03
anteayadstufft: yeah, this is why I do temp either way20:04
*** akshai has quit IRC20:04
*** openstackgerrit has quit IRC20:04
irtermitepabelanger: why for you break rackspace? at least Rackspace Private Cloud is solid.  ;)20:04
anteayaI always talk to my dad about the weather and he is in farenheit20:04
*** openstackgerrit has joined #openstack-infra20:04
fungii do all of my baking in metric, mainly because it's convenient that 1ml or 1cc of water is ~ 1g20:04
anteayafungi: yes20:05
fungiand much of my baking is proportional by weight to the water content20:05
dstufftI have to do math to figure out if 100kg is a lot or not but I know what 220lbs is :V20:05
anteayamy scales are metric, my measure devices are in cups most of them20:05
irtermitewell, seeing as how 1m =/= 1yard, that statement is correct no matter how you look at it anteaya  ;)20:05
anteayadstufft: :)20:05
mgagneweather is celsius for us (canada), pool temperature is fahrenheit. :D20:05
fungithat's a fun one20:05
anteayairtermite: yep20:06
anteayamgagne: you do pool temp in fahrenheit?20:06
irtermitemgagne: canada is weird to begin with, and that's why we love you, oh hat of the uS20:06
anteayamgagne: where do you live?20:06
mgagneanteaya: montreal, canada20:06
anteayamgagne: awesome20:06
anteayamgagne: I'm in ontario20:06
*** armax has joined #openstack-infra20:06
*** tqtran has quit IRC20:07
*** gouthamr has joined #openstack-infra20:07
anteayamgagne: I'll try to remember to ping next time I'm going to be in town20:07
anteayalike when I get EmilienM to come to montreal20:07
mgagnebody temp is C too. 100F? ¯\_(ツ)_/¯ it's impossibru, you would be boiling :D20:07
anteayaI can do body temp either way20:07
anteayastarting learning lifeguard things and the first aid bits still used F20:08
anteayawhen I started to learn lifeguard things20:08
anteayalong time ago20:08
*** spzala has quit IRC20:10
anteayafungi: how much is left to reindex?20:12
fungianteaya: still neutron and nova20:12
fungidunno how much progress there is on them20:13
anteayaah thanks20:13
fungias long as we don't rename nova, or neutron (again), this online reindexing seems to be a great speed-up for us20:13
anteayaha ha ha20:13
*** spzala has joined #openstack-infra20:14
anteayawith added bonus we debugged the playbook plus tested it20:14
anteayaand I know better now what you need for your ansible config file20:14
*** ddieterly[away] is now known as ddieterly20:14
anteayaand updated workflow instructions patch20:15
*** ilyashakhat_mobi has quit IRC20:15
*** roxanaghe has quit IRC20:15
*** shashank_hegde has joined #openstack-infra20:16
fungiyep. we still need some .gitreview patches proposed to the 5 renamed repos20:17
fungiif anybody feels like pushing those up20:17
pleia2ah, sure20:17
dhellmannclarkb : looking over my notes, I forgot to tell you that it was safe to go ahead with whatever image rebuilding work you were planning (which I think is separate from the gerrit maintenance). We're done with branches.20:17
fungithanks pleia2!20:18
fungii think i'm going to crack open a beer and finish some storm prep20:18
pleia2good idea20:18
anteayafungi: sounds good, what do you have left to do?20:18
*** spzala has quit IRC20:18
fungianteaya: no idea--that's what i'm going to go find out next20:18
anteayaah, good hunting20:19
pleia2window focus20:20
pleia2(karbor was already done at some point btw)20:20
*** kgiusti has left #openstack-infra20:20
*** jamesdenton has quit IRC20:21
fungithey may be on the ball and already proposing these themselves20:21
pleia2python-karborclient too20:21
fungii'm betting karbor-dashboard as well then20:21
*** e0ne has quit IRC20:21
pleia2seems not20:21
pleia2gotta keep me on my toes20:22
fungii so wanted to shorten that one to kardboard20:22
*** jcoufal_ has joined #openstack-infra20:22
anteayaha ha ha20:22
anteayaclarkb: did you see tim bell was asking about running elastic search on the operators mailing list?20:23
anteayajesusaur: ^^20:23
pleia2alright, all done20:23
pabelangerclarkb: I reverted to a working ubuntu-xenial image in rackspace, it is using an older version of glean that doesn't have the bug20:23
*** jcoufal has quit IRC20:23
anteayapleia2: thank you20:24
clarkbpabelanger: oj20:24
clarkber ok20:24
pabelangerclarkb: this is also likely why we cannot launch nodes in rax-iad right now too... glean is broken20:24
*** eharney has quit IRC20:25
EmilienManteaya: September 13 and 14 i'll be in MTL20:25
EmilienManteaya: we have an OpenStack Meetup20:25
EmilienManteaya: I bought train tickets today, feel free to join20:25
anteayaEmilienM: awesome, thank you :)20:26
anteayaI'll see which way the wind blows for those days20:27
fungitristanC: tonyb: that deathclock on the election page is an awesome feature. do the events advance automatically or do you have to merge changes to switch to the next current event?20:27
anteayacould stand some good monteal food for a bit :)20:27
anteayaEmilienM: does the meetup have a url?20:29
*** shashank_hegde has quit IRC20:29
prometheanfirehi, we (requirements) good to do the smaug migration
anteayaprometheanfire: are you asking or announcing?20:30
pabelangerclarkb: We might as well update the upstart script for glean too while we are at it. I'm seeing some EOFErrors on them too20:30
*** esikachev has quit IRC20:30
prometheanfireforgot the ?20:30
fungiprometheanfire: 353305 should be safe at any time20:30
fungiprometheanfire: i believe 353371 needs to wait until python-karborclient has a release on pypi20:31
prometheanfirebecause of the depends20:31
fungisince requirements jobs are going to try to install everything listed in the global reqs and fail on that addition20:32
prometheanfireI think we have to wait on the release for all of it20:32
prometheanfireunless karbor (and dashboard) have been published already20:32
*** Na3iL has quit IRC20:32
fungiprometheanfire: the projects.txt file should be fine now... it's only used to determine which projects get requirements update proposals submitted to them i think20:32
prometheanfireah, right20:33
*** tongli has quit IRC20:33
prometheanfirenot all there today20:33
fungiand if 353305 isn't merged, requirements proposals are going to fail (perhaps at best skip the old repos since it can no longer check them out, i forget if it's been made that robust yet)20:33
prometheanfireand yes, that's all that it's used for20:34
*** gouthamr has quit IRC20:34
anteayafungi: I missed a system config change for smaug to karbor: cc pleia220:34
prometheanfireso the bot knows what to submit to20:34
fungianteaya: oh, i expect they want 353309 to coincide with a channel move20:34
EmilienManteaya: I'm not sure :(20:35
anteayaEmilienM: okay thanks20:35
EmilienManteaya: but it's not updated :(20:35
fungianteaya: anyway, if they're okay having channel logs cut over from the old channel from the new one right now, then i'm happy to approve that20:35
fungithere aren't any meetings at this time on a friday afaik20:35
anteayathe channel is operative20:35
fungii don't have bandwidth currently to help them set up channel forwarding from old to new or anything like that at the moment though20:36
anteayafor the channel info it says Flags : GUARD20:37
anteayaI asked in #openstack-karbor if someone could join us and tell us what they want20:38
anteayabased on the names in the channel I'm guessing for many of them it is early saturday morning for them20:39
jesusauranteaya: i hadn't see that email, thanks for the poke20:39
anteayajesusaur: sure, I'm feeling tim would enjoy discussing things with you20:40
anteayaheap size and all that20:40
*** askb has joined #openstack-infra20:41
*** tonytan_brb has quit IRC20:42
anteayaEmilienM: indeed the website needs a bit of a refresh20:42
EmilienManteaya: only a bit?20:42
anteayaEmilienM: is the meetup two days or that the meetup occurs when you are there for those two days20:42
anteayaEmilienM: I'm trying to be nice, hey it exists20:42
* anteaya learned extant today, thanks jeblair20:43
EmilienManteaya: 13th evening only20:43
anteayaEmilienM: ah thank you20:43
EmilienMit will be from 6 to 8 something like this20:43
anteayaEmilienM: sitting in an office other than that for you?20:43
anteayaEmilienM: awesome thanks20:43
EmilienManteaya: I guess you can come, I'll do visitor badge :)20:43
anteayathank you :)20:43
anteayalet me think about it20:43
anteayabeen very busy with home things the last two weeks20:44
anteayahaven't thought that far ahead yeat20:44
*** salv-orlando has joined #openstack-infra20:44
openstackgerritMerged openstack-infra/system-config: Add images profile for openstackci oscc clouds and add it to infracloud
*** shashank_hegde has joined #openstack-infra20:46
*** thorst has joined #openstack-infra20:46
*** markvoelker has joined #openstack-infra20:47
anteayathink I'll weekend now20:47
pleia2enjoy, anteaya20:48
pleia2thanks for your help20:48
*** amitgandhinz has joined #openstack-infra20:48
*** matt-borland has quit IRC20:48
*** trown is now known as trown|outtypewww20:49
*** shashank_hegde has quit IRC20:49
*** AnarchyAo has quit IRC20:53
*** eranrom has quit IRC20:53
*** amitgandhinz has quit IRC20:54
*** tqtran has joined #openstack-infra20:54
*** salv-orlando has quit IRC20:54
*** Goneri has quit IRC20:55
*** piet has quit IRC20:56
jesusaurthe elasticsearch data is backed on cinder volumes, right? cacti only shows disk usage for /20:58
*** ddieterly is now known as ddieterly[away]20:58
*** fguillot has quit IRC20:59
*** e0ne has joined #openstack-infra21:01
*** e0ne has quit IRC21:03
*** rajinir has quit IRC21:05
fungijesusaur: yep21:06
fungimaybe net-snmp is having trouble finding those filesystems21:06
*** eharney has joined #openstack-infra21:07
*** salv-orlando has joined #openstack-infra21:08
*** cardeois has quit IRC21:09
raddaouipabelanger, fungi, clarkb networking is fixed as you wanted, mirror can get public ipv4 address assinged directly21:09
fungiraddaoui: oh, that was fast!21:10
mrhillsmanraddaoui has done a great job seeing this through :)21:11
*** roxanaghe has joined #openstack-infra21:11
*** madhuvishy|food is now known as madhuvishy21:11
raddaouiyeah I didn't think it will be that fast as well21:11
*** raildo has quit IRC21:11
raddaouithx mrhillsman :)21:12
mrhillsmanman thank you lol21:12
mrhillsmanmaking my job 200% easier all the time21:12
pabelangermordred: clarkb: okay, have glean testing fixed. will push something up later tonight21:13
openstackgerritMerged openstack/os-client-config: Update reno for stable/newton
*** shardy has quit IRC21:14
*** Thelo_ has quit IRC21:15
*** ldnunes has quit IRC21:15
*** gyee has quit IRC21:16
*** rtheis has quit IRC21:18
*** sdake has joined #openstack-infra21:19
*** ilyashakhat_mobi has joined #openstack-infra21:19
*** Thelo_ has joined #openstack-infra21:20
*** bauwser is now known as bauzas21:20
*** Hal1 has quit IRC21:21
mordredraddaoui: wow! nicely done!21:21
*** ijw has joined #openstack-infra21:21
* mordred hands raddaoui a mostly unused pie he found that looks probably tasty21:21
*** dprince has quit IRC21:22
*** Thelo_ has quit IRC21:24
*** lascii is now known as alaski21:27
*** roxanaghe has quit IRC21:28
*** roxanaghe has joined #openstack-infra21:30
*** inc0 has quit IRC21:31
*** gyee has joined #openstack-infra21:32
*** Hal1 has joined #openstack-infra21:33
clarkbraddaoui: tytyty21:34
clarkbpabelanger: cool I am back to my desk now is there anything I can help with that?21:34
clarkbpleia2: fungi anteaya everything seems to have gone ok? anything else I can do to help while other people make sure their homes don't fly away in a hurricane?21:36
clarkbif not I am going to ask for reviews on topic:newton-xenial and
pleia2I don't think we confirmed the online reindex was done21:37
*** roxanaghe has quit IRC21:37
pleia2but seems otherwise finished21:37
clarkbpleia2: cool. Is that just running the show queues command ong errit to see if its done?21:37
pleia2I think we just look at the log21:37
pleia2I haven't logged in21:38
clarkbkk /me looks21:38
zaroyou can check either21:38
pleia2o/ zaro21:38
zarocpu is way down now21:38
zaromeaning its probably done21:38
*** shashank_hegde has joined #openstack-infra21:39
clarkbbaaa7f86              19:02:09.171      Index all changes of project openstack/nova21:39
clarkbthats the only one that says index so I think likely all but nova are done21:39
*** roxanaghe has joined #openstack-infra21:40
*** ianw has quit IRC21:40
clarkbI ran gerrit show queue -w21:40
*** xarses has quit IRC21:41
clarkblooks like we may have some broken github repos that are failing replication. I am going to poke around on gerrit and see if I can fix that21:41
zaroi guess it's working even better than i expected21:41
zarodidn't even eat up that much CPU21:41
raddaouiclarkb: anytime :)21:41
*** tonytan4ever has joined #openstack-infra21:42
*** signed8bit is now known as signed8bit_Zzz21:43
clarkbya looks like they don't have the gerrit team on them with write perms21:43
clarkbso gonna add that and see if I can get the gerrit queues to empty21:44
*** tpsilva has quit IRC21:44
pleia2was this ones I renamed today, or just random other ones?21:44
*** ijw has quit IRC21:45
*** rfolco has quit IRC21:45
jeblairit's the deb repos21:46
clarkbpleia2: jeblair ya its the deb repos21:46
jeblairthey didn't get set up right because jeepyb exceeded the github api limit because they were all created at once21:46
*** tonytan4ever has quit IRC21:48
*** amitgandhinz has joined #openstack-infra21:50
openstackgerritJames E. Blair proposed openstack-infra/zuul: Use NodeSet objects in jobs and NodeRequests
openstackgerritJames E. Blair proposed openstack-infra/zuul: Re-enable test_failed_change_at_head
openstackgerritJames E. Blair proposed openstack-infra/zuul: Add nodeset as a top-level configuration item
*** rlandy has quit IRC21:52
*** tphummel has quit IRC21:52
*** gordc has quit IRC21:54
*** Goneri has joined #openstack-infra21:55
*** amitgandhinz has quit IRC21:55
*** ilyashakhat_mobi has quit IRC21:57
*** gluytium has quit IRC21:58
*** ddieterly[away] has quit IRC21:58
*** Goneri has quit IRC21:59
openstackgerritMerged openstack-infra/zuul: Use {{ ansible_host }} for ssh-keyscan
*** ijw has joined #openstack-infra22:01
*** ijw has quit IRC22:02
*** gluytium has joined #openstack-infra22:03
jeblairpabelanger: you might want to see the tortured comment i left on
*** tphummel has joined #openstack-infra22:05
*** hashar has quit IRC22:05
*** jcoufal_ has quit IRC22:06
*** jcoufal has joined #openstack-infra22:07
*** xyang1 has quit IRC22:09
*** ianw has joined #openstack-infra22:11
*** dimtruck is now known as zz_dimtruck22:12
*** xarses has joined #openstack-infra22:12
clarkbI think reindexing is done22:13
*** ijw has joined #openstack-infra22:15
*** david-lyle has quit IRC22:15
*** ijw has quit IRC22:15
*** ijw has joined #openstack-infra22:16
*** david-lyle has joined #openstack-infra22:17
* fungi checks22:19
fungi[2016-09-02 19:02:09,155] INFO : Starting online reindex from schema version 14 to 1422:19
fungi[2016-09-02 22:07:48,331] INFO : Reindex to version 14 complete22:20
fungi[2016-09-02 22:07:48,331] INFO : Using schema version 1422:20
jeblairyay we're at version 14!22:20
fungiso only took 3 hours22:20
jeblairversioun 14 is dead! long live version 14!22:20
funginot bad at all22:20
jeblairoops, there's that extra u from earlier22:20
*** pradk has quit IRC22:20
* fungi vanishes again22:21
clarkbjeblair: btw 312647 seemed fine after a recheck.22:21
zarohhallelujah! can have a good weekend22:21
clarkbthe queue is also clear now22:22
clarkbI got those github projects sorted22:22
pleia2online reindex is good22:22
*** javeriak has quit IRC22:22
jeblairclarkb: yep! logs look good22:23
pabelangerjeblair: Agreed, part of the reason why I haven't moved forward with it too.  I think we can revisit once zuulv3 has landed22:25
pabelangerclarkb: just pushing up patch now22:25
*** shashank_hegde has quit IRC22:27
*** tkelsey has joined #openstack-infra22:28
clarkbwe launched over 30k jobs per day a couple days last week22:29
jeblairtime to update the slides! :)22:30
openstackgerritPaul Belanger proposed openstack-infra/glean: Fix glean when launched by systemd
pabelangerclarkb: ^22:31
*** tkelsey has quit IRC22:32
*** ijw has quit IRC22:32
*** jcoufal_ has joined #openstack-infra22:32
*** ddieterly has joined #openstack-infra22:32
tonybfungi: the events iin the table get marked as past and the timer automagically moves to the next one IIRC22:33
*** jcoufal has quit IRC22:33
zaroalright, good work today infra team.  feeling good after that online index so i'm gonna do early weekend :)22:34
clarkbpabelanger: I am going to need a whole lot more comments :) I have no idea why skipping an interface would be what we want there22:35
*** ddieterly has quit IRC22:36
pabelangerclarkb: Its a hack right now, we should likely ask mordred how to handle that better.  Basically the test fixtures are setup to provider multiple interfaces, when we run against a single interface, tests fail because the fixtures still reference the other eth interfaces, but glean only build eth022:37
tonyb#success election content moved from the wiki to a gerrit workflow!  Thanks tristanC and everyone on the Infra team that made it happen!22:37
openstackstatustonyb: Added success to Success page22:37
clarkbpabelanger: right but what is the fix for rax?22:37
clarkbpabelanger: basically the change doesn't explain what was broken before or how it fixes it so I am lost22:38
*** claudiub has quit IRC22:38
pabelangerclarkb: sure, I can update the commit message. The fix is line 371 in We didn't 'continue' in the top level loop, only the sub loop.22:39
clarkbpabelanger: basically glean is full of hacks and none of them are documented. When I run into new ones that I have to debug I try to push a change up to explain what is going on like with the multiple json keys for network info. So would be good to be very clear what is going on in the code there that makes rax and not rax work22:40
openstackgerritMerged openstack-infra/irc-meetings: Update the QA team meeting chair
pabelangerclarkb: yes, let me push up some comments in line22:40
clarkbpabelanger: thansk. Also reading the old code its basically just a noop22:41
clarkbpabelanger: all it does is go through each of the mac addrs from the config drive and see if they arein sysfs but doesn't do anything based on that?22:41
clarkbyay fun bugs I see the problem now22:42
clarkbbut ya a comments saying "If we can't map the interfaces mac addresses to sysfs we skip this interface entirely" would probably help maybe with an example/explanation for why this might occur if we know why22:42
pabelangerclarkb: Right22:42
pabelangerWill add that now22:43
clarkbthe intent of that code was to continue the outer loop but only continued the inner loop22:43
pabelangerya, that's how it originally worked22:43
pabelangerbut when raw_macs what added, it got broken22:43
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add deb-python-{keystoneauth1,requestsexceptions}
openstackgerritPaul Belanger proposed openstack-infra/glean: Fix glean when launched by systemd
pabelangerclarkb: open to suggestions for line 191 in too22:47
*** shashank_hegde has joined #openstack-infra22:47
*** yamahata has quit IRC22:47
*** markvoelker has quit IRC22:47
openstackgerritThomas Goirand proposed openstack-infra/project-config: Add deb-python-{keystoneauth1,requestsexceptions}
clarkbpabelanger: might be slightyl clearer to have those checks in another sub function22:48
zigopabelanger: clarkb: Could you please accept these 2 repos? ^22:48
clarkbpabelanger: but other wise I think thats the right way to filter that22:49
clarkbpabelanger: because in the case of just setting eth0 the other interfaces won't be touched and we would fail otherwise22:49
*** yamahata has joined #openstack-infra22:49
*** snarwade has quit IRC22:50
pabelangerzigo: in the middle of something atm, I'll have to look in a bit22:50
zigoclarkb: Do you have time?22:50
zigoIt's yet another blocker... :/22:50
zigofungi: You maybe?22:51
clarkbpabelanger: left a nit but you have to repush anyways because pep8 failed22:51
*** amitgandhinz has joined #openstack-infra22:51
*** mmedvede has quit IRC22:51
clarkbpabelanger: so hopeflly we can address the nit :)22:51
clarkbpabelanger: and seems like the overall issue is that systemd invokes glean per interface rather than doing a whole bunch at once. Which is what led to that issue. Makes sense22:52
*** snarwade has joined #openstack-infra22:52
*** Apoorva has quit IRC22:53
*** ijw has joined #openstack-infra22:55
openstackgerritPaul Belanger proposed openstack-infra/glean: Fix glean when launched by systemd
*** amitgandhinz has quit IRC22:55
*** ijw has quit IRC22:56
pabelangerclarkb: right, I wonder if we should revisit that for systemd at a later time. Based on the noise we seen from neutron yesterday22:56
clarkbya if we triggered on "bring up host networking" and just did it rather than trigger on every if up22:57
pabelangerYa, something to play with another day.22:58
jeblair[i really miss the named loops in perl.  eg, "last INTERFACE"]22:58
*** ijw has joined #openstack-infra22:59
zigojeblair: <--- Can you +2 workflow that one, so we can move ahead ?23:00
jeblairzigo: yep23:01
zigoThanks so much.23:01
*** rbrndt has quit IRC23:01
*** jeblair is now known as corvus23:01
*** asettle has joined #openstack-infra23:05
*** asettle has quit IRC23:10
clarkbpabelanger: +2 probably best to get that in next week rather than try and shove it in and rebuild this late in the week23:11
*** jcoufal has joined #openstack-infra23:15
openstackgerritMerged openstack-infra/project-config: Add deb-python-{keystoneauth1,requestsexceptions}
*** jcoufal_ has quit IRC23:16
*** mmedvede has joined #openstack-infra23:18
*** openstackgerrit has quit IRC23:18
*** openstackgerrit has joined #openstack-infra23:18
*** signed8bit_Zzz is now known as signed8bit23:19
*** spzala has joined #openstack-infra23:19
*** signed8bit is now known as signed8bit_Zzz23:19
openstackgerritMerged openstack-infra/irc-meetings: Add networking-ovn meeting
*** spzala has quit IRC23:23
pabelangerclarkb: I am okay with that23:26
*** thorst has quit IRC23:30
*** Swami has quit IRC23:32
*** thorst has joined #openstack-infra23:32
*** spzala has joined #openstack-infra23:34
*** jamesdenton has joined #openstack-infra23:37
*** ijw has quit IRC23:40
*** tqtran has quit IRC23:41
*** ijw has joined #openstack-infra23:41
openstackgerritMerged openstack-infra/devstack-gate: Use fallocate for swapfile if on ext
*** spzala has quit IRC23:44
*** ijw has quit IRC23:45
*** asettle has joined #openstack-infra23:46
*** thcipriani is now known as thcipriani|afk23:47
*** senk__ has quit IRC23:51
*** senk has joined #openstack-infra23:51
*** amitgandhinz has joined #openstack-infra23:52
*** amitgandhinz has quit IRC23:57

Generated by 2.14.0 by Marius Gedminas - find it at!