Wednesday, 2015-11-25

mtreinishbut maybe that just me00:00
*** shashank_hegde has quit IRC00:01
*** yamahata has quit IRC00:01
*** camunoz has quit IRC00:02
*** apoorvad has quit IRC00:02
*** Sukhdev has quit IRC00:02
openstackgerritSpencer Krum proposed openstack-infra/system-config: Add metrics as a reason to contribute a cloud  https://review.openstack.org/24101100:03
openstackgerritMerged openstack-infra/puppet-jeepyb: Remove mysql::python class inclusion  https://review.openstack.org/23498400:03
*** sridhar_ram has quit IRC00:03
*** sflanigan has quit IRC00:04
*** sridhar_ram has joined #openstack-infra00:04
mordrednibalizer: ZOMG ZOMG ZOMG00:05
*** pahuang has quit IRC00:05
mordrednibalizer: ZOMG ZOMG ZOMG00:05
mordrednibalizer: ZOMG ZOMG ZOMG IT WORKED ZOMG ZOMG ZOMG00:05
*** apoorvad has joined #openstack-infra00:05
mordredhttp://puppetdb.openstack.org/report/ask-staging.openstack.org/eddc27c30b9e78495dee77950f18808e1131b95200:06
nibalizermordred: oshi!00:06
*** alexpilotti has joined #openstack-infra00:06
nibalizermuAHAHHAHAHAHA00:06
mordredthat is a report in puppetdb of running puppet apply via ansible00:06
mordredwith copied hiera data00:06
clarkbnice00:06
mordredok00:06
nibalizer'private apis' *scofff*00:06
nibalizerokay ya whats the status00:06
nibalizersince this trove thing is just BUILDING00:06
mordredI need to copy in a few of my changes and push them up00:06
nibalizerlike what patches need to land etc etc00:06
*** tsg has quit IRC00:06
mordredthen I'm going to want you fine people to land them00:06
nibalizerhow much did you ninja on the puppetmaster et etc00:06
mordrednothing is ninja instlaled00:07
mordredI did some things in my home dir00:07
clarkb(this is one of the really nice things working with ansible I have found)00:07
clarkbits super self contained00:07
*** shashank_hegde has joined #openstack-infra00:08
nibalizerok cool00:08
mordredclarkb: ++00:08
mtreinishnibalizer: so there is a saving state after building: https://github.com/openstack/trove/blob/master/trove/backup/state.py00:08
mtreinishI was looking to see if there was an api call to see the status (which I don't think there is)00:09
*** Sukhdev has joined #openstack-infra00:10
nibalizermtreinish: wooo00:10
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Fetch puppet logs and facts after puppet run  https://review.openstack.org/24044600:10
mordrednibalizer, clarkb: ^^ that please00:10
nibalizermordred: i can approve 249434 ya?00:11
mordredonce that is there, switching to apply is a patch to system-config00:11
mordrednibalizer: yes please00:11
*** apoorvad has quit IRC00:11
mordredand 24944500:12
clarkbmordred: nibalizer on https://review.openstack.org/#/c/249466/2 does that mean it will start copying for agent runs?00:12
clarkbnot a big deal just making sure I understand what is oging on00:12
mordredclarkb: nope00:13
mordredclarkb: it already defaults to no-op if you haven't set that value00:13
clarkbok00:13
mordredclarkb: so setting that value there was just preventing us from being able to flip the switch sanely00:13
* mordred is learning more about ansible :00:13
mordred:)00:13
clarkbright it overrode whatever you do outside the module00:13
mordredyup00:13
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow setting the number of threads for indexing Gerrit changes  https://review.openstack.org/24106000:14
openstackgerritMerged openstack-infra/ansible-puppet: Remove copy_hieradata value from vars file  https://review.openstack.org/24946600:14
clarkbmordred: where does the manfiest is defined thing come from?00:15
clarkbmordred: wait00:15
*** camunoz has joined #openstack-infra00:15
clarkbis is actually wow00:15
clarkbnot sure I care for that00:15
clarkbmordred: its saying if the var called manifest is not null00:15
mtreinishSlickNik: is there any way to check how much longer a trove backup is gonna take?00:15
* mtreinish asks optimistically00:15
mordredclarkb: ya - the current logic is "if you say manifest, do apply, otherwise do agent"00:16
mordredclarkb: we can make that different/better if you want00:16
clarkbmordred: no the behavior is correct, I am just grumping about ansibles language00:16
clarkbits no longer just yaml00:16
mordredactually ... I think there is a little bug00:16
clarkbits ctually tokenizing the values00:16
*** gildub has joined #openstack-infra00:16
mordredyah00:16
clarkband doing stuff00:16
clarkbwhich is :(00:16
*** sflanigan has joined #openstack-infra00:17
*** sflanigan has joined #openstack-infra00:17
mordredit should be when: puppetdb is defined in one place00:17
mordredone sec00:17
clarkbthe post facts is already when puppetdb is defined00:17
mordredya00:17
mordredbut the log things are all if manifest is defined00:17
mordredand we don't need to do that if we're not posting them00:17
clarkbah00:17
openstackgerritKhai Do proposed openstack-infra/system-config: Allow Gerrit to use 4 threads for indexing changes.  https://review.openstack.org/24173600:17
*** thorst has joined #openstack-infra00:18
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Fetch puppet logs and facts after puppet run  https://review.openstack.org/24044600:18
mordredclarkb: that00:18
openstackgerritMerged openstack-infra/system-config: Copy system-config and puppet modules everywhere  https://review.openstack.org/24943400:18
*** aspiers_ has joined #openstack-infra00:18
clarkbmordred: but puppetdb isn't defined until that last portion of the yaml file00:18
clarkboh it must be coming out of your vars00:19
*** thorst has quit IRC00:19
clarkband the one I see there is masking00:19
clarkbmordred: for synchronize the docs don't say if dest is a dest dir or a dest file path00:21
mordredyah00:21
openstackgerritK Jonathan Harker proposed openstack-infra/log_processor: Make the subunit-files config section optional  https://review.openstack.org/24949000:21
mordredclarkb: it's rsync synta00:21
nibalizerI'm going to approve 248940 because all conditions have been met (no more fedora-22) nodes00:21
mordredsyntax00:21
clarkbmordred: you ahve written it in a way that implies dest path00:21
*** pahuang has joined #openstack-infra00:22
mordredclarkb: yah. dest path.00:22
openstackgerritMonty Taylor proposed openstack-infra/system-config: Start copying hiera data everywhere  https://review.openstack.org/24949100:22
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949200:22
mordredclarkb: there are the patches to make it go live00:22
mordredin case those help reviewing the ones you're looking at00:22
clarkbmordred: ok so we will have a dir full of reports for that host or only ever have the last report?00:24
clarkbsorry I am having a hard time parsing ansible docs00:24
greghaynesjesusaurus: just stats IMO00:24
*** markvoelker has quit IRC00:26
clarkboh its rsync00:27
clarkbso that should work and make a bunch of files in that dir00:27
*** apoorvad has joined #openstack-infra00:27
*** bcourt has quit IRC00:28
clarkbmordred: still there? I think I see an actual bug00:29
*** dims has joined #openstack-infra00:30
*** dims_ has quit IRC00:31
*** akshai_ has quit IRC00:31
clarkbnevermind greghaynes straightened me out00:32
greghaynesclarkb: I mean, every time I use synchronize I have to spend about 10 mins to figure out what is actually going to happen00:33
greghaynesso it is just par for the course00:33
nibalizermtreinish: we're still building00:37
*** amuller has quit IRC00:37
*** sergmelikyan has joined #openstack-infra00:37
mtreinishnibalizer: sigh, are you online tomorrow?00:38
nibalizeryes00:38
nibalizerleave the daemons off or turn em back on?00:38
mtreinishok, we might have to pick it tomorrow00:38
*** tsg has joined #openstack-infra00:38
mtreinishwe can probably leave them off00:38
mtreinishwe were expecting the migration to take days, so if the backup takes forever theres no harm00:38
mtreinishgearman should queue up all of the events for when we turn it back on anyway00:39
openstackgerritK Jonathan Harker proposed openstack-infra/log_processor: Create processor subclasses for different job types  https://review.openstack.org/24949600:39
*** sfinucan has quit IRC00:39
jeblairjesusaurus: what greghaynes said :)00:40
*** dizquierdo has quit IRC00:42
*** baoli has joined #openstack-infra00:42
*** sergmelikyan has quit IRC00:43
*** baoli_ has joined #openstack-infra00:43
mordredclarkb: yeah?00:44
mordredcool!00:44
*** yamamoto has joined #openstack-infra00:44
clarkbmordred: well then I found other things so I -1'd00:44
clarkbsorry :)00:44
mordredpiddle00:44
*** sridhar_ram has quit IRC00:44
clarkbone is a wrong comment the other is actually I think you were testing and going easy mode but should be fixed for production00:44
*** Sukhdev has quit IRC00:45
*** yamamoto has quit IRC00:46
mordredso - the second on00:46
mordredif you look at http://puppetdb.openstack.org/report/ask-staging.openstack.org/eddc27c30b9e78495dee77950f18808e1131b95200:46
mordredit's got runtimes in it00:46
*** baoli has quit IRC00:47
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add *.egg* to .gitignore  https://review.openstack.org/24900800:47
*** Sukhdev has joined #openstack-infra00:47
clarkbmordred: cool let me see what it looks like fro mthe node list00:48
clarkbreport is correct there00:48
mordredI can make that value real though easy enough00:48
clarkbnah it works in puppetdb00:49
clarkber puppetboard00:49
jeblairmordred: keepin it real00:49
clarkbso I think just the comment and we are good00:49
*** sridhar_ram has joined #openstack-infra00:50
mtreinishnibalizer: err, maybe it doesn't do a snapshot under the covers. It looks like it's doing a mysql backup (either with mysqldump or innobackupex) and then uploading that to swift00:50
mtreinishthe mysqldump fungi ran before took a long time iirc00:50
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Fetch puppet logs and facts after puppet run  https://review.openstack.org/24044600:51
openstackgerritgreghaynes proposed openstack-infra/nodepool: Add test for failing provider loading  https://review.openstack.org/24947000:51
mordredclarkb: I went ahead and did the other thing :)00:51
mtreinishnibalizer: https://github.com/openstack/trove/blob/master/trove/guestagent/strategies/backup/mysql_impl.py00:51
mtreinishnibalizer: I almost don't care anymore about the backup, the snapshot step was added because it was supposed to be quick00:52
*** nelsnelson has quit IRC00:52
*** Swami has quit IRC00:53
openstackgerritMerged openstack/os-testr: Add *.egg* to .gitignore  https://review.openstack.org/24900800:55
*** yamamoto has joined #openstack-infra00:55
*** yamamoto has quit IRC00:55
*** yamamoto has joined #openstack-infra00:56
*** bknudson has quit IRC00:56
*** bardia has quit IRC00:56
*** bknudson has joined #openstack-infra00:58
*** yamamoto has quit IRC00:58
*** kozhukalov_ has quit IRC00:58
*** alexpilotti has quit IRC00:59
*** apoorvad has quit IRC00:59
*** ajmiller has quit IRC00:59
*** alexpilotti has joined #openstack-infra00:59
*** salv-orl_ has quit IRC01:01
openstackgerritK Jonathan Harker proposed openstack-infra/nodepool: Write per-label nodepool demand info to statsd  https://review.openstack.org/24603701:01
jesusaurusjeblair: greghaynes: thanks for the feedback01:02
*** yamahata has joined #openstack-infra01:03
*** sridhar_ram1 has joined #openstack-infra01:03
*** sridhar_ram has quit IRC01:04
*** eil397 has quit IRC01:06
*** ccrouch has quit IRC01:07
*** doug-fish has joined #openstack-infra01:07
ruagairHeya fungi, if you're still about. Just read your comment from this morning's meeting and I'll answer it via emai lto -infra. I think that's best.01:08
*** sridhar_ram1 is now known as sridhar_ram01:08
*** wshao has joined #openstack-infra01:09
anteayaruagair: well how about start in channel01:10
anteayaso we don't send the message even inadvertantly that we are offering folks accounts01:10
*** eil397 has joined #openstack-infra01:11
*** doug-fish has quit IRC01:12
ruagairTrue.01:12
ruagairGood idea anteaya.01:12
ruagairI'll chat about it in the AEDT AM when everyone's back online.01:12
anteayaor just share now01:13
anteayafolks will read when next the are around01:13
anteayaso I think the main question is who is using the instance01:13
ruagairJust drove 1,500K, I think my mind has not arrived yet :-)01:13
anteayaand what is there understanding for doing so01:13
anteayaruagair: oh okay, well glad you arrived safely01:13
anteayaruagair: rest well01:14
reedanteaya, re:activity.o.o the project got never ported fully into infra so the only way to have modifications is by filing a ticket with the maintainers, Bitergia who used to be under contract but I don't know if the contract is still valid01:14
ruagairSo am I. There were numerous unsuccessful attempts on my life :-)01:14
*** Swami has joined #openstack-infra01:14
anteayaruagair: but you don't have to wait for folks to appear to post in channel, people will read01:14
anteayaruagair: I'm glad they were unsuccessful01:14
ruagairNoted.01:14
anteayaruagair: dodging bounty hunters were you?01:15
ruagairBouncy hunters.01:15
* jhesketh notes that some people will read... jhesketh is particularly bad at keeping up with scrollback01:15
anteayaruagair: interesting01:15
anteayajhesketh: we try to find you01:15
jheskethyep, I appreciate that :-)01:15
ruagairKangaroos are the suicide bombers of outback Australia.01:15
anteayareed: who would know if the contract is still valid?01:15
jheskethI try to read back through highlights, it's just the rest of the messages I miss01:15
anteayareed: as a way of figuring out how to address questions about activity.o.o?01:16
* ruagair is like jhesketh. Bad at scrollback.01:16
anteayajhesketh: well you are really good about asking, which helps01:16
anteayaruagair: okay well I think the salient point is folks in infra would like to have a discussion when decisions are made that affect expectations of users01:17
jheskethruagair: with my speed of reading/comprehension it would take me all day to read the scrollback in the number of channels I'm in01:17
reedanteaya, probably ttx is the most knowledgeable on the topic01:17
anteayaruagair: for instance handing out accounts on services01:17
reedanteaya, him or thingee01:17
anteayareed: wonderful thank you, hopefully I will find one of them tomorrow01:17
anteayareed: appreciate the reply01:17
ruagairGood point. Thanks for the headsup anteaya.01:17
anteayaruagair: yup, drink a lot of water and get some rest01:18
*** baoli_ has quit IRC01:18
*** baoli has joined #openstack-infra01:19
TheloHi all, I am trying to deploy devstack and stack.sh stop right after this instruction 2015-11-24 10:44:19.749 | + openstack --os-cloud=devstack-admin image create cirros-0.3.4-x86_64-uec --public --container-format ami --disk-format ami01:20
Thelo2015-11-24 10:44:20.499 | Could not determine a suitable URL for the plugin01:20
Thelodoes anyone know why?01:20
*** baoli has quit IRC01:21
*** glauco_ has quit IRC01:21
*** glauco_ has joined #openstack-infra01:22
*** baoli has joined #openstack-infra01:22
*** markvoelker has joined #openstack-infra01:27
*** darrenc is now known as darrenc_afk01:27
*** tiswanso has quit IRC01:29
*** Liuqing has joined #openstack-infra01:29
*** alexpilotti has quit IRC01:31
*** markvoelker has quit IRC01:31
*** baoli has quit IRC01:33
*** jyuso1 has joined #openstack-infra01:34
*** baoli has joined #openstack-infra01:34
*** zhenguo has joined #openstack-infra01:37
*** yamamoto has joined #openstack-infra01:40
stevemaranteaya: you'll be happy to know that the OSC gate is still passing :)01:40
openstackgerritDi Xu proposed openstack-infra/jenkins-job-builder: Support deep merge of defaults  https://review.openstack.org/15073001:43
*** tsg has quit IRC01:44
*** wshao has quit IRC01:45
*** gongysh__ has joined #openstack-infra01:46
*** Swami has quit IRC01:48
*** gildub has quit IRC01:49
*** zhurong has joined #openstack-infra01:52
*** btully has quit IRC01:52
*** mriedem has joined #openstack-infra01:52
*** shuquan has joined #openstack-infra01:52
cp16netmtreinish: howdy01:53
cp16netmtreinish: asking about trove backups?01:54
*** otter768 has joined #openstack-infra01:54
*** Daisy has joined #openstack-infra01:54
openstackgerritJames E. Blair proposed openstack-infra/project-config: Add git haproxy dashboard  https://review.openstack.org/24951301:54
*** fitoduarte has quit IRC01:55
*** tiswanso has joined #openstack-infra01:55
*** eil397 has quit IRC01:56
jeblairclarkb: ovh says it's customer ips, so we can shut down the old haproxy01:57
cp16netmtreinish: nibalizer: the trove mysql backup is defaulted to use the percona xtrabackup strategy.01:58
clarkbjeblair cool01:58
*** otter768 has quit IRC01:58
openstackgerritJames E. Blair proposed openstack-infra/project-config: Increase OVH to 79 nodes  https://review.openstack.org/24951401:59
mordredwoot02:00
*** shuquan has quit IRC02:01
*** Sukhdev has quit IRC02:02
*** darrenc_afk is now known as darrenc02:05
mordredclarkb: does https://review.openstack.org/#/c/240446/ look good to you now?02:06
*** camunoz has quit IRC02:08
*** shuquan has joined #openstack-infra02:09
*** sflanigan has quit IRC02:10
mtreinishcp16net: yep, I dug it up in the code02:10
*** glauco_ has quit IRC02:10
mtreinishcp16net: we should have just done a nova snapshot because the db has big perf issues right now so the backups take forever02:12
*** mtanino has quit IRC02:13
jeblairmtreinish: you can do a nova snapshot of a trove instance?02:13
mtreinishjeblair: there isn't anything I'm aware of that would block it. trove does lock the instance, but that doesn't prevent snapshotting iirc02:13
*** shashank_hegde has quit IRC02:14
jeblairmtreinish: i don't think the instance is visible in our tenant02:14
mordredyeah. we don' thave access to the nova02:15
mtreinishmordred: it uses a seperate nova?02:16
mtreinishjeblair: ok, then I guess we can't snapshot the instance02:16
mordredmtreinish: yes02:16
cp16netmtreinish: no you cant02:16
cp16netmtreinish: this trove instance is running at rax right?02:17
mtreinishcp16net: yep, I wasn't aware it talked to a seperate nova02:17
cp16netyeah its separate02:17
*** shuquan has quit IRC02:17
cp16netif you could snapshot the nova instance it would a security issue for trove02:18
*** Daisy has quit IRC02:18
*** Daisy has joined #openstack-infra02:19
*** aspiers_ has quit IRC02:19
*** thorst has joined #openstack-infra02:20
mtreinishcp16net: I don't really buy that argument, but whatever02:20
*** camunoz has joined #openstack-infra02:21
cp16netthere is a separate volume where the data lives02:21
*** signed8bit_ZZZzz is now known as signed8bit02:22
*** sflanigan has joined #openstack-infra02:23
*** Daisy has quit IRC02:23
*** shuquan has joined #openstack-infra02:23
*** nelsnelson has joined #openstack-infra02:24
cp16netmtreinish: it would be nice to be able to snapshot the volume02:24
*** bcourt has joined #openstack-infra02:24
*** thorst has quit IRC02:24
mtreinishcp16net: sure, I wasn't sure if it was a volume or the image disk being used. I don't actually have a rax account so I was just guessing02:24
mtreinishcp16net: oh, that actually reminds me of an issue we had like 7 months ago with the mysql tmpdir. You're probably the right person to ask about that02:24
cp16netmtreinish: what that?02:25
mtreinishwe weren't able to run alter tables on large tables because we ate all the tmp storage (I assume it was a tmpfs)02:25
mtreinishso we had to manually do it by making seperate tables and then renaming them02:26
mtreinishbut that's something I'm surprised hasn't come up before02:26
cp16netyeah thats very likely02:26
mtreinishit would probably be better to make a dir on the cinder volume for the tmp storage (or give that option)02:27
*** markvoelker has joined #openstack-infra02:28
cp16netyeah... there is a configuration management options available02:28
cp16netallows you to change conf values of the my.cnf02:29
mtreinishthere are, but we don't know the partition table of the disk02:29
*** tsg has joined #openstack-infra02:29
mtreinishso how can we change the tmpdir value to be on the volume02:29
mtreinishif we don't know where it's mounted02:29
mtreinishor if there is an empty dir we can use02:29
cp16netahh i doubt that would be available02:29
cp16netthe volume is usually mounted to /var/lib/mysql02:30
*** mrunge has quit IRC02:30
openstackgerritRichard Su proposed openstack-infra/project-config: Transitioning tripleo-common to tripleo  https://review.openstack.org/24952102:30
cp16netbut a deployer can make changes to that02:30
mtreinishsure, this was specifically on rax02:30
*** mrunge has joined #openstack-infra02:32
*** markvoelker has quit IRC02:33
*** gongysh__ has quit IRC02:35
*** shuquan has quit IRC02:36
* mordred brings up once again that things 'deployers can change' in almost all cases make things harder on users02:36
*** signed8bit is now known as signed8bit_ZZZzz02:38
*** sputnik13 has joined #openstack-infra02:38
*** dims has quit IRC02:39
ruagair+1 mordred02:39
mtreinishmordred: no argument from me02:40
mtreinishmordred: I'm just waiting for your infra mysql cluster to save the world02:40
*** bcourt has quit IRC02:41
mordredmtreinish: if only I could save the world with mysql clusters02:41
mtreinishmordred: I have faith :)02:42
openstackgerritRichard Su proposed openstack-infra/project-config: Transitioning tripleo-common to tripleo  https://review.openstack.org/24952102:45
cp16net:-P02:48
*** sputnik13 has quit IRC02:50
*** Piet has joined #openstack-infra02:51
*** zhurong has quit IRC02:53
*** signed8bit_ZZZzz is now known as signed8bit02:54
*** sridhar_ram has quit IRC02:54
*** Daisy has joined #openstack-infra02:59
*** Daisy has quit IRC03:04
*** eil397 has joined #openstack-infra03:04
*** salv-orlando has joined #openstack-infra03:05
*** eil397 has quit IRC03:10
*** eil397 has joined #openstack-infra03:11
*** dramalho has joined #openstack-infra03:11
*** camunoz has quit IRC03:12
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add meeting_ids to several meetings that were missing them  https://review.openstack.org/24953003:12
*** mriedem has quit IRC03:13
*** zhurong has joined #openstack-infra03:15
*** dramalho has quit IRC03:16
*** binbincong has joined #openstack-infra03:21
*** yuanying has quit IRC03:21
*** mtanino has joined #openstack-infra03:21
*** camunoz has joined #openstack-infra03:22
*** thorst has joined #openstack-infra03:22
*** _ody has quit IRC03:23
*** thorst has quit IRC03:26
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add meeting_ids to several meetings that were missing them  https://review.openstack.org/24953003:29
*** markvoelker has joined #openstack-infra03:29
*** amitgandhinz has joined #openstack-infra03:29
*** amitgandhinz has quit IRC03:30
*** amitgandhinz has joined #openstack-infra03:30
*** Sree has joined #openstack-infra03:31
*** weshay has quit IRC03:32
*** Sree has joined #openstack-infra03:32
*** camunoz has quit IRC03:33
*** darrenc is now known as darrenc_afk03:33
*** sflanigan has quit IRC03:33
*** changbl has quit IRC03:34
*** markvoelker has quit IRC03:34
*** unicell has quit IRC03:35
*** markvoelker has joined #openstack-infra03:35
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add colored output feature to subunit-trace  https://review.openstack.org/24900503:37
*** salv-orlando has quit IRC03:38
*** boris-42 has quit IRC03:38
*** achanda has joined #openstack-infra03:41
openstackgerritIan Wienand proposed openstack/diskimage-builder: Update default to F23  https://review.openstack.org/24729603:41
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use dnf to cleanup old kernels  https://review.openstack.org/24953903:41
openstackgerritIan Wienand proposed openstack/diskimage-builder: Don't remove yum in 99-remove-extra-packages  https://review.openstack.org/24954003:41
openstackgerritIan Wienand proposed openstack/diskimage-builder: Don't remove python3 & grubby in 99-remove-extra-packages  https://review.openstack.org/24954103:41
ianwgreghaynes: ^ my simple change is getting more complex :(03:42
*** changbl has joined #openstack-infra03:47
*** hichihara has joined #openstack-infra03:48
*** camunoz has joined #openstack-infra03:49
*** sflanigan has joined #openstack-infra03:50
*** sflanigan has joined #openstack-infra03:50
*** achanda has quit IRC03:51
*** salv-orlando has joined #openstack-infra03:51
*** otter768 has joined #openstack-infra03:55
*** salv-orlando has quit IRC03:56
*** darrenc_afk is now known as darrenc03:59
*** otter768 has quit IRC04:00
*** kushal has joined #openstack-infra04:04
*** gildub has joined #openstack-infra04:05
*** yuanying has joined #openstack-infra04:06
*** Thelo has quit IRC04:07
*** otter768 has joined #openstack-infra04:07
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add colored output feature to subunit-trace  https://review.openstack.org/24900504:08
*** thorst has joined #openstack-infra04:14
openstackgerritMasayuki Igawa proposed openstack/os-testr: Add colored output feature to subunit-trace  https://review.openstack.org/24900504:15
*** ivar-lazzaro has quit IRC04:16
*** Thelo has joined #openstack-infra04:17
*** thorst has quit IRC04:18
*** bhunter71 has quit IRC04:22
*** ramineni has joined #openstack-infra04:22
*** ramineni has left #openstack-infra04:22
*** signed8bit has quit IRC04:26
tsgclarkb: F22 liberasurecode package is available and tested fine with Swift.  Please OK patch 247936 if all else looks good.  Thanks :)04:26
*** rguillebert has quit IRC04:27
*** shashank_hegde has joined #openstack-infra04:34
*** tsg has quit IRC04:36
*** mahatic has joined #openstack-infra04:36
*** mohankumar has joined #openstack-infra04:37
*** tsg has joined #openstack-infra04:39
*** Liuqing has quit IRC04:41
*** doug-fish has joined #openstack-infra04:42
*** sergmelikyan has joined #openstack-infra04:42
*** Liuqing has joined #openstack-infra04:42
*** sdake has quit IRC04:42
*** hichihara has quit IRC04:43
openstackgerritIan Wienand proposed openstack/diskimage-builder: Update default to F23  https://review.openstack.org/24729604:44
openstackgerritIan Wienand proposed openstack/diskimage-builder: Don't remove yum in 99-remove-extra-packages  https://review.openstack.org/24954004:44
openstackgerritIan Wienand proposed openstack/diskimage-builder: Don't remove python3 & grubby in 99-remove-extra-packages  https://review.openstack.org/24954104:44
openstackgerritIan Wienand proposed openstack/diskimage-builder: Use dnf to cleanup old kernels  https://review.openstack.org/24953904:44
StevenKianw: While you're pushing up code, where does 'package-cleanup' come from, is that a Fedora thing?04:45
StevenK(See elements/redhat-common/finalise.d/01-clean-old-kernels failing everywhere in CI)04:45
*** doug-fish has quit IRC04:46
*** sergmelikyan has quit IRC04:46
*** ramineni has joined #openstack-infra04:49
*** aukhan has joined #openstack-infra04:49
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add meeting_ids to several meetings that were missing them  https://review.openstack.org/24953004:54
*** tsg_ has joined #openstack-infra04:55
*** FallenPegasus has joined #openstack-infra04:56
*** Ramanjaneya has joined #openstack-infra04:57
*** ddieterly has quit IRC04:57
*** tsg has quit IRC04:58
*** mtanino has quit IRC04:59
*** tsg_ has quit IRC05:02
raminenihi, im not able to open http://status.openstack.org/zuul/ , is it me only , or is it slow?05:04
ianwStevenK: yeah, i think it's the dnf removal05:05
ianwsee https://review.openstack.org/24954005:05
StevenKianw: Ah, in your chain rather than say master broken?05:07
*** rossella_s has quit IRC05:14
*** rossella_s has joined #openstack-infra05:14
*** Piet has quit IRC05:15
gmannramineni: yea, its down05:15
raminenigmann: ok, thanks05:15
*** salv-orlando has joined #openstack-infra05:18
*** thorst has joined #openstack-infra05:18
*** amitgandhinz has quit IRC05:19
*** btully has joined #openstack-infra05:20
*** doug-fish has joined #openstack-infra05:23
*** thorst has quit IRC05:23
*** mohankumar has quit IRC05:23
*** salv-orlando has quit IRC05:24
*** doug-fish has quit IRC05:27
*** rlandy has quit IRC05:29
*** mohankumar has joined #openstack-infra05:29
*** spzala has quit IRC05:35
ianwStevenK: yeah, there might be some ordering issues there.  i'll need to dig a bit05:36
*** hdd has joined #openstack-infra05:36
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Cloudpulse meeting  https://review.openstack.org/24956205:39
*** gongysh__ has joined #openstack-infra05:39
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Kosmos meeting  https://review.openstack.org/24956305:40
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Libvirt meeting  https://review.openstack.org/24956405:40
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Neutron QoS meeting  https://review.openstack.org/24956505:41
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Nova Hyper-V Team  meeting  https://review.openstack.org/24956605:42
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the OpenStack Community Infrastructure meeting  https://review.openstack.org/24956705:43
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Operators Tag Group meeting  https://review.openstack.org/24956805:44
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Satori meeting  https://review.openstack.org/24957105:45
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Remove the Neutron VPNaaS meeting  https://review.openstack.org/24957205:45
*** Liuqing has quit IRC05:47
*** Liuqing has joined #openstack-infra05:48
*** FallenPegasus has quit IRC05:48
*** abregman has joined #openstack-infra05:49
*** amotoki has quit IRC05:49
*** nadya_ has joined #openstack-infra05:56
*** nadya_ has quit IRC05:58
*** amotoki has joined #openstack-infra06:01
*** hurgleburgler has quit IRC06:01
*** hurgleburgler has joined #openstack-infra06:02
*** Sukhdev has joined #openstack-infra06:02
*** sflanigan has quit IRC06:03
*** asselin has quit IRC06:04
*** asselin has joined #openstack-infra06:04
*** camunoz has quit IRC06:05
*** hdd has quit IRC06:06
*** hurgleburgler has quit IRC06:07
*** pahuang has quit IRC06:07
*** amotoki has quit IRC06:10
*** binbincong has quit IRC06:11
*** pcrews has quit IRC06:12
*** amotoki has joined #openstack-infra06:15
*** pcrews has joined #openstack-infra06:16
*** camunoz has joined #openstack-infra06:17
*** sflanigan has joined #openstack-infra06:17
*** ramineni has left #openstack-infra06:17
*** mahatic has quit IRC06:18
*** rcernin has joined #openstack-infra06:20
*** amitgandhinz has joined #openstack-infra06:20
*** pahuang has joined #openstack-infra06:20
*** yamamoto has quit IRC06:22
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Flag that the Operators Tag Group Meeting is monthly  https://review.openstack.org/24958306:22
*** yamamoto has joined #openstack-infra06:23
*** amitgandhinz has quit IRC06:25
*** otter768 has quit IRC06:25
*** ddieterly has joined #openstack-infra06:28
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Add check_channels.py and testenv to run it  https://review.openstack.org/24724906:29
*** binbincong has joined #openstack-infra06:29
*** ddieterly has quit IRC06:33
*** aopincar has joined #openstack-infra06:37
openstackgerritMerged openstack-infra/irc-meetings: Flag that the Operators Tag Group Meeting is monthly  https://review.openstack.org/24958306:37
*** Liuqing_ has joined #openstack-infra06:38
*** Liuqing has quit IRC06:39
*** amotoki has quit IRC06:39
*** sdake has joined #openstack-infra06:40
*** unicell has joined #openstack-infra06:41
*** pcrews has quit IRC06:42
*** sdake has quit IRC06:43
openstackgerritTony Breeds proposed openstack-infra/irc-meetings: Reset Neutron UPgrades to Weekly  https://review.openstack.org/24958906:44
*** unicell has quit IRC06:45
openstackgerritOpenStack Proposal Bot proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/24959006:46
*** unicell has joined #openstack-infra06:47
*** mrmartin has joined #openstack-infra06:49
*** binbincong has quit IRC06:53
*** gildub has quit IRC06:56
*** amotoki has joined #openstack-infra06:57
openstackgerritYAMAMOTO Takashi proposed openstack-infra/project-config: tap-as-a-service: Add a missing export for ENABLED_SERVICES  https://review.openstack.org/24959906:58
*** Sukhdev_ has joined #openstack-infra06:59
*** nadya_ has joined #openstack-infra07:02
*** Sukhdev has quit IRC07:03
*** flepied has quit IRC07:05
*** Sukhdev_ has quit IRC07:06
*** mohankumar has quit IRC07:06
*** armax has quit IRC07:08
*** shardy has joined #openstack-infra07:10
*** FallenPegasus has joined #openstack-infra07:10
*** maishsk has joined #openstack-infra07:10
*** maishsk has quit IRC07:11
*** binbincong has joined #openstack-infra07:12
*** amitgandhinz has joined #openstack-infra07:21
*** bmwiedemann has joined #openstack-infra07:21
openstackgerritMerged openstack-infra/project-config: Remove python26 from keystone projects  https://review.openstack.org/24940007:23
bmwiedemannjeblair, SergeyLukjanov : would be nice if someone could have a look into why  http://tarballs.openstack.org/neutron/neutron-stable-kilo.tar.gz contains neutron-8.0.0.dev486 instead of 2015.1.3 that it still had yesterday. liberty one is the same.07:24
*** amitgandhinz has quit IRC07:25
*** Ramanjaneya has quit IRC07:26
*** Ramanjaneya has joined #openstack-infra07:26
*** scheuran has joined #openstack-infra07:26
*** sdake has joined #openstack-infra07:28
*** mrmartin has quit IRC07:29
*** ddieterly has joined #openstack-infra07:29
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Modifying docimpact group  https://review.openstack.org/24851507:30
*** doug-fish has joined #openstack-infra07:30
*** nmagnezi has joined #openstack-infra07:33
*** ddieterly has quit IRC07:34
*** dims has joined #openstack-infra07:35
*** doug-fish has quit IRC07:35
*** mohankumar has joined #openstack-infra07:35
*** Ramanjaneya has quit IRC07:36
*** fedexo has quit IRC07:36
*** sdake has quit IRC07:38
*** salv-orlando has joined #openstack-infra07:38
*** asselin has quit IRC07:39
*** dims_ has joined #openstack-infra07:39
*** asselin has joined #openstack-infra07:40
*** Ramanjaneya has joined #openstack-infra07:40
*** dims has quit IRC07:40
openstackgerritMerged openstack-infra/project-config: Add release notes jobs for senlin  https://review.openstack.org/24924407:41
*** salv-orlando has quit IRC07:41
*** flepied has joined #openstack-infra07:43
*** gildub has joined #openstack-infra07:45
*** rcernin_ has joined #openstack-infra07:47
*** HeOS has quit IRC07:50
*** AJaeger has joined #openstack-infra07:52
AJaegeryolando, could you +2A https://review.openstack.org/#/c/249058/ , please? Also, https://review.openstack.org/249403, https://review.openstack.org/249599 and https://review.openstack.org/#/c/249380/ would be nice - nothing urgent. Just whenever you review next time, please07:53
*** Ala has joined #openstack-infra07:54
*** mpavone has joined #openstack-infra07:54
*** mpavone has quit IRC07:55
*** mohankumar has quit IRC07:57
*** mahatic has joined #openstack-infra08:00
*** e0ne has joined #openstack-infra08:07
*** sdake has joined #openstack-infra08:09
*** shashank_hegde has quit IRC08:10
*** e0ne has quit IRC08:12
*** Ramanjaneya has quit IRC08:13
*** markus_z has joined #openstack-infra08:15
*** e0ne has joined #openstack-infra08:16
*** matrohon has joined #openstack-infra08:17
*** achanda has joined #openstack-infra08:19
*** dingyichen has quit IRC08:21
*** ildikov has quit IRC08:22
*** amitgandhinz has joined #openstack-infra08:22
*** thorst has joined #openstack-infra08:23
*** arxcruz has joined #openstack-infra08:23
*** eil397 has quit IRC08:24
*** vponomaryov has left #openstack-infra08:25
*** otter768 has joined #openstack-infra08:26
*** vponomaryov has joined #openstack-infra08:26
*** zhurong has quit IRC08:26
*** amitgandhinz has quit IRC08:27
*** thorst has quit IRC08:27
*** e0ne has quit IRC08:28
*** sshnaidm has joined #openstack-infra08:30
*** ddieterly has joined #openstack-infra08:30
*** otter768 has quit IRC08:30
*** sergmelikyan has joined #openstack-infra08:31
*** aopincar has quit IRC08:32
*** achanda has quit IRC08:33
openstackgerritMerged openstack-infra/system-config: Ensure system-config hieradata link exists  https://review.openstack.org/24944508:33
*** e0ne has joined #openstack-infra08:33
*** ddieterly has quit IRC08:35
openstackgerritMerged openstack-infra/project-config: Document check-requirements template  https://review.openstack.org/24937508:39
*** Ramanjaneya has joined #openstack-infra08:39
*** fhubik has joined #openstack-infra08:39
openstackgerritMerged openstack-infra/project-config: Add docs-on-rtfd hook for ansible-role-{nodepool|zuul}  https://review.openstack.org/24839608:40
*** jlanoux has joined #openstack-infra08:41
*** sdake has quit IRC08:41
*** samuelBartel has joined #openstack-infra08:41
*** HeOS has joined #openstack-infra08:41
openstackgerritMerged openstack-infra/project-config: Fix job deffinition for ceph job  https://review.openstack.org/24914808:42
openstackgerritMerged openstack-infra/project-config: Don't trigger gate-fuel-web-python26 job for fuel-web  https://review.openstack.org/24924208:42
*** aopincar has joined #openstack-infra08:44
*** hashar has joined #openstack-infra08:44
*** e0ne has quit IRC08:44
*** Daisy has joined #openstack-infra08:44
StevenKyolanda: That Zanata 3.8.1 change is really easy to check. After its approved and puppet has run on proposal.slave.openstack.org, check that zanata-cli is still a symlink and zanata-cli --version reports 3.8.108:46
StevenKyolanda: Client updates are trivial, it's the server upgrades that make me twitch08:46
yolandaStevenK, good to know thanks. I'll approve them08:47
yolandai remembered that zanata upgrades needed some special operation, but well, it may be on server :)08:47
StevenKRight, Zanata server upgrades need special handling and babysitting. The client, not so much.08:47
yolandaok approving08:49
*** hashar has quit IRC08:49
*** aopincar has quit IRC08:50
*** jordanP has joined #openstack-infra08:51
*** ihrachys has joined #openstack-infra08:52
openstackgerritMerged openstack-infra/puppet-zanata: Update Zanata client to 3.8.1  https://review.openstack.org/24853208:53
*** sergmelikyan has quit IRC08:53
yolandahere is goes ^08:54
openstackgerritGal Sagie proposed openstack-infra/project-config: Fix Kuryr project repository  https://review.openstack.org/24964108:54
*** sridhar_ram has joined #openstack-infra08:54
*** sridhar_ram has quit IRC08:59
openstackgerritGal Sagie proposed openstack-infra/system-config: Add #openstack-dragonflow to channel logging  https://review.openstack.org/24964309:00
*** mohankumar has joined #openstack-infra09:01
*** hashar has joined #openstack-infra09:04
yolandaStevenK, so puppet was applied on proposal slave, but zanata is still on 3.7.3..· i didn't see puppet updating any client09:09
yolandaonly saw pyopenssl upgraded to latest09:10
StevenKMaybe the module didn't get updated in time for the run?09:10
yolandayep, let's wait a bit more09:11
*** mkoderer_ is now known as mkoderer09:11
*** achanda has joined #openstack-infra09:11
*** shardy has quit IRC09:12
*** dtantsur|afk is now known as dtantsur09:14
*** nadya_ has quit IRC09:14
*** rossella_s has quit IRC09:14
*** rossella_s has joined #openstack-infra09:14
*** shardy has joined #openstack-infra09:14
*** mrmartin has joined #openstack-infra09:15
*** Liuqing_ has quit IRC09:15
*** Liuqing has joined #openstack-infra09:15
*** kushal has quit IRC09:17
*** ildikov has joined #openstack-infra09:18
*** dizquierdo has joined #openstack-infra09:20
*** lezbar has quit IRC09:21
*** mohankumar has quit IRC09:22
*** skraynev has quit IRC09:23
*** amitgandhinz has joined #openstack-infra09:24
*** aspiers_ has joined #openstack-infra09:25
*** dguitarbite has quit IRC09:26
*** isaacb has joined #openstack-infra09:27
*** salv-orlando has joined #openstack-infra09:27
*** salv-orlando has quit IRC09:28
*** amitgandhinz has quit IRC09:28
*** lezbar has joined #openstack-infra09:28
*** salv-orlando has joined #openstack-infra09:29
*** FallenPe_ has joined #openstack-infra09:29
*** dguitarbite has joined #openstack-infra09:29
*** ddieterly has joined #openstack-infra09:31
*** FallenPegasus has quit IRC09:31
yolandaStevenK, done09:32
openstackgerritAnastasia Kuznetsova proposed openstack-infra/project-config: Disable mistral-python34 in all branches except mitaka ang higher  https://review.openstack.org/24966609:34
*** dims_ has quit IRC09:34
*** jistr has joined #openstack-infra09:35
*** ddieterly has quit IRC09:36
*** electrofelix has quit IRC09:37
StevenKyolanda: \o/ Thanks!09:37
*** bexelbie_ has joined #openstack-infra09:38
openstackgerritRenat Akhmerov proposed openstack-infra/project-config: Disable mistral-python34 in all branches except mitaka and higher  https://review.openstack.org/24966609:39
*** fhubik is now known as fhubik_brb09:40
*** fhubik_brb is now known as fhubik09:41
*** Liuqing has quit IRC09:41
*** achanda has quit IRC09:41
openstackgerritSébastien Han proposed openstack-infra/project-config: Add Ceph DevStack plugin to Stackforge  https://review.openstack.org/18876809:42
*** dansmith has quit IRC09:42
openstackgerritMerged openstack-infra/irc-meetings: Remove the Cloudpulse meeting  https://review.openstack.org/24956209:44
*** aopincar has joined #openstack-infra09:45
*** dansmith has joined #openstack-infra09:45
*** dansmith is now known as Guest1115309:45
openstackgerritQiming Teng proposed openstack/requirements: Drop python-openstacksdk from requirements  https://review.openstack.org/24967809:45
*** sshnaidm has quit IRC09:47
*** amotoki has quit IRC09:48
openstackgerritQiming Teng proposed openstack/requirements: Add '.venv' to .gitignore  https://review.openstack.org/24968009:48
*** mohankumar has joined #openstack-infra09:49
*** salv-orlando has quit IRC09:50
*** fhubik is now known as fhubik_brb09:51
*** Daisy has quit IRC09:51
*** Daisy has joined #openstack-infra09:51
*** dizquierdo has quit IRC09:52
openstackgerritMerged openstack-infra/irc-meetings: Remove the Satori meeting  https://review.openstack.org/24957109:53
openstackgerritMerged openstack-infra/irc-meetings: Remove the OpenStack Community Infrastructure meeting  https://review.openstack.org/24956709:53
*** swat30 has quit IRC09:53
*** bapalm has quit IRC09:53
*** bapalm has joined #openstack-infra09:55
*** Sree_ has joined #openstack-infra09:56
*** swat30 has joined #openstack-infra09:56
*** Sree_ is now known as Guest3213509:56
*** Daisy has quit IRC09:57
*** salv-orlando has joined #openstack-infra09:57
*** bexelbie__ has joined #openstack-infra09:58
*** bexelbie__ is now known as bexelbie09:58
*** nadya_ has joined #openstack-infra09:58
*** bexelbie_ has quit IRC09:58
openstackgerritMerged openstack-infra/irc-meetings: Remove the Libvirt meeting  https://review.openstack.org/24956409:59
*** Sree has quit IRC09:59
openstackgerritSébastien Han proposed openstack-infra/project-config: Add Ceph DevStack plugin to Stackforge  https://review.openstack.org/18876810:00
*** rguillebert has joined #openstack-infra10:00
*** e0ne has joined #openstack-infra10:00
*** zhenguo has quit IRC10:01
*** leseb has joined #openstack-infra10:02
lesebAJaeger: thanks for the review10:03
AJaegerleseb: welcome10:03
*** sshnaidm has joined #openstack-infra10:03
lesebsdague: when you have time :) https://review.openstack.org/#/c/188768/10:04
*** Liuqing has joined #openstack-infra10:05
*** baoli has quit IRC10:06
*** rcernin_ has quit IRC10:06
AJaegerleseb: sdague is on vacation this week AFAIK10:07
lesebAJaeger: do you know if someone else could +2 this patch?10:07
AJaegerleseb: let's ask yolanda to review 188768, she's the only one around at the moment. Otherwise come back later.10:09
yolandasure10:09
AJaegeryolanda: did you see my long list of reviews from earlier in the backscroll?10:09
AJaegeryolanda: thanks!10:09
yolandaAJaeger, you are a review machine :)10:10
lesebAJaeger: ok thanks :)10:10
AJaegeryolanda: sometimes ;)10:12
yolandaok approved that one10:13
*** fhubik_brb is now known as fhubik10:13
AJaegeryolanda: you're getting there as well! ;)10:13
yolandayep, learning slowly10:13
AJaegerleseb: once your change is in, and the project exists in gerrit, please come back and ask to get the ACL updated to include you10:14
ajottx ping about: https://review.openstack.org/#/c/249565/  , can we abandon it?, I will make sure we remove the meeting ourselves to free up resources if we stop meeting10:14
lesebAJaeger: will do thanks :)10:15
*** yamamoto has quit IRC10:15
ttxajo: ack, I -2ed it, will let tonyb abandon it10:16
*** kozhukalov_ has joined #openstack-infra10:16
ajottx : thanks, and thanks tonyb for the cleanups10:16
*** glauco_ has joined #openstack-infra10:16
openstackgerritMerged openstack-infra/project-config: Add Ceph DevStack plugin to Stackforge  https://review.openstack.org/18876810:18
*** Liuqing has quit IRC10:22
openstackgerritIsaac Beckman proposed openstack-infra/devstack-gate: create GRE bridge interfaces only once  https://review.openstack.org/23936310:23
*** Liuqing has joined #openstack-infra10:23
*** amitgandhinz has joined #openstack-infra10:25
*** otter768 has joined #openstack-infra10:27
*** dtantsur is now known as dtantsur|brb10:29
*** Liuqing has quit IRC10:29
*** amitgandhinz has quit IRC10:29
*** Liuqing has joined #openstack-infra10:30
*** otter768 has quit IRC10:31
*** ddieterly has joined #openstack-infra10:33
*** ddieterly has quit IRC10:38
rcarrillocruzcrinkle: when you are around, give us a nudge10:41
StevenKyolanda: Do you also want to review https://review.openstack.org/#/c/249058/ to make AJaeger happy?10:41
yolandasure10:42
* AJaeger is happy now ;) Thanks!10:42
StevenKHaha10:43
yolandaglad to make you happy :)10:43
*** yamahata has quit IRC10:43
StevenKAJaeger: Once that's landed, how did you want to trigger another job?10:43
openstackgerritMerged openstack-infra/project-config: Fix two errors in setup_training_guides  https://review.openstack.org/24905810:46
*** FallenPe_ has quit IRC10:47
AJaegerwe have to ask the training-labs team, let me do that...10:48
*** ldnunes has joined #openstack-infra10:48
*** aukhan has quit IRC10:51
AJaegerargh, training-guides, not training-labs...10:51
*** bexelbie has quit IRC10:51
StevenKWell, and puppet has updated the proposal slave again10:52
StevenKAnd puppetboard doesn't seem to love me10:52
lesebAJaeger: the patch got merged, can you please add me to the acl?10:52
*** yamamoto has joined #openstack-infra10:53
AJaegerleseb: I cannot but any infra-root can10:55
AJaegerStevenK: https://review.openstack.org/#/c/249702/ is for training-guide, there's no other obvious patch in the queue10:55
lesebAJaeger: who can help me on that?10:56
yolandaleseb, what do you need?10:56
lesebyolanda: hey, looks like I need to be included into the ACL10:56
*** yamamoto has quit IRC10:56
*** yamamoto has joined #openstack-infra10:57
yolandasure, what's the group and username you need?10:57
*** rfolco_ has joined #openstack-infra10:58
lesebyolanda: is devstack-ceph-core as a group ok? and username: sebastien-han10:59
yolandaok10:59
lesebyolanda: is it normal that the repo doesn't exist yet on github?11:00
yolandamm, no group as well11:00
*** glauco_ has quit IRC11:00
yolandaleseb, when did the change land? we may need to give some time11:00
lesebyolanda: like 40min ago11:00
yolandaleseb, let's wait a bit more, it needs to land on gerrit and run a process to create project and group11:01
*** k4n0 has joined #openstack-infra11:01
openstackgerritThierry Carrez proposed openstack-infra/irc-meetings: Move cross-project meeting to -cp channel  https://review.openstack.org/24970811:01
lesebyolanda: ok, I'll check that, and will ping you when it's on github?11:01
yolandaleseb, yes, or if it hasn't happened in next 30 minutes, let me know11:02
lesebyolanda: sure thanks, lunch time for me :)11:02
*** yamamoto_ has joined #openstack-infra11:11
*** Guest11153 has quit IRC11:12
*** Guest32135 has quit IRC11:12
notmorganclarkb, zaro, fungi: have a hack to fix the double leading slash it looks like.11:13
openstackgerritMerged openstack-infra/storyboard-webclient: Don't add unexpected `story_id` attribute to Comment  https://review.openstack.org/24921911:13
notmorganclarkb, fungi, zaro: I'll propose the change up so you can take a look and verify11:13
*** yamamoto has quit IRC11:15
*** dansmith has joined #openstack-infra11:15
*** dansmith is now known as Guest1050411:15
*** kushal has joined #openstack-infra11:16
*** aysyd has joined #openstack-infra11:18
*** gongysh__ has quit IRC11:19
*** yamamoto_ has quit IRC11:20
*** aysyd has quit IRC11:22
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: 301 Redirect stripping out double leading slash  https://review.openstack.org/24971411:23
*** aysyd has joined #openstack-infra11:23
notmorganfungi, clarkb, zaro, mordred: ^ that should address double leading slash.11:24
*** yamamoto has joined #openstack-infra11:25
*** aspiers_ has quit IRC11:25
*** amitgandhinz has joined #openstack-infra11:25
* notmorgan has spent too much time looking at the internals of mod_rewrite %{<var>}s this morning11:26
*** fhubik is now known as fhubik_brb11:26
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: 301 Redirect stripping out double leading slash  https://review.openstack.org/24971411:27
*** NithyaG has joined #openstack-infra11:28
*** dtantsur|brb is now known as dtantsur11:28
*** Guest10504 has quit IRC11:29
*** samuelBartel has quit IRC11:29
*** glauco_ has joined #openstack-infra11:30
*** amitgandhinz has quit IRC11:31
*** wuhg has quit IRC11:31
*** dansmith_ has joined #openstack-infra11:31
*** yamamoto has quit IRC11:32
*** dizquierdo has joined #openstack-infra11:33
*** ddieterly has joined #openstack-infra11:34
notmorganlooks like http://status.openstack.org/zuul/ might be down.11:35
notmorganwhenever someone is up11:35
*** hashar has quit IRC11:35
openstackgerritIsaac Beckman proposed openstack-infra/devstack-gate: create GRE bridge interfaces only once  https://review.openstack.org/23936311:36
*** kushal has quit IRC11:36
yolandahi notmorgan, i can check11:37
notmorganyolanda: awesome, thnx :)11:37
yolandaif you say infra-root, you summon me :)11:38
notmorganyolanda: hehe, good to know.. i shall make sure not to abuse said knowledge ;)11:38
yolandamm, i cannot even ssh to the instance11:39
*** ddieterly has quit IRC11:39
notmorganyah my external checks showed it was hard down.11:39
notmorganor at least totally non-responsive11:39
yolandai'll check in the control panel11:39
notmorgancool11:39
openstackgerritMerged openstack-infra/project-config: Disable mistral-python34 in all branches except mitaka and higher  https://review.openstack.org/24966611:40
*** jamielennox is now known as jamielennox|away11:42
*** Daisy has joined #openstack-infra11:42
odyssey4mehmm, logstash.openstack.org is just giving me a spinning wheel and no data11:43
yolandanotmorgan, status is not good, i'll restart11:44
*** Daisy has quit IRC11:45
yolandaok, status up11:45
notmorganyolanda: yay!11:46
notmorganyolanda: have a fantastic day!11:47
*** marcusvrn_ has joined #openstack-infra11:47
yolandanotmorgan, thanks!11:47
yolandaodyssey4me, logstash doesn't look good, i see11:48
*** kozhukalov_ has quit IRC11:48
openstackgerritSachi King proposed openstack-infra/project-config: Fix '{name}-docs-constraints' jobs  https://review.openstack.org/24972211:48
openstackgerritMerged openstack-infra/irc-meetings: Add check_channels.py and testenv to run it  https://review.openstack.org/24724911:49
yolandaodysse4me, i see it times out on search, but not sure about the solution11:51
yolandalet's comment that with mordred , fungi, clarkb... they should connect soon11:51
*** derekh has joined #openstack-infra11:52
*** dansmith_ has quit IRC11:52
notmorganyah it's like 7am east coast so at least mordred and fungi should appear soon(er) vs later.11:52
*** dansmith_ has joined #openstack-infra11:52
*** glauco__ has joined #openstack-infra11:52
*** glauco_ has quit IRC11:52
*** bexelbie_ has joined #openstack-infra11:54
*** amotoki has joined #openstack-infra11:54
*** gongysh_ has joined #openstack-infra11:54
*** zeih has joined #openstack-infra11:57
*** dramalho has joined #openstack-infra11:57
*** gongysh_ has quit IRC11:57
AJaegerjhesketh, yolanda: https://review.openstack.org/#/c/249722/ fixes a broken post job, would be nice to get this in today once you have time...11:57
openstackgerritMerged openstack-infra/project-config: Fix Kuryr project repository  https://review.openstack.org/24964111:58
jheskethAJaeger: looking11:59
AJaegerthanks, jhesketh11:59
openstackgerritMerged openstack-infra/project-config: Modifying docimpact group  https://review.openstack.org/24851511:59
jheskethAJaeger: lgtm12:00
*** jamielennox|away is now known as jamielennox12:01
*** mrmartin has quit IRC12:01
AJaegergreat12:01
*** dramalho has quit IRC12:01
*** ddieterly has joined #openstack-infra12:02
*** FallenPegasus has joined #openstack-infra12:02
*** yamamoto has joined #openstack-infra12:02
AJaegerjhesketh: https://review.openstack.org/249403, https://review.openstack.org/249599 and https://review.openstack.org/#/c/249380/ would also be nice to fix problems with existing setup. Whenever you review next time, please...12:03
jheskethAJaeger: will look now12:03
jheskethAJaeger: do you know much about the meetbot puppet module?12:03
AJaegerjhesketh: nothing at all ;(12:04
jheskethokay, no worries12:04
jheskethpleia2: don't suppose you're around per chance?12:04
*** dansmith_ is now known as dansmith12:04
AJaegerjhesketh: I expect it's too early for her.12:05
jheskethoh yeah, it was just a try ;-)12:05
*** dansmith is now known as Guest8784312:05
*** achanda has joined #openstack-infra12:05
AJaegerjhesketh: I know ;) Thanks for the reviews12:05
rakhmerovAJaeger: https://review.openstack.org/249666 got merged in but the gate is still voting. How soon will it take effect?12:05
openstackgerritMerged openstack-infra/project-config: Fix '{name}-docs-constraints' jobs  https://review.openstack.org/24972212:05
*** jaypipes has joined #openstack-infra12:05
*** FallenPegasus has quit IRC12:06
AJaegerrakhmerov: you have to wait for our next puppet run. It should take effect in the next hour.12:07
rakhmerovok, thanks12:07
openstackgerritMerged openstack-infra/project-config: Typo in pypi-extract-universal.py  https://review.openstack.org/24940312:08
openstackgerritMerged openstack-infra/project-config: tap-as-a-service: Add a missing export for ENABLED_SERVICES  https://review.openstack.org/24959912:09
AJaegerrakhmerov: just to clarify: The gate queue will always be voting. The change is that the job will not run at all on older releases12:09
*** gongysh_ has joined #openstack-infra12:10
*** ddieterl_ has joined #openstack-infra12:10
*** placko has joined #openstack-infra12:12
*** ddieterly has quit IRC12:13
openstackgerritMerged openstack-infra/project-config: Add check-requirements for cliff  https://review.openstack.org/24938012:14
*** zeih has quit IRC12:18
*** fhubik_brb is now known as fhubik12:19
*** trown|outttypeww is now known as trown12:20
openstackgerritMerged openstack-infra/system-config: Customize the 'Report Bug' link on Gerrit UI  https://review.openstack.org/22141712:20
*** achanda has quit IRC12:24
*** thorst has joined #openstack-infra12:24
*** gongysh_ has quit IRC12:25
*** gongysh_ has joined #openstack-infra12:25
*** amitgandhinz has joined #openstack-infra12:26
*** otter768 has joined #openstack-infra12:28
*** electrofelix has joined #openstack-infra12:28
*** zeih has joined #openstack-infra12:28
*** thorst has quit IRC12:29
AJaegerStevenK: https://jenkins.openstack.org/job/training-guides-upstream-translation-update/3/console12:30
AJaegerStevenK: I fear we need to wait until the proposal slave gets updated. ;(12:31
*** gongysh_ has quit IRC12:31
*** amitgandhinz has quit IRC12:31
*** otter768 has quit IRC12:33
*** achanda has joined #openstack-infra12:33
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Clean up compute functional tests  https://review.openstack.org/24944412:34
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Fix resource leak in test_compute  https://review.openstack.org/24945212:34
*** aspiers has joined #openstack-infra12:34
StevenKAJaeger: Yes. I'm not sure when /usr/local/jenkins/slave_scripts is updated by puppet. :-/12:35
*** weshay_xchat has joined #openstack-infra12:35
openstackgerritDougal Matthews proposed openstack/requirements: Add ipaddress to global-requirements.txt  https://review.openstack.org/24973912:36
AJaegerStevenK: it's on the proposal slave, I think that's handled differently and needs a new node to be built first. So, we should try again tomorrow ;)12:36
openstackgerritDougal Matthews proposed openstack/requirements: Add ipaddress to global-requirements.txt  https://review.openstack.org/23269412:37
*** Sree has joined #openstack-infra12:40
*** gordc has joined #openstack-infra12:41
lesebyolanda: hey, looks like https://github.com/openstack/devstack-plugin-ceph is not there yet :/12:41
*** achanda has quit IRC12:41
openstackgerritMerged openstack-infra/project-config: Skip unsupported tests for BlockDeviceDriver  https://review.openstack.org/24802112:41
yolandaleseb, so that's not good, i'll try run manage projects manually12:41
*** dims has joined #openstack-infra12:42
lesebyolanda: thanks, please keep me posted :)12:42
*** ociuhandu has quit IRC12:43
*** ldnunes has quit IRC12:45
*** ldnunes has joined #openstack-infra12:46
*** weshay_xchat is now known as weshahy12:46
*** weshahy is now known as weshay12:46
*** e0ne_ has joined #openstack-infra12:48
*** claudiub has joined #openstack-infra12:49
*** k4n0 has quit IRC12:49
*** boris-42 has joined #openstack-infra12:49
*** maurosr has joined #openstack-infra12:50
*** jaypipes_ has joined #openstack-infra12:51
*** e0ne has quit IRC12:51
*** jaypipes has quit IRC12:51
openstackgerritMerged openstack/requirements: Remove networking-bgpvpn from projects.txt  https://review.openstack.org/24931212:52
*** jaypipes_ is now known as jaypipes12:53
*** zeih has quit IRC12:54
*** gildub has quit IRC12:54
*** zhurong has joined #openstack-infra12:55
*** korzen has joined #openstack-infra12:59
*** flepied has quit IRC13:00
mordredAJaeger, StevenK: proposal slave is managed like a long-lived server- slave scripts should be updated on puppet run13:01
*** dprince has joined #openstack-infra13:01
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Return stderr/stdout on changes too  https://review.openstack.org/24976513:02
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: The parameter is --server not -- server  https://review.openstack.org/24976613:02
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Don't use the timeout command if it's not there  https://review.openstack.org/24976713:02
AJaegermordred: mmh, but it wasn't ;(13:02
mordredhrm13:02
AJaegerthanks mordred .13:02
AJaegerlet me dig a bit...13:02
openstackgerritMerged openstack-infra/project-config: networking-bgpvpn: remove check-requirements jobs  https://review.openstack.org/24932213:02
*** kozhukalov_ has joined #openstack-infra13:03
*** olaph has quit IRC13:03
*** chlong has joined #openstack-infra13:03
*** asselin has quit IRC13:04
AJaegermordred, StevenK https://review.openstack.org/#/c/249058/ merged at 10:46 UTC but http://puppetboard.openstack.org/report/proposal.slave.openstack.org/454aa78822232d1898a60e532364a224b157adba shows that latest run was at 9:3213:04
*** olaph has joined #openstack-infra13:04
*** asselin has joined #openstack-infra13:04
*** jaypipes has quit IRC13:04
*** hashar has joined #openstack-infra13:04
openstackgerritRico Lin proposed openstack-infra/project-config: add release notes jobs for Heat  https://review.openstack.org/24976913:05
*** yamamoto has quit IRC13:05
*** aopincar has quit IRC13:05
AJaegermordred: any idea why puppet was not run for 249058?13:06
mordredAJaeger: no - I just tried running it again and it worked fine13:09
mordredAJaeger: so you should be good to do now13:09
*** david-lyle has quit IRC13:09
AJaegermordred: indeed, the files are transferred. thanks13:09
AJaegerStevenK: will try it again ;)13:09
StevenKAJaeger: I am heading to bed, but please hilight me and let me know how it goes.13:10
*** gongysh_ has joined #openstack-infra13:11
*** salv-orlando has quit IRC13:12
*** Ramanjaneya has quit IRC13:13
*** rossella_s has quit IRC13:14
*** hashar has quit IRC13:14
AJaegerStevenK: good night - I'll tell you...13:14
*** rossella_s has joined #openstack-infra13:14
*** flepied has joined #openstack-infra13:15
yolandamordred, StevenK, what happened with proposal slave? puppet didn't update? i saw zanata-cli version updated13:16
AJaegeryolanda: yes, zanata-cli was updated but then nothing happened ;(13:17
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Fetch puppet logs and facts after puppet run  https://review.openstack.org/24044613:17
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: The parameter is --server not -- server  https://review.openstack.org/24976613:17
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Don't use the timeout command if it's not there  https://review.openstack.org/24976713:17
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Return stderr/stdout on changes too  https://review.openstack.org/24976513:17
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Test for copy_hieradata being set  https://review.openstack.org/24977513:17
*** skraynev has joined #openstack-infra13:17
AJaegeryolanda, looking at http://puppetboard.openstack.org/ it seems that nothing was wrong for ages - which looks wrong.13:18
openstackgerritZara proposed openstack-infra/storyboard-webclient: Fix subscription display for project groups  https://review.openstack.org/24977613:18
*** yamamoto has joined #openstack-infra13:19
*** hashar has joined #openstack-infra13:19
AJaegerhttp://puppetboard.openstack.org/reports/proposal.slave.openstack.org - interesting, three runs in 10 mins and nothing before ;(13:20
* AJaeger knows too little about puppet13:20
mordredAJaeger: well13:20
mordredso there was a bug I introduced last night13:20
yolandamaybe there is some block?13:20
*** bltavares has left #openstack-infra13:20
mordredthe first run is me testing running by hand13:21
mordredthe second 2 are me testing the fix to teh block13:21
AJaegerthe top page looks wrong, the /nodes one shows activity13:21
yolandai also tested by hand13:21
mordred:)13:21
openstackgerritMerged openstack-infra/ansible-puppet: Test for copy_hieradata being set  https://review.openstack.org/24977513:21
yolandabut looked fine, and updated the cli actually13:21
AJaegermordred: ah...13:21
mordredyolanda: ^^ that fixes the thing that was broken13:21
AJaegermordred: thanks for the explanation!13:21
mordredyolanda: btw ... I have great news - I got puppet apply working all the way yesterday13:22
AJaegerand tahnks for fixing!13:22
mordredonly thing left to do is land the patches and we're off puppetmaster13:22
yolandamordred, that's great!13:22
mordredyolanda: https://review.openstack.org/#/q/status:open++topic:puppet-apply,n,z if you get bored13:23
yolandai wish i could get bored at some point :)13:23
AJaegercool, mordred !13:23
mordredyolanda: when we land all of those, we'll be good to go13:23
mordredyolanda: haha13:23
*** salv-orlando has joined #openstack-infra13:23
*** salv-orlando has quit IRC13:24
yolandamodred, i'll try get some time after lunch, getting rid of master/agent is great13:24
*** salv-orlando has joined #openstack-infra13:25
yolandabtw, mordred, i'm running manage-projects manually on gerrit. leseb launched a change more than 2 hours ago and it still doesnt' show13:25
yolandait may be related with puppet not applying changes ?13:25
AJaegermordred: so, we should now see a lot of puppet activity, correct?13:25
*** kzaitsev_mb has joined #openstack-infra13:25
mordredAJaeger: yes13:25
mordredand yes - puppet wasn't running because I missed something in testing one of the patches (I missed the intermediate state - we made a change to let us set the config for copying hieradata files - but I somehow missed testing that it worked without the setting at all. blast)13:26
mordredit should be all better now13:26
AJaegergreat13:26
yolandaleseb, so that should fix your issue13:27
*** jyuso1 has quit IRC13:27
*** amitgandhinz has joined #openstack-infra13:27
*** tiswanso has quit IRC13:27
*** tiswanso has joined #openstack-infra13:27
openstackgerritDerek Higgins proposed openstack-infra/project-config: Switch TripleO jobs to F22 jenkins nodes  https://review.openstack.org/24368413:28
*** ayoung has joined #openstack-infra13:28
yolandamordred, i trade reviews for several on my side , i have a pair that only miss a +213:29
*** ildikov has quit IRC13:29
kzaitsev_mbAJaeger: anteaya: hi! I've talked to murano team and PTL. We decided, that we should push to make requirements job voting for murano-agent. we currently have one non-global requirement, that should be more or less trivial to get rid of. I plan to remove it this week, and after that — make a commit that would make requirement job voting =)13:29
kzaitsev_mbAJaeger: anteaya: does the plan sound good? =)13:30
*** jordanP has quit IRC13:30
openstackgerritDerek Higgins proposed openstack-infra/project-config: Switch TripleO jobs to F22 jenkins nodes  https://review.openstack.org/24368413:30
AJaegerkzaitsev_mb: perfect! Thanks!13:30
*** ociuhandu has joined #openstack-infra13:30
kzaitsev_mbok, cool. will keep you updated =)13:31
AJaegerkzaitsev_mb: just send a change and we happily review ;)13:31
asselincan we get some reviews here? this is urgently needed by folks trying to setup 3rd party ci: https://review.openstack.org/#/c/243623/13:31
asselinotherwise puppet fails for duplicate definitions errors b/c zuul and jenkins are co-installed13:32
mordredyolanda: :) which ones do you nede?13:32
*** amitgandhinz has quit IRC13:32
AJaegermordred: the change asselin was written by yolanda, so she can't review ;)13:32
lesebyolanda: still not there: https://github.com/openstack/devstack-plugin-ceph13:33
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Don't post facts if there are none  https://review.openstack.org/24978113:33
asselinAJaeger, mordred I wrote the same patch to solve the issue. Didn't realize yolanda already had that patch written although I had previously +1 it.13:34
yolandamordred https://review.openstack.org/232991, https://review.openstack.org/24349313:34
mordredasselin: :)13:34
yolandajust miss a +213:34
*** gongysh_ has quit IRC13:35
yolandaand if you look at downstream-puppet topic, there are several pending on reviews13:35
*** ildikov has joined #openstack-infra13:35
yolandaasselin, that shows that change is really needed :)13:35
asselinmordred, AJaeger while you're reviewing, these 2 are also needed for correctly functionality, although at least puppet runs: https://review.openstack.org/#/c/248223/ https://review.openstack.org/#/c/248223/13:35
AJaegerasselin: I'm only reviewing project-config...13:36
AJaegermordred: if you want to have two of my favorites as well for review, please tell me ;)13:38
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Don't post facts if there are none  https://review.openstack.org/24978113:40
*** jffische has joined #openstack-infra13:40
openstackgerritMerged openstack-infra/puppet-graphite: Add support for apache >= 2.4  https://review.openstack.org/23299113:40
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Introduce space in a very important place  https://review.openstack.org/24978913:41
*** zhurong has quit IRC13:41
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Fetch puppet logs and facts after puppet run  https://review.openstack.org/24044613:41
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Introduce space in a very important place  https://review.openstack.org/24978913:41
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: The parameter is --server not -- server  https://review.openstack.org/24976613:41
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Don't use the timeout command if it's not there  https://review.openstack.org/24976713:41
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Don't post facts if there are none  https://review.openstack.org/24978113:41
openstackgerritMonty Taylor proposed openstack-infra/ansible-puppet: Return stderr/stdout on changes too  https://review.openstack.org/24976513:41
hasharoh man13:41
hasharspaaaaam :-}13:41
* mordred hides13:41
mordredyolanda: I did it again - would you please +A this: https://review.openstack.org/24978913:41
notmorganmordred: hey no hiding!13:41
yolandasure13:42
lesebyolanda: any idea why it's still not working?13:42
yolandai love your commit messages13:42
openstackgerritMerged openstack-infra/zuul: Fix newrev comparison for NullChange  https://review.openstack.org/24349313:42
mordredleseb: I may have broken it again - fix landing now13:42
*** mohankumar has quit IRC13:43
mordredyolanda: the amuing thing is - http://puppetdb.openstack.org/fact/ansible_interfaces ... with https://review.openstack.org/249781 we now will also post ansible gathered facts to puppetdb, not just facter gathered facts :13:44
lesebmordred: ok I see thanks13:44
*** skraynev has quit IRC13:44
*** gongysh_ has joined #openstack-infra13:44
*** baoli has joined #openstack-infra13:45
openstackgerritMerged openstack-infra/ansible-puppet: Introduce space in a very important place  https://review.openstack.org/24978913:47
yolandamordred, i sent a question for that code13:47
*** ildikov has quit IRC13:47
mordredyolanda: responded13:49
*** olaph has quit IRC13:49
*** FallenPegasus has joined #openstack-infra13:50
*** olaph has joined #openstack-infra13:50
yolandamordred,i see, could not be on same loop with an extra check?13:51
*** alivigni has joined #openstack-infra13:51
yolandachecking k[7_13:51
yolandaups13:51
mordredyolanda: no - say you had facter_foo and foo as keys13:51
yolandak[7:].startwith ?13:51
mordredyou want facts['foo'] to equal the value in facter_foo not foo13:52
yolandaah, you overwrite13:52
mordredyah13:52
mordredI mean, I actually cannot think of a case where there will be actual overlap13:52
yolandaok, also we shouldn't have lots of facts, to consider that double iteration a problem right'13:52
mordredyah13:52
mordredit's not a REAL problem13:53
mordredeither way13:53
mordredit's a small loop, so double iteration is not slow - but also I believe there will never be an overlap :)13:53
yolandaok +213:53
*** tiswanso has quit IRC13:54
mordredyay!13:54
openstackgerritMerged openstack-infra/irc-meetings: Remove the Neutron VPNaaS meeting  https://review.openstack.org/24957213:54
*** claudiub_ has joined #openstack-infra13:55
*** FallenPegasus has quit IRC13:55
funginotmorgan: thanks for working on that! reviewing now13:55
gordchi, hoping someone with powers can help me add lianhao lu to the ceilometer-stable-maint gerrit group, or give me the power to add :)13:56
*** mrmartin has joined #openstack-infra13:57
notmorganfungi: :) happy to. that was scary horrible things to dig into the depths of mod_rewrite13:57
notmorganfungi: i may have woken up at 4am and realized what was wrong.13:57
notmorgan:P13:57
*** jaypipes has joined #openstack-infra13:57
*** xyang1 has joined #openstack-infra13:57
*** dkranz has joined #openstack-infra13:57
*** ildikov has joined #openstack-infra13:57
*** abregman has quit IRC13:58
*** abregman has joined #openstack-infra13:58
*** flepied has quit IRC13:58
*** zhenguo has joined #openstack-infra13:59
fungiheh14:01
mordredfungi: when you get done with that ...14:01
*** amuller has joined #openstack-infra14:01
mordredfungi: puppet apply is ready14:01
mordredfungi: https://review.openstack.org/#/q/status:open++topic:puppet-apply,n,z14:01
*** skraynev has joined #openstack-infra14:01
*** dtardivel has joined #openstack-infra14:01
mordredfungi: I've procedural -2'd the only patch that isn't safe to land14:02
*** jffische has quit IRC14:02
mordredfungi: well, and it is SAFE - but it's the zomg-go-live patch14:02
mordredso we might want to be paying attention to it14:02
notmorganmordred: so what you're saying is "roll it out and go on vacation" *shiftyeyes*14:03
mordredYES14:03
*** [1]Thelo has joined #openstack-infra14:04
*** Daisy has joined #openstack-infra14:05
*** zz_dimtruck is now known as dimtruck14:05
*** rlandy has joined #openstack-infra14:05
fungii don't always roll out major changes, but when i do i make sure i'm leaving on vacation immediately14:06
*** openstackgerrit has quit IRC14:06
notmorganfungi: ++14:06
*** openstackgerrit has joined #openstack-infra14:07
*** Thelo has quit IRC14:07
*** [1]Thelo is now known as Thelo14:07
AJaegerfungi, what about changing docimpact default - see also Lana's emails. Here's my change that needs another +2A: https://review.openstack.org/#/c/248549/ . Could you review, please? It's a simple major change ;)14:09
*** Daisy has quit IRC14:09
*** bexelbie_ is now known as bexelbie14:10
AJaegeryolanda, leseb http://git.openstack.org/cgit/openstack/devstack-plugin-ceph/ has been created but is not finished yet14:11
*** flepied has joined #openstack-infra14:11
lesebAJaeger: thanks for the update14:11
*** bdemers has joined #openstack-infra14:12
notmorganfungi: i think it always should be HTTPS, but with the added redirect from HTTP it seemed to work14:12
*** peristeri has joined #openstack-infra14:12
notmorganfungi: since that was in the mod_ssl block. confirming it still works with https specifically called out in that rule14:12
AJaegeryolanda: ACLs are up if you want to add leseb: https://review.openstack.org/#/admin/groups/1196,members and https://review.openstack.org/#/admin/groups/1197,members14:12
AJaegerleseb, yolanda: setup looks finished14:13
yolandagreat14:13
yolandawill do14:13
*** tmorin has joined #openstack-infra14:13
funginotmorgan: i suppose the request protocol variable could be substituted there instead, but as long as it works i'm not too worried14:13
*** olaph has quit IRC14:13
fungiAJaeger: where's the corresponding project-config change for the docimpact work?14:14
*** olaph has joined #openstack-infra14:14
notmorganfungi: but that seems superfluous since we are explicitly in an HTTPS vhost14:14
lesebAJaeger: awesome thanks :)14:14
tmorinhi fungi AJaeger: how long before disabling the check-requirements takes effects ?  (https://review.openstack.org/249322 was merged one hour ago)14:14
AJaegerfungi: https://review.openstack.org/#/c/248515/ - or is more needed?14:14
notmorganugh... i broke my java install *grump*14:14
funginotmorgan: well, it's a puppet module which may or may not be deployed using https i think (then again, i guess it's always https because it sets up snakeoil certs if you don't provide your own)14:14
AJaegertmorin: we had some problems updating our servers, should be done in an hour I hope14:14
tmorinAJaeger: ok, thanks14:15
yolandaleseb, i can find a han-sebastien there, not a sebastien-han , is that ok?14:15
*** mohankumar has joined #openstack-infra14:15
lesebyolanda: oh hum should be me :p14:16
lesebyolanda: or leseb ?14:16
fungiAJaeger: 248515 seems backwards14:16
*** rameshg87 has joined #openstack-infra14:16
rameshg87hello14:16
yolandahan.sebastien@gmail.com?14:16
yolandai don't see a leseb14:16
rameshg87we started a new third party ci14:16
rameshg87and started posting the logs in one of our servers14:17
fungiAJaeger: i expected a pending change to gerrit/projects.yaml which sets the docimpact-group to openstack-manuals for the projects the docs team wants to maintain14:17
rameshg87https://15.184.134.36/24/249024/2/check/gate-tempest-ironic-ilo-driver-iscsi_ilo/37d9ac8/q-svc.log.2015-11-25-122250.gz14:17
lesebyolanda: is it on launchpad? or on review.openstack?14:17
rameshg87I followed http://docs.openstack.org/infra/system-config/third_party.html#faq-frequently-asked-questions to make it serve gz files, but it is not rendering properly on browser14:17
rameshg87if someone knows it handy, can someone please let me know if there is any quick hack which makes it render properly in official openstack CI14:17
yolandareview.openstack14:18
AJaegerfungi: let me review the list...14:18
*** tiswanso has joined #openstack-infra14:18
yolandaleseb, seb@redhat.com?14:19
*** rcernin has quit IRC14:19
lesebyolanda: yes14:19
yolandaok, that's you :)14:19
lesebyolanda: yes :)14:19
fungiAJaeger: oh, i see, it's already set to openstack-manuals there for swift, nova, et cetera14:19
AJaegerfungi, I think the current status is correct - only the projects that loquacities mentions in her patch now have docimpact-group to openstack-manuals14:19
AJaegerfungi: indeed14:19
yolandadone14:19
fungiAJaeger: so after the jeepyb change merges to flip the default, there can be a follow-up project-config change to clean up the places where docimpact-group is now set to what amounts to the default value for other projects14:20
lesebyolanda: awesome, thanks you :)14:20
AJaegerfungi: yes, indeed.14:20
*** mohankumar has quit IRC14:20
AJaegerfungi: I will do that...14:20
yolandafinally! sorry it took long14:20
lesebyolanda: no worries, thanks for your support, AJaeger as well :)14:20
notmorganfungi: confirmed works with HTTPS explicitly14:21
notmorganfungi: going to update the review.14:21
funginotmorgan: awesome--thanks!14:21
*** thorst_ has joined #openstack-infra14:21
openstackgerritMerged openstack-infra/jeepyb: Change default for docimpact  https://review.openstack.org/24854914:21
openstackgerritMorgan Fainberg proposed openstack-infra/puppet-gerrit: 301 Redirect stripping out double leading slash  https://review.openstack.org/24971414:21
*** thorst__ has joined #openstack-infra14:22
*** spzala has joined #openstack-infra14:22
*** arxcruz has quit IRC14:22
notmorganfungi: and looking at the mod_rewrite trace, this set of redirects suddenly looks better.  wonder if we can [L] the 30114:22
* notmorgan checks14:22
notmorganfungi: looks like we can also [L] it since we're doing a hard 301..but... not sure if we want to14:23
*** yamamoto has quit IRC14:23
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove obsolete docimpact-group settings  https://review.openstack.org/24981114:24
AJaegerfungi, here's that change ^14:24
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add baremetal node definition for hpuseast  https://review.openstack.org/24673914:25
funginotmorgan: yeah, without a more thorough set of test urls that might tickle other redirect rules, i'd be hesitant to [L] that 30114:25
*** arxcruz has joined #openstack-infra14:25
AJaegerfungi: thanks!14:25
AJaegerStevenK: https://jenkins.openstack.org/job/training-guides-upstream-translation-update/4/console is now working!14:25
funginotmorgan: this should be more than sufficient as an interim workaround, i think14:25
notmorganfungi: nod, kindof fun to play systems engineer again for a bit14:26
notmorgan:)14:26
*** thorst_ has quit IRC14:26
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use puppet apply instead of puppet agent  https://review.openstack.org/24949214:27
openstackgerritMonty Taylor proposed openstack-infra/system-config: Stop running puppetmaster  https://review.openstack.org/24981514:27
*** rameshg87 has left #openstack-infra14:27
mordredfungi: I was just updating the docs for the agent/apply move and realized there is one more feature we need to add - disabling nodes14:28
*** amitgandhinz has joined #openstack-infra14:28
mordredfungi: the rest of the patches are still safe to land (and we should)14:28
akuznetsovahi there, I need a little help or explanation, I made a commit https://review.openstack.org/#/c/249666/, so do I see python34 gate here https://review.openstack.org/#/c/249591/ ?14:28
mordredfungi: and I'll poke at options for disabling today14:28
*** dimtruck is now known as zz_dimtruck14:28
*** otter768 has joined #openstack-infra14:28
*** thorst__ has quit IRC14:29
openstackgerritKATO Tomoyuki proposed openstack-infra/project-config: Fix wrong path name  https://review.openstack.org/24981814:29
mordredfungi: puppet apply doesn't seem to have a built-in locking mechanism, so I'm thinking of a puppetmaster-side blacklist to just prevent running the puppet playbook against that host in the first place14:30
*** thorst_ has joined #openstack-infra14:30
openstackgerritSteven Hardy proposed openstack-infra/tripleo-ci: Only build each delorean package once  https://review.openstack.org/24981914:30
mordred(since we also likely do not want ansible rsyncing git repos and hiera data if a server is out of rotation anyway)14:30
fungimordred: seems fine, we just need to socialize it with all the infra-root sysadmins that disabling puppet agent will break when you land that pending change you have (and we should check to see if we have it disabled anywhere we're afraid of breaking, such as review-dev)14:31
mordredfungi: ++14:31
fungii have a feeling landing that change and having puppet suddenly start applying on review-dev again might be a setback for our upgrade testing? zaro can confirm14:32
mordreduh - yeah :)14:32
*** Guest87843 is now known as dansmith14:32
*** amitgandhinz has quit IRC14:33
asselinmordred, can I bug you again for these. It's solves a very old and annoying problem! https://review.openstack.org/#/c/248226/ https://review.openstack.org/#/c/248223/14:33
*** otter768 has quit IRC14:33
mordredfungi: oh - I think I have an idea ... we should be able to make a "disabled" group in the ansible inventory and add !disabled to our host selectors in our playbooks14:34
*** zz_dimtruck is now known as dimtruck14:34
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Remove obsolete docimpact-group settings  https://review.openstack.org/24981114:34
fungimordred: sounds simple. just need to make sure everyone knows how to toggle it before or at worst very shortly after that gets implemented14:34
mordredyup14:34
*** thorst_ has quit IRC14:35
*** Swami has joined #openstack-infra14:35
fungii'm torn on whether that's something that would be better in configuration management or outside/manual (or optionally both?)14:35
*** kgiusti has joined #openstack-infra14:35
openstackgerritThierry Carrez proposed openstack-infra/release-tools: Remove deprecated scripts  https://review.openstack.org/24982214:37
mordredfungi: no reason we can't have it be both I think14:39
mordredfungi: I'll play with some options14:39
openstackgerrityolanda.robla proposed openstack-infra/system-config: Add baremetal node definition for hpuseast  https://review.openstack.org/24673914:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Add some missing docimpact flags  https://review.openstack.org/24982314:39
*** yamamoto has joined #openstack-infra14:40
*** dramalho has joined #openstack-infra14:43
*** Sree has quit IRC14:43
*** gongysh_ has quit IRC14:43
*** akshai has joined #openstack-infra14:47
*** jordanP has joined #openstack-infra14:47
*** rhallisey has joined #openstack-infra14:48
*** dramalho has quit IRC14:48
*** bhunter71 has joined #openstack-infra14:48
rcarrillocruzgreghaynes: heya14:49
rcarrillocruzyou around yet?14:49
openstackgerritMerged openstack-infra/puppet-jenkins: Add the ability to set your own plugin url  https://review.openstack.org/24822314:49
*** ildikov has quit IRC14:52
*** gongysh_ has joined #openstack-infra14:53
*** bexelbie has quit IRC14:53
pabelangermorning! Has anybody actually used nodepool to launch SSH slaves without setting the credentials in secure.conf?14:54
*** pradk has joined #openstack-infra14:54
pabelangerI don't think it actually ever worked14:54
*** olaph has quit IRC14:54
fungimordred: pondering, probably if we do it in config management then we also have a manual workaround by temporarily adjusting it on the puppetmaster while we put together a patch to make it official. and that way we have a better indication of when and why puppet got disabled on specific hosts14:55
pabelangerAlso, newer version of jenkins support creating global credentials via the REST api, so I would thinking of adding something to JJB to maybe manage them now. Thoughts?14:55
fungimordred: the more i think about it the more i like it. never sat well with me that puppet agent --disable state was opaque and lacking in a usable audit trail14:56
openstackgerritMerged openstack-infra/puppet-openstackci: Automate installation of scp jenkins plugin  https://review.openstack.org/24822614:56
*** whoops has quit IRC14:56
fungimordred: and we've all spent more time than we probably should have trying to track down whether and why puppet agent was disabled somewhere in past years14:57
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Fix wrong path name for training-guide translations  https://review.openstack.org/24981814:57
*** olaph has joined #openstack-infra14:57
*** whoops has joined #openstack-infra14:57
*** gongysh_ has quit IRC15:01
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-6wind-virtual-accelerator in Openstack NS  https://review.openstack.org/24922915:03
bmwiedemannfungi: would this be an infra or a neutron issue? https://bugs.launchpad.net/neutron/+bug/151975015:03
openstackLaunchpad bug 1519750 in neutron "stable tarball contains master version" [Undecided,New]15:03
rcarrillocruzmordred: you have a sec to chat about glean?15:05
*** ildikov has joined #openstack-infra15:05
AJaegerbmwiedemann: we switched neutron to use constraints, might be related to that.15:05
bmwiedemannstill worked yesterday (or the day before)15:06
AJaegerbmwiedemann: yep, yesterday merged I152ed78813e46ffc3f72ff7dec5ed2872268f5bd15:06
openstackgerritMateusz Matuszkowiak proposed openstack-infra/project-config: Added new repo for fuel-plugin-6wind-virtual-accelerator in Openstack NS  https://review.openstack.org/24922915:06
AJaegerlet me send an email to Sachi to fix it, not sure what's the IRC nick is...15:07
mtreinishmordred, fungi: if we wanted to resize the trove instance for the subunit2sql db to give it a little more oomph, is there any reason not to just max it out withe largest flavor?15:07
mtreinishAJaeger: Nakato15:07
*** mjturek1 has joined #openstack-infra15:07
mordredmtreinish: dunno?15:07
rcarrillocruzNakato: ^15:07
bmwiedemannAJaeger: thanks :-)15:07
rcarrillocruzAJaeger: ^15:08
mordredfungi: yes to config management15:08
mordredjeblair: ping when you're awake15:08
mordredjeblair: I need help figuring out the right thing to do with python logging15:08
AJaegermtreinish: thansk!15:08
AJaegerNakato: Just send an email with the basic information if you read this once you're awake...15:09
*** e0ne_ has quit IRC15:09
*** Qiming has joined #openstack-infra15:09
mtreinishmordred: well I thought maybe cost concerns. It's like ~$4/hr for the largest 64gb flavor15:09
rcarrillocruzmordred: the vlan interface fix worked, glean created the conf file under /etc/network/interfaces.d just fine. But we are not getting the vlan interfaces configured on boot (along with some other interfaces as well). If I run 'glean' manually after boot then all the nics get their corresponding file under /etc/network/interfaces.d . I believe the issue is due to how glean init service works: http://git.openstack.org/cgit/openstack-infra/glean/15:10
AJaegerfungi, FYI - there are some more neutron constraints jobs broken, one was fixed already today.15:10
rcarrillocruzthat says glean will run in a per-physical device, whenever it gets actioned by upstart and the local filesystem is mounted15:11
rcarrillocruzhowever, the vlan is virtual, therefore that's never actioned on boot15:11
rcarrillocruzimho, we should change the glean init service to be run post-boot, only once, and without any arguments15:11
*** e0ne has joined #openstack-infra15:11
rcarrillocruzso glean creates files for ALL interfaces15:11
rcarrillocruznot just whichever interface gets activated during boot15:11
rcarrillocruzand yeah, we have to get glean to run prior to the network script of  ubuntu, so it has right config files in place15:12
rcarrillocruzthoughts?15:12
*** rbrndt has joined #openstack-infra15:13
*** signed8bit has joined #openstack-infra15:14
openstackgerritMiguel Angel Ajo proposed openstack-infra/irc-meetings: Update neutron-qos meeting to bi-weekly and fix chairs  https://review.openstack.org/24984215:14
fungiAJaeger: sachi is Nakato in here15:14
fungiAJaeger: bmwiedemann: i'll look into it too15:14
fungimtreinish: sure, we can max it out but i think that's only 500gb. looking now15:14
AJaegerfungi: https://review.openstack.org/#/c/247306/ broke it, and https://review.openstack.org/#/c/249722/ is one fix...15:15
*** tsg_ has joined #openstack-infra15:15
fungimtreinish: trove instances are sized by backing volume15:15
mtreinishfungi: well we need to max out the volume to do the migration anyway15:16
mtreinishfungi: but there is also a trove instance resize to change the backing nova flavor15:16
fungimtreinish: oh, i'll see if rackspace exposes that15:16
*** nmagnezi has quit IRC15:16
fungimtreinish: part of the issue i foresee is that rackspace also doesn't allow us to resize nova instances to larger flavors. they recommend rebuilding and restoring your data from a backup instead15:17
fungichecking to see if they're any different with trove15:17
mtreinishfungi: http://docs.rackspace.com/cdb/api/v1.0/cdb-getting-started/content/trove_summary_instanceactions.html it looks like it supports it15:17
openstackgerritsebastian marcet proposed openstack-infra/openstackid-resources: Summit Application API  https://review.openstack.org/22196415:18
rcarrillocruzcrinkle: i believe that's also the issue you are seeing ^15:18
fungimtreinish: ahh, yep they do15:18
mtreinishfungi: well we also did a trove backup last night (hopefully it's done by now) before we triggered the migration15:18
rcarrillocruzre: infra servers not having all interfaces setup, that's due to glean running-per device on init activation15:18
fungimtreinish: any idea when you want to do the volume increase? no guess as to how long it might end up offlining the database since we don't have any others approaching this size15:18
mtreinishfungi: assuming the backup is done we're on step 6: https://etherpad.openstack.org/p/the-big-subunit2sql-migration15:18
mordredrcarrillocruz: problem is ...15:19
rcarrillocruzif we force glean to run against all of them regardless of nic events, we'll get everything under /etc/network/interfaces.d15:19
* rcarrillocruz listens15:19
mordredrcarrillocruz: there are services that will not start properly if network isn't up - like ssh15:19
openstackgerritClaudiu Belu proposed openstack-infra/irc-meetings: Updates the Nova Hyper-V Team meeting  https://review.openstack.org/24956615:19
mtreinishfungi: as soon as you can. (assuming the backup is done) It's a prereq for the migration15:19
fungimtreinish: just confirmed, it will let us go up to 500gb disk and 64gb ram15:19
mordredrcarrillocruz: so we need to run glean at the time in the boot sequence that the server thinks it's confiruging interfaces15:19
mtreinishfungi: awesome15:20
mordredrcarrillocruz: BUT - I think figuring out how to lock/block/only-run-once-and-do-all-interfaces-from-config even though glean might get invoked multiple times is a fine idea15:20
mordredalso15:20
fungimtreinish: it might make sense to do the ram resize first since it's possible that could make the disk resize a little faster (also the ram resize might be slower with a larger volume, depending on whether we get rescheduled to a new host and how they implement the backing volumes)15:21
mordredTHE PEOPLE WHO THOUGH EVENT BASED INIT SYSTEMS WERE CLEVER NEED TO HAVE THEIR EYES GOUGED OUT15:21
notmorganmordred: ++15:21
*** ntpttr has joined #openstack-infra15:21
rcarrillocruzyeah really15:21
rcarrillocruzi looked at upstart and is not straightforward15:21
rcarrillocruz:/15:21
notmorgani liked my old sysvinit tyvm...and i want it back... for my servers (staying out of the laptop argument)15:21
mordrednotmorgan: ++15:21
mtreinishfungi: ok, makes sense to me15:22
jeblairmordred: mrfph.15:22
notmorganjeblair: you sound so awake15:22
mordredjeblair: yay! you're awake!15:22
jeblairmrfph.15:22
ntpttrHey everyone, so the inital commit for puppet magnum was merged last week, and you can see it here http://git.openstack.org/cgit/openstack/puppet-magnum/ but on the actual github page nothing is showing up https://github.com/openstack/puppet-magnum15:22
* mordred bounces up and down15:22
rcarrillocruzi mean, there must be some way to express 'run glean before the network configuration and network services start'15:22
mtreinishfungi: although I'd assume the volume resize just calls cinder and does some san magic to make a bigger volume. But, who knows..15:22
rcarrillocruzi'm sure!15:22
ntpttrdoes anyone know why that is?15:22
* mordred tugs on jeblair's shirt15:22
fungimordred: unfortunately event-driven hardware discovery and hotplugging go hand-in-hand15:23
fungihard to have the latter work reliably without the former15:23
mordredfungi: sure. but  how many times have you wanted action taken on a server automatically when you plug in new hardware?15:23
*** mrmartin has quit IRC15:23
jeblairmordred: how else do those rootkits get run?15:24
rcarrillocruzmordred: i could write a superdumb element that does a firstboot 'glean; ifup -a' at the end of the postsequence15:24
mordredlike, is that _ever_ a desire you've had? hell, I plug in a new hot-plug drive I don't want it to auto-join the raid15:24
fungimordred: true. i didn't mind the good old days of telling my scsi controller to reprobe the bus, or rescanning my usb devices15:24
*** sfinucan has joined #openstack-infra15:24
mtreinishmordred: I had a rack without a kvm once, so it was nice to have a usb keyboard to plug in as I worked my way up the rack15:24
mordredrcarrillocruz: please re-explain "postsequence"15:24
jeblairfungi: to be fair, those are both kernel actions...15:24
rcarrillocruzpost all init sequence15:24
rcarrillocruzat the end of all init15:24
mordredrcarrillocruz: it can't go there15:24
notmorganmordred: It looks like you are trying to plugin a new hot-pluggable hard drive! ... /clippy15:24
rcarrillocruzjust to be sure glean runs and sets up ALL interfaces15:24
funginotmorgan: udevd as clippy. i like!15:25
mordredrcarrillocruz: because other things, like ssh, or other programs, expect working network15:25
rcarrillocruznot the ones that upstart got events from...15:25
mordredrcarrillocruz: OR15:25
notmorganfungi: EXACTLY!15:25
jeblair(which kernel actions don't need an event based init to happen)15:25
mordredrcarrillocruz: that means that you'll have interfaces trying to configure themselves up with dhcp in places without dhcp15:25
jeblairmordred: anyway -- logging -- you should log things!15:25
mordredjeblair: I should!15:25
mordredjeblair: here's the thing15:25
mordredjeblair: our magical "log at error level in shade exceptions" thing?15:26
jeblair(but not too many things)15:26
notmorganjeblair: But... but... my server will boot 10 seconds slower without the event based system! </watches the 5 minutes of post>15:26
mordredjeblair: it makes the ansible inventory suck15:26
mordredjeblair: because there are several things in the inventory script where we try something, it fails, that's fine, we move on, it's a fine error to have15:26
fungintpttr: i expect the github api derped when our rube goldberg machine tried to grant gerrit push rights to that repo. i'll check/fix15:26
mordredjeblair: but the console now has a bazillion exceptions printed15:26
notmorganmordred: euuw.15:26
mordredjeblair: so I'm trying to figure out how to log exceptoins - except for the exceptions that are our only way of control flow15:27
mordredand thus not _really_ exceptions15:27
ntpttrfungi: thanks :)15:27
*** ccrouch has joined #openstack-infra15:27
fungintpttr: that seems to have been it. the github mirror of that repo has content now15:28
*** mtanino has joined #openstack-infra15:28
ntpttrfungi: awesome, looks like it's working now yep15:28
jeblairmordred: those are only a problem if they get wrapped in shade exceptions... but i guess that happens15:28
*** amitgandhinz has joined #openstack-infra15:28
mordredjeblair: yes. this is specifically a wrapped exception :)15:29
mordredjeblair: it's "hey, get me a list of volumes for this server" ... if that doesn't work on the cloud in question, it's fine, the volume list is just []15:29
*** devl_ has joined #openstack-infra15:29
jeblairmordred: it seems like the definish characteristic is where the exception is propagated, so perhaps we should return to this idea: https://review.openstack.org/22536115:30
jeblairer difining15:30
signed8bitzaro: Got the company all clear to contribute to Gerrit. Has the situation changed since the last time we spoke?15:30
jeblairdefining15:30
jeblairwhatever.  morning.15:30
*** chlong has quit IRC15:31
mordredjeblair: so - we sort of do that already, but that would still have the problem I've got now15:32
*** ntpttr has left #openstack-infra15:32
mordredjeblair: which is in the calling context there is specifically one call that the consuming program knows is not a problem - but there are other calling programs that could very much care about it15:32
notmorganjeblair: yeah the context manager thing i just floated toward mordred :P in fact15:33
jeblairmordred: no, i mean, just stop logging it in shade and put the onus on the user to examine the inner exception.15:33
notmorganhaha15:33
mordredjeblair: OH15:33
mordredjeblair: I guess we could do that- it seems like it makes the common case more onerous15:33
*** amitgandhinz has quit IRC15:33
*** fhubik is now known as fhubik_brb15:34
notmorganmordred: well only a little, you could just write a simple helper function to mirror what is done today15:34
notmorganyes it's an extra line15:34
notmorganbut15:34
notmorganfunction/whatever.15:34
mordredright - but as a consumer you'd have to wrap _every_ shade call with a with shade_exception_logging15:34
*** bswartz has joined #openstack-infra15:34
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: ceilometerclient: enable aodh  https://review.openstack.org/24985315:34
mordredwhich is yuck. what I really want is a way to wrap a single call in a context manager that says "don't log errors please"15:34
*** mohankumar has joined #openstack-infra15:34
jeblairmordred: it does, but otherwise i think you're trying to predict within shade whether an exception you are wrapping is going to be interesting or not15:34
*** kushal has joined #openstack-infra15:35
mordredjeblair: I'm fine with end-user asserting that an exception is not interesting - I just don't know physically how to implement that15:35
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: ceilometerclient: enable aodh  https://review.openstack.org/24985315:35
jeblairmordred: no, i mean you15:35
jeblairmordred: in shade15:35
*** ddieterly has joined #openstack-infra15:36
notmorganif the interfaces are kept the same but you know you don't care about the "exception" you use something like a partial (or meta-programming layer) to call ._sqaush_exception ?15:36
notmorganso common case makes no change but when you don't care (aka in the special cases) you have a way to de-wrape.15:37
notmorgande-wrap the uninteresting stuff*15:37
* notmorgan is trying to communicate that in some coherant way on irc.15:37
jeblairnotmorgan: i think the problem is that the logging happens at the time of wrapping15:37
fungimtreinish: rackspace claims the "backup-before-migration" mysql instance backup was last modified "04:28 AM WET" (i have _no_ idea what timezone that is, but i guess it completed?)15:37
notmorganjeblair: right, which is why a partial is used or somesuch where you can change the wrap control flow.15:38
*** fhubik_brb is now known as fhubik15:38
*** signed8bit has quit IRC15:38
*** fhubik is now known as fhubik_brb15:38
notmorganjeblair: instead of relying on "this has been wrapped no change is allowed"15:38
jeblairfungi: http://www.timeanddate.com/time/zones/wet  ?  apparently rather close to utc?15:38
*** FallenPegasus has joined #openstack-infra15:38
mtreinishfungi: sounds that way15:39
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: ceilometerclient: enable aodh  https://review.openstack.org/24985315:39
mordredjeblair: so back to the thing you were saying to think through it out loud15:39
*** ddieterl_ has quit IRC15:39
mtreinishfungi: heh, it's the wet timezone, I imagine it's in the middle of an ocean :)15:39
mtreinishoh, google says western european time15:40
fungiahh15:40
mordredjeblair: you're saying we'd not log.error in OpenstackExceptoin and then in nodepool do a try: except OpenStackException as e: e.log_error() or something15:40
jeblairnotmorgan: i don't quite what you're getting at, but that doesn't mean it's not right, it just means i haven't had breakfast.15:40
*** signed8bit has joined #openstack-infra15:40
fungii wonder why rackspace would list times in wet?15:40
notmorganjeblair: fair enough. ;) breakfast is important15:40
mordredjeblair: so, we log no exceptions and we leave it to the user to log exceptions if they catch them and care, yeah?15:40
*** thiagop has joined #openstack-infra15:40
*** matrohon has quit IRC15:41
fungimtreinish: anyway, i guess i'll grow the flavor and then the backing volume15:41
mtreinishfungi: cool, thanks15:41
jeblairmordred: yeah, and since python2 doesn't have chained exceptions, if you want the inner one, as a user, you have to do extra work to print it.  python3 will make this easier i think.15:41
jeblairnotmorgan: do you think you could sketch out your idea (maybe just with a simple test program)?15:41
mordredjeblair: what about this: http://paste.openstack.org/show/48000815:41
mordrednotmorgan: ^^15:41
notmorganjeblair: trying to figure that out actually if i can.15:42
notmorganjeblair: :)15:42
notmorganjeblair: but mulling over how it looks.15:42
jeblaircool15:42
mordredwell, without the syntax error15:42
mordredI do not propose that we solve the problem with a syntax error15:42
notmorganmordred: if you can solve it with a syntax error more power to you15:42
mordredif there is a problem that can be solved with syntax errors, I will be the KING of solving those problems15:43
jeblairmordred: that'll make the wrapper a one liner15:43
*** FallenPegasus has quit IRC15:43
*** sergmelikyan has joined #openstack-infra15:43
mordredwell, you don't really even need a wrapper in that case15:43
notmorganmordred: that might be the easiest / lowest cost to achieve the goal right there15:43
jeblairmordred: sure you do -- you need something to call that function :)15:43
mordred:)15:43
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add bifrost manifest  https://review.openstack.org/23175715:44
mordredjeblair: lemme go look at nodepool consumption real quick15:44
fungimtreinish: cpu/load don't seem historically high on the current flavor, but it does seem to be using all 8gb of its ram (according to rackspace's monitoring data anyway) currently so sizing up seems prudent15:44
*** fhubik_brb is now known as fhubik15:44
notmorganfungi: i'm not surpised it is using up the ram.15:44
notmorganfungi: i mean... ;)15:44
fungimtreinish: when i size up, the instance is going to restart, btw15:44
openstackgerrityolanda.robla proposed openstack-infra/puppet-infracloud: Add bifrost manifest  https://review.openstack.org/23175715:45
* notmorgan stops trying to carry on three convos at once15:45
mtreinishfungi: that should be fine, we already killed the worker15:45
*** angdraug has joined #openstack-infra15:45
fungimtreinish: also it seems to allow for sizing down, so we can run it at 64gb for a bit and then scale back if it can't make use of all that15:45
mordredjeblair: yeah - I think we're still screwed with that - nodepool doesn't catch any OpenStackCloudExceptions anywhere15:45
mtreinishhow openstack-health responds will be interesting though15:45
jeblairmordred: you either wrap a try/except block around every shade call from nodepool, or you wrap a context manager around every shade call from nodepool; the latter is less typing.15:45
jeblairmordred: or.... you make a class that wraps the shade cloud class and transparently wraps every call before automatically dispatching it to the inner class15:45
jeblairmordred: right, it catches ose here where i've added it: https://review.openstack.org/22536115:46
rcarrillocruzmordred: when i said creating an element that does 'glean; ifup -a' at the end of the init sequence i mean that's an additional one. Glean would run just as is now (configuring devices as they are detected on boot, so ssh and all can work afterwards) and the element i'd put would run that dummy script at the end of the init sequence to get the configuration files of the devices not detected (like vlan ones)15:47
rcarrillocruzanyway, i'll check it out15:47
mordredrcarrillocruz: OHHHHHH15:47
mordredrcarrillocruz: gotcha. so a thing to get the vlans - not a change to how the current ones trigger15:47
mordredrcarrillocruz: yes. that will be great15:47
jeblair(if you did that last idea of automatically wrapping the class in shade itself, you could probably have "log inner exceptions at the border between shade and its users" automatically happen within shade)15:47
jeblair(you could also just decorate every public api method with something that did that too)15:48
jeblair(which is more typing but also more straightforward)15:48
mordredjeblair: I feel like we wanted the error logging in the exception for nodepool because there were things that were not being propogated as errors that we still wnated to be abl to log because they were interesting for us to track in our logs - but I also might be making that up15:49
*** diana_clarke has quit IRC15:49
jeblairmordred: no, it's because shade is hiding real error exceptions inside of OSE and then raising OSE, so nodepool gets "derp, shade said something went wrong, no idea what" as an error from shade.15:49
mordredjeblair: ok. that's a way easier thing to deal with and I agree that the wrapper is an easy way to handle it15:50
mordredjeblair: I may have been tryingt o solve a MUCH more complex problem15:50
*** flepied1 has joined #openstack-infra15:50
*** yonglihe has joined #openstack-infra15:50
jeblairmordred: things like what you are talking about are being handled by copious debug statements in shade itself15:50
*** nelsnelson has quit IRC15:50
jeblairmordred: i think the thing you mentionid is handled by the pattern of issuing debug statements before handling internal flow-control exceptions15:51
jeblairmordred: which i this is pretty solid at this point15:51
mordredya. I agree. and cool.15:51
AJaegerjeblair: could you review the relesae notes post job, please? https://review.openstack.org/#/c/248256/ or should I ask others to review?15:51
jeblairmordred: the problem we're trying to solve is the one where the swiftclient parameter is wrong, but we couldn't tell that because shade wrapped it inside OSE15:52
*** flepied has quit IRC15:52
*** tsg_ has quit IRC15:53
*** tsg has joined #openstack-infra15:53
*** ddieterly has quit IRC15:54
*** Sukhdev has joined #openstack-infra15:54
openstackgerritMonty Taylor proposed openstack-infra/shade: Only log errors in exceptions on demand  https://review.openstack.org/24986815:55
jeblairmordred: so....15:56
jeblairmordred: i actually want to log that in the nodepool context15:57
jeblairmordred: so that nodepool users don't have to have shade logging enabled in order to see an error that nodepool is failing to handle15:57
jeblairmordred: but _inner_exception now has a _15:57
* AJaeger will be back later...15:57
*** AJaeger has quit IRC15:57
jeblairmordred: (i mean, the log_error method is nice, but i don't think i want to use it for that reason)15:57
mordredjeblair: ah. maybe optional param to log_error that is a logger to use?15:58
*** claudiub_ has quit IRC15:58
jeblairmordred: why not make the inner exception public?15:58
nibalizergood morning15:58
mordredjeblair: or maybe also a "get_inner_exception"15:58
mordredor we could do that15:58
fungimtreinish: instance resize completed and volume is being grown now15:58
jeblairmordred: get_inner_exception would be fine :)15:58
jeblaireither way15:58
*** pcrews has joined #openstack-infra15:58
*** ashleighfarnham has joined #openstack-infra15:58
notmorgannibalizer: oh hi.15:58
openstackgerritMehdi Abaakouk (sileht) proposed openstack-infra/project-config: Gnocchi: Fix mapping between of stable branches  https://review.openstack.org/24805015:58
mtreinishfungi: awesome, I'll get my trigger finger ready for pushing the release15:58
*** boris-42 has quit IRC15:58
fungimtreinish: no clue how long resizing from 300-500gb will require15:59
mtreinishfungi: heh, hopefully not too long15:59
fungii'll keep an eye on it and let you know16:00
mtreinishthanks16:00
fungimtreinish: claims to be done! that was quicker than i ever hoped... you might want to do some test queries16:01
openstackgerritMonty Taylor proposed openstack-infra/shade: Only log errors in exceptions on demand  https://review.openstack.org/24986816:01
mordredjeblair: so like that perhaps (do all of the above :) )16:01
*** diana_clarke has joined #openstack-infra16:01
nibalizeroh good the subunit2sql procedure is proceeding16:02
jeblairmordred: yep, but problem; comment inline16:02
mtreinishfungi: seems to be working. http://status.openstack.org/openstack-health/#/ also works16:02
mordrednibalizer: https://review.openstack.org/#/q/status:open++topic:puppet-apply,n,z everything up to the -2'd patch is ready16:02
mtreinishok, I'll push the release to trigger the migration now16:03
nibalizermordred: fungi part of the good about puppet agent --disable was that for a human running puppet on the machine manually it was disabled16:03
mordredjeblair: DOH16:03
mordrednibalizer: yup. and that is not possible, best I can tell, with apply16:03
funginibalizer: that always struck me as a hindrance, but each to his own16:03
nibalizerso if we only disable it on the puppetmaster, what is our procedure for running pupet manually while debugging and how do we make sure we respected disabled16:03
mordrednibalizer: but I believe I have a solution at the ansiblelevel16:03
funginibalizer: don't run puppet manually unless you know it's okay to do so?16:04
mordredyah16:04
mordrednibalizer: well, our procedure becomes running puppet apply on the host itself16:04
mordredand in fact, you should probably never do that without checking to see if it's been disabled16:04
openstackgerritMerged openstack-infra/project-config: Fix releasenotes post job  https://review.openstack.org/24825616:04
mtreinishok, release pushed16:04
mordredok. neat16:04
fungiand if we disable it in system-config then it's hopefully clear to everyone what's disabled, when it happened and why16:04
mordredI can put a static inventory file along side the dynamic one16:04
mordredand in that I can make groups that I can add the dynamic hosts to16:05
nibalizerok if thats the procedure then thats fine16:05
mordredso it's easy to make a 'disabled' group16:05
*** Qiming has quit IRC16:05
nibalizeri'm wondering if I can get puppetboard to show a tab for disabled nodes16:05
mordredand change the playbooks to include '!disabled'16:05
jeblairoh, disabling will happen through system-config?16:05
mordredjeblair: that was what fungi was suggesting, yeah16:06
mordredI tink there are two parts16:06
mordreda) what is the physical mechanism16:06
jeblairthat has some nice things about it, but sometimes we disable puppet because there's something wrong with the ci system, in which case it would be nice not to have to wait indefinitely for a 'disable' change to land16:06
mordredb) how is it driven by system-config / by hand16:06
jeblair(and/or elevate gerrit privs to force-merge it)16:07
fungijeblair: right, seems like we could do something similar to make the same change locally on the puppetmaster in advance of getting a system-config change merged to reflect that16:07
nibalizeri think mordred is saying we do it at the inventory layer16:07
mordredjeblair: if we do it right, then editing the file on disk on puppetmaster should be able to disable puppet running on puppetmaster16:07
mordredwhich should disable updating that file16:07
nibalizerthen a follow up change through system-config makes it more permanent16:08
mordredfor the times when we need to emergency-manual stop things16:08
jeblairyeesh, that sounds dangerous16:08
jeblair(and racy)16:08
mordredwe can also have 2 files16:08
mordredone that is puppet managed and one that is not16:08
nibalizer2 files souns good to me16:08
mtreinishok, the release has been uploaded: https://pypi.python.org/pypi/subunit2sql/1.0.016:08
mordred"static-groups" and "zomg-emergency"16:08
*** rhallisey has quit IRC16:08
* mordred tests that real quick16:08
mtreinishnow to wait for puppet to pick it up on logstash.o.o and trigger the migration16:08
jeblairthat sounds less racy16:09
*** dims has quit IRC16:09
openstackgerritJames E. Blair proposed openstack-infra/nodepool: Log shade inner exceptions  https://review.openstack.org/22536116:09
*** armax has joined #openstack-infra16:10
*** Piet has joined #openstack-infra16:10
*** ddieterly has joined #openstack-infra16:10
*** dims has joined #openstack-infra16:11
openstackgerritMonty Taylor proposed openstack-infra/shade: Only log errors in exceptions on demand  https://review.openstack.org/24986816:11
mordredjeblair: how bout that ^^ ?16:11
*** sergmelikyan has quit IRC16:12
*** sergmelikyan has joined #openstack-infra16:13
jeblairlgtm16:13
openstackgerritLiuqing Jing proposed openstack-infra/project-config: Add check-requirements job for evoque-dashboard  https://review.openstack.org/24988416:13
pabelangerany infra-root mind helping fedora-23 nodes for nodepool land? https://review.openstack.org/#/c/247908/416:16
*** dizquierdo has quit IRC16:16
jeblairpabelanger: does that need a root?16:17
pabelangerjeblair: project-config team I guess16:17
pabelangerkeep messing up the teams16:17
jeblairpabelanger: also it still needs it's dib dependent patch merged16:18
pabelangerjeblair: which one is that?16:18
pabelangereye not working this morning16:18
jeblairpabelanger: Depends-On: I1a6e45d04755515286b3d49f8280c16b527e2f48     https://review.openstack.org/24729516:19
*** mahatic has quit IRC16:19
pabelangerjeblair: gerrit web show it is merged here16:20
pabelangershows*16:20
jeblairpabelanger: doh, sorry16:20
openstackgerritIhar Hrachyshka proposed openstack-infra/irc-meetings: neutron-upgrades: moved the meeting to -alt slot that is free now  https://review.openstack.org/24988916:21
openstackgerritDong Ma proposed openstack-infra/jenkins-job-builder: Add support for the RunDeck plugin  https://review.openstack.org/20572116:21
pabelangerjeblair: NP! I was sweating for a second16:21
openstackgerritDmitry Tantsur proposed openstack/diskimage-builder: Add kmod to package-installs of ironic-agent  https://review.openstack.org/23523616:23
*** Sukhdev has quit IRC16:23
openstackgerritMerged openstack/requirements: Rename semantic-version dep to semantic_version  https://review.openstack.org/24932716:25
mordredfungi, nibalizer: you have a sec to +A https://review.openstack.org/#/c/249868/ ? I'd like to get that in and cut a bugfix release - ansible openstack inventory is super ugly without it - and I'm working on the system-config ansible inventory patch currently :)16:25
mordredjeblair: hrm - I guess ^^ means that I need to do the system-config clouds.yaml patch first16:25
*** zhenguo has quit IRC16:25
nibalizerlooking16:25
*** flepied1 has quit IRC16:25
*** sergmelikyan has quit IRC16:26
jeblairmordred: what patch?16:26
mordredjeblair: the one to start managing the clouds.yaml file on puppetmaster for launch_node16:27
*** mohankumar has quit IRC16:27
jeblairah gotcha16:27
mordredjeblair: which will also be important for using the ansible inventory16:27
jeblairmordred: i made some local changes to that file, btw.16:27
mordredwhich will be important when we don't have certs in puppetmaster to list anymore16:27
nibalizermordred: the one you linked is the logging one16:27
mordredjeblair: cool16:27
mordrednibalizer: yes16:27
mordrednibalizer: bad logging == bad inventory unhappy16:28
*** mahatic has joined #openstack-infra16:28
*** devl_ has quit IRC16:29
mtreinishhmm, it looks like it picked up the new package: http://puppetboard.openstack.org/report/logstash.openstack.org/e16a3c1494eb5bb264eac5c02190ca62d7a9c384 but I don't see the exec16:29
openstackgerritOleksii Chuprykov proposed openstack-infra/project-config: Add experimental job for testing heat api against apache2  https://review.openstack.org/22887816:29
*** otter768 has joined #openstack-infra16:29
*** angdraug has quit IRC16:30
*** scheuran has quit IRC16:30
mtreinishnibalizer: if it triggered the exec (http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/manifests/server.pp#n52 ) it would show up on the report right?16:30
*** placko has quit IRC16:30
*** rhallisey has joined #openstack-infra16:31
nibalizermtreinish: ya16:31
mtreinishhmm, I wonder why it didn't get triggered then?16:32
*** mahatic has quit IRC16:32
pleia2I think someone said we were having trouble seeing execs in puppetboard16:32
*** mahatic has joined #openstack-infra16:32
*** achanda has joined #openstack-infra16:32
fungithat might be a security "feature"16:32
mtreinishheh, it might be :)16:33
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Make delete_server() return True/False  https://review.openstack.org/24989816:33
fungiroot     32023  0.2  1.7 185624 35736 ?        S    16:26   0:00 /usr/bin/python /usr/local/bin/subunit2sql-db-manage --config-file /etc/subunit2sql.conf upgrade head16:33
fungiso it's still going i guess16:33
mtreinishfungi: ok cool thanks (faster than I could even ask for someone to check)16:33
mtreinishfungi: yeah SpamapS and I are expecting it to take days...16:34
nibalizerok16:34
fungihere's hoping the server doesn't crash/get spontaneously rebooted in the interim16:34
mtreinishfungi: https://github.com/openstack-infra/subunit2sql/blob/master/subunit2sql/migrations/versions/2822a408bdd0_uuid_to_integer_ids.py16:34
*** otter768 has quit IRC16:34
jeblairpuppet is running that?16:34
*** pblaho has quit IRC16:34
mtreinishjeblair: we background it16:34
mtreinishwe've had that issue before16:34
jeblairoh, you did "puppet agent --test &" ?16:35
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Make delete_server() return True/False  https://review.openstack.org/24989816:35
pleia2airplane for thanksgiving now, will be back monday o/16:35
mtreinishjeblair: no, the exec is backgrounded16:35
*** fedexo has joined #openstack-infra16:35
jeblairpleia2: godspeed16:35
*** ddieterly has quit IRC16:35
jeblairmtreinish: ah ok16:35
jeblaircool16:35
fungiright, don't want the puppet timeout to kill it (again!)16:35
jeblairand hopefully in a way that separates it from the timeout process group16:35
mtreinishjeblair: http://git.openstack.org/cgit/openstack-infra/puppet-subunit2sql/tree/files/run_migrations.sh16:35
mtreinishfungi, jeblair: it's actually the ansible timeout which was the problem iirc16:36
armaxjeblair: hi16:36
mtreinishwe turn off the puppet timeout in the manifest16:36
armaxjeblair: it looks like you’re the admin for https://launchpad.net/python-neutronclient16:36
mtreinishbut ansible was killing puppet after like 30min16:36
armaxjeblair: could you change the maintainer to the LP neutron drivers team, please?16:37
armaxjeblair: https://launchpad.net/~neutron-drivers16:37
jeblairarmax: done16:37
armaxjeblair: thanks!!16:37
mtreinishjeblair, fungi: iirc, it worked the last time we did a lengthy migration16:38
jeblair(neutron-drivers is owned by openstack-admins, so we still have emergency access if needed)16:38
*** kushal has quit IRC16:38
jeblairmtreinish: cool16:38
onovyhttp://lists.openstack.org/pipermail/openstack-infra/2015-November/003475.html gentle ping, someone?16:38
*** Liuqing has quit IRC16:40
*** devl_ has joined #openstack-infra16:40
*** devl_ has quit IRC16:42
*** devl_ has joined #openstack-infra16:42
*** ajmiller has joined #openstack-infra16:43
*** unicell1 has joined #openstack-infra16:43
*** devl_ has quit IRC16:44
*** devl_ has joined #openstack-infra16:44
*** mohankumar has joined #openstack-infra16:44
*** unicell has quit IRC16:45
*** _ody has joined #openstack-infra16:45
*** Sukhdev has joined #openstack-infra16:45
*** devl_ has quit IRC16:46
ihrachyscan we get some attention from infra to the patch that is sent week+ ago to fix some periodic stable jobs? https://review.openstack.org/#/c/245525/16:46
*** eil397 has joined #openstack-infra16:49
*** trown is now known as trown|lunch16:52
*** hdd has joined #openstack-infra16:54
*** mohankumar has quit IRC16:55
fungiihrachys: can you link to a log where you tested my original recommendation? i based it on what variables the log claimed were set. is it possible we're running afoul of tox sanitizing the environment?16:55
*** mrmartin has joined #openstack-infra16:56
*** mjturek1 has left #openstack-infra16:56
ihrachysfungi: I can't recollect what recommendation you refer to.16:57
*** jordanP has quit IRC16:57
*** jordanP has joined #openstack-infra16:57
Jokke_Is the merge-check Q ok or someone looking into it already?16:58
*** amuller is now known as amuller_afk16:59
fungiihrachys: per your comment to me on patchset 1 of the change you just mentioned16:59
fungiihrachys: why do you think merge-check might not be ok?17:00
fungier, Jokke_ ^17:00
fungisorry ihrachys17:00
*** asselin has quit IRC17:00
*** asselin has joined #openstack-infra17:00
ihrachysfungi: because I see other variables passed into zuul-cloner, but not that one. and in non-periodic jobs it works17:01
*** dizquierdo has joined #openstack-infra17:01
Jokke_fungi: just looking the Q size of 465 items ... looks quite a lot17:01
Jokke_fungi: thus asking if that's normal17:02
*** jistr has quit IRC17:02
fungiihrachys: my original recommendation was to set ZUUL_BRANCH not BRANCH17:02
fungiihrachys: zuul-cloner ignores BRANCH, it expects ZUUL_BRANCH17:02
ihrachysfungi: ok. but we set ZUUL_BRANCH in tox_install.sh anyway. I am happy to change to ZUUL_BRANCH if that's the only blocker here.17:03
*** markvoelker has quit IRC17:03
*** abregman has quit IRC17:03
*** jlanoux has quit IRC17:04
fungiJokke_: it's normal. when a change merges, all open changes on the same branch of the same project get enqueued into the merge-check pipeline17:04
*** ihrachys has quit IRC17:04
fungiJokke_: that can end up being rather a lot of open changes17:04
Jokke_fungi: ok, thnx ... won't pay too much attention to it in future then ;)17:04
*** rwsu has quit IRC17:04
fungiihrachys: i'm a couple weeks out of context since having previously discussed the issue, so mainly asking for an example log of the test tox.ini change for a neutron-*aas repo to refresh my memory/confirm the current issue17:05
*** ifarkas has quit IRC17:06
*** tmorin has quit IRC17:06
*** bcourt has joined #openstack-infra17:07
*** AJaeger has joined #openstack-infra17:08
*** derekh has quit IRC17:08
*** fhubik has quit IRC17:08
*** e0ne has quit IRC17:09
*** eil397 has quit IRC17:12
*** ildikov has quit IRC17:12
*** mohankumar has joined #openstack-infra17:12
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add pip-and-virtualenv element  https://review.openstack.org/23956117:13
*** rossella_s has quit IRC17:14
*** olaph has quit IRC17:15
*** sergmelikyan has joined #openstack-infra17:15
*** amotoki has quit IRC17:15
*** flepied has joined #openstack-infra17:16
*** rossella_s has joined #openstack-infra17:16
*** flwang1 has joined #openstack-infra17:16
*** _ody has quit IRC17:16
jeblairfungi: oh wait, i'm trying to avoid having jobs set ZUUL_BRANCH17:17
flwang1AJaeger: sdague: anteaya: can you please help approve this tiny patch https://review.openstack.org/#/c/248490/ ?17:17
jeblairfungi: oh, periodic -- that's fine.  nm.  :)17:18
anteayastevemar: I am happy to know that the OSC gate is still passing, thank you17:18
*** dzamboni has joined #openstack-infra17:18
*** sdake has joined #openstack-infra17:21
openstackgerritMonty Taylor proposed openstack-infra/system-config: Manage clouds.yaml on puppetmaster with puppet  https://review.openstack.org/24992817:22
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add shade and openstack inventory to system  https://review.openstack.org/24992917:22
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use OpenStack for inventory instead of puppet  https://review.openstack.org/24993017:22
mordredjeblair, nibalizer, fungi: ^^ there are the steps to get clouds.yaml files, openstack inventory and manual disabling of hosts overrides in place17:22
*** ivar-lazzaro has joined #openstack-infra17:22
SpamapSfungi: mtreinish and I have been discussing the subunit2sql migration over in -qa...17:22
*** olaph has joined #openstack-infra17:23
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Adjust conditions when enable_snat is specified  https://review.openstack.org/24993117:23
*** shashank_hegde has joined #openstack-infra17:23
SpamapSfungi: the resized instance isn't being used properly17:23
AJaegerjeblair: thanks for the approval!17:23
mordredShrews: heya. https://review.openstack.org/#/c/249868/ if you get a chance?17:23
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Adjust conditions when enable_snat is specified  https://review.openstack.org/24993117:23
SpamapSfungi: basically, mysql is only using 4GB of that 64GB17:23
*** ivar-laz_ has joined #openstack-infra17:24
mordredSpamapS: w00t!17:24
zaromorning17:25
SpamapSoh we can also raise join_buffer_size much higher17:25
SpamapSthat should help a bit17:25
*** sergmelikyan has quit IRC17:25
SpamapSmtreinish: ^17:25
*** FallenPegasus has joined #openstack-infra17:25
mtreinishSpamapS: ok cool17:26
mtreinishSpamapS: fwiw, I know practically nothing about mysql tuning params :)17:26
SpamapSmordred: I appreciated that mtreinish thought trove would actually re-tune mysql to use that memory. ;)17:26
SpamapSbut it made me chuckle :)17:26
Shrewsmordred: do you really want logger.error() there and not log.error() ?17:26
mtreinishSpamapS: it's supposed to do magical things automagically for users isn't that the point of something like trove :)17:26
anteayafungi jeblair can you take a look at this change to the template for the third party systems wikipage please? https://wiki.openstack.org/w/index.php?title=Template:ThirdPartySystemInfo&diff=0&oldid=5612417:27
fungiSpamapS: mtreinish: sorry, multiplexing a meeting and lunch17:27
*** rossella_s has quit IRC17:27
Shrewsmordred: oh, nm17:27
*** ivar-lazzaro has quit IRC17:27
anteayathis was not discussed with me and I didn't ask for this17:27
Shrewsmordred: missed the parameter17:27
fungiSpamapS: mtreinish: you need me to kill the subunit2sql-db-manage process on logstash.o.o?17:27
anteayafungi jeblair can I have your opinion?17:27
SpamapSmtreinish: neither do we, but we've seen it done before ;-)17:27
jeblairovh just doubled our capacity into BHS-117:27
mordredjeblair: woot!17:27
jeblairmordred: which suggests ovh has a bhs-1 :)17:27
*** eil397 has joined #openstack-infra17:27
mordredthey do indeed17:28
jeblairmordred does http://docs.openstack.org/developer/os-client-config/vendor-support.html need updating?17:28
anteayafungi jeblair and I don't know who tbp is17:28
SpamapSfungi: we'll also need you to tune up some variables.17:28
mtreinishfungi: yeah, we probably should kill it, delete the temp tables, and then retune the db17:28
mtreinishthen manually trigger the migration again17:28
mordredjeblair: oh! so it does17:28
mtreinishfungi: any infra-root can help with it though if you're busy17:28
fungimtreinish: i'm probably out of my depth without some precise commands you want run/output you want checked anyway17:29
openstackgerritErno Kuvaja proposed openstack/requirements: Revert "Rename semantic-version dep to semantic_version"  https://review.openstack.org/24993617:29
SpamapStuning should be:    innodb_buffer_pool_size = 51G17:29
mtreinishfungi: then mordred sounds like a good candidate :)17:29
SpamapSjoin_buffer_size = 8M17:29
anteayaasselin: do you know who made this change to the third party systems wiki template? https://wiki.openstack.org/w/index.php?title=Template:ThirdPartySystemInfo&diff=0&oldid=5612417:29
*** ihrachys has joined #openstack-infra17:29
SpamapSinnodb_log_buffer_size = 256M17:30
mordredjeblair: can you confirm that the region is "BHS1" and not "BHS-1"? I have no '-' in the other regions17:30
asselinanteaya, that's probably a result of today's cinder meeting17:30
asselinanteaya, I'd ask smcginnis17:30
*** tjones has joined #openstack-infra17:30
*** nelsnelson has joined #openstack-infra17:30
jeblairmordred: yeah, i'll do that and propose the change17:30
smcginnisanteaya: What'd I do?17:30
SpamapSfungi: these have to be changed in trove17:30
*** david-lyle has joined #openstack-infra17:30
mordredjeblair: cool. thanks!17:30
anteayasmcginnis: hi17:30
SpamapSmordred: ^ want to help?17:30
anteayaany idea who made this change to the third party systems wiki template? https://wiki.openstack.org/w/index.php?title=Template:ThirdPartySystemInfo&diff=0&oldid=5612417:30
jeblairSpamapS, fungi, mordred: sounds like maybe we have to make a new profile for this database... i think that was a gui clicky thing before17:31
mordredit's possible with trove client17:31
*** nelsnelson has quit IRC17:31
smcginnisanteaya: That was me. I can remove it.17:31
jeblairanteaya: i have looked at that, and i don't know anything about it and i'm not enamored with it17:31
jeblairmordred: cool17:31
*** nelsnelson has joined #openstack-infra17:31
*** nelsnelson has quit IRC17:31
anteayajeblair: thanks for looking and sharing your opinion17:31
anteayasmcginnis: can I have some context?17:31
smcginnisanteaya: There's been some confusion of what to do for each CI other than the big hammer of "recheck".17:31
smcginnisanteaya: So I wanted to capture some of that there.17:32
anteayasmcginnis: that has been going on since the dawn of the third party cis17:32
*** nelsnelson has joined #openstack-infra17:32
smcginnisanteaya: But it was pointed out we actually have several documentation locations for CIs, so we're consolidating a bit.17:32
smcginnisanteaya: Yeah. :)17:32
openstackgerritMerged openstack-infra/project-config: Add missing env variable for zaqar functional test  https://review.openstack.org/24849017:32
anteayasmcginnis: well the thing is that template is what infra expects each ci to provide, at the project level it is really hard to get agreement17:32
mtreinishSpamapS, mordred: so we need to kill the migration on logstash.o.o, use troveclient to make the config changes SpamapS listed, run http://paste.openstack.org/show/480016/, and then start the migration again17:32
*** Swami has quit IRC17:33
anteayasmcginnis: I have no say in what cinder decides it wants to do17:33
SpamapSoh another one17:33
anteayasmcginnis: but that template affects all systems from an infra perspective17:33
flwang1anteaya: AJaeger: thanks for the approval :)17:33
*** dtantsur is now known as dtantsur|afk17:33
mtreinishSpamapS: let's throw this all up on the etherpad: https://etherpad.openstack.org/p/the-big-subunit2sql-migration17:33
anteayasmcginnis: I'm not saying we can't update the template if need be17:33
SpamapShm no.. we don't need innodb_locks_unsafe_for_binlog .. that won't actually help17:33
smcginnisanteaya: Is that programatically parsed?17:33
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Gnocchi: Fix mapping between of stable branches  https://review.openstack.org/24805017:33
fungianteaya: no clue who tpb is, but you can just roll it back17:33
openstackgerritgreghaynes proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463917:33
anteayafungi: thanks for looking17:33
anteayasmcginnis: no, I created the template17:34
anteayasmcginnis: but it is what infra says we expect from systems17:34
fungialso we were pushing for third-party ci systems to not all invent their own individual recheck comment formats anyway17:34
anteayasmcginnis: so having each project alter that causes problems for me17:34
anteayafungi: yes17:34
fungiso it seems counterproductive for the template to make it look like we want that17:34
jeblairmordred, notmorgan: is there a way to get a list of regions?17:34
*** HeOS has quit IRC17:34
smcginnisanteaya: Hmm, it was also pointed out this is what infra expects: https://wiki.openstack.org/wiki/ThirdPartySystems17:34
zaronotmorgan: which review did you update for the apache [L] config?17:34
anteayafungi: agreed17:35
smcginnisanteaya: And that is per-CI specifics, so we decided to move to there.17:35
anteayasmcginnis: wonderful glad to hear that17:35
smcginnisanteaya: I'm in the process of moving over the information captured in the new column to those pages.17:35
openstackgerritgreghaynes proposed openstack/diskimage-builder: Add centos7 test  https://review.openstack.org/19252117:35
smcginnisanteaya: Then I can remove that column again.17:35
mordredjeblair: no17:35
mordredjeblair: BUT17:35
mordredjeblair: you can print the catalog and look at it17:35
fungianteaya: also, updating wiki pages, especially templates, without leaving a comment as to why they were updated is a terrible habit. please leave a change comment when you roll it back stating why it was undone so that it's hopefully less likely to get redone17:35
anteayasmcginnis: what I am saying is that changing that template affects all ci systems, not just cinders17:35
mtreinishSpamapS: do the instructions I put on the bottom of the etherpad look correct?17:36
jeblairmordred: how's that accomplished?17:36
anteayafungi: I will do so, thank you17:36
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a dashboard page for boards and worklists  https://review.openstack.org/23197317:36
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add buttons for creating new boards and worklists  https://review.openstack.org/23197217:36
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Update $resource wrappers to use new permissions API  https://review.openstack.org/23547517:36
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add UI for complex permissions on boards and worklists  https://review.openstack.org/23547617:36
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Add a page to view a board  https://review.openstack.org/23197117:36
smcginnisanteaya: Right. Just an optional column. I'll remove it if those still using that page don't care to know that info.17:36
fungianteaya: i say that because whoever made that change left no comment as to why they thought it was necessary17:36
anteayasmcginnis: like all things, the template can be assessed and updated if need be, but having a conversation with infra is the best door to have that conversation17:36
* fungi tries to _always_ leave a descriptive comment when updating a wiki page/resource17:36
fungias if it were a git commit17:37
jeblairmordred: https://review.openstack.org/24994417:37
mordredjeblair: "openstack catalog list"17:37
anteayafungi: yes, thank you I too try to leave comments on wikipage changes17:37
*** baoli has quit IRC17:37
jeblairmordred: does that need to land+release before we can nodepool it?17:37
SpamapSmtreinish: reading17:37
anteayasmcginnis: the point is that the process to change that template is to discuss it with infra17:37
mordredjeblair: yup. but I'll do that right now17:38
smcginnisanteaya: OK, next time I update a shared wiki I will file a spec for it. :)17:38
smcginnisanteaya: Kidding. Mostly. :)17:38
*** unicell1 has quit IRC17:38
anteayasmcginnis: no, please don't think I am saying that17:38
SpamapSmtreinish: hey we don't care if we lose data on crash do we?17:38
anteayasmcginnis: can you comprehend how much work the third party space is for infra?17:38
mtreinishSpamapS: we haven't been adding any data since yesterday afternoon17:38
SpamapSmtreinish: I mean in general17:38
anteayasmcginnis: can you understand how hard I work to find a balance, even when it mostly looks like i am failing?17:38
smcginnisanteaya: I do. And sorry, I shouldn't be so flipant about my response. But like I've said a few times now, I will remove it.17:39
SpamapSoh you know what, that won't help here17:39
anteayasmcginnis: I'm trying to find a way to include you whilst maintaing the integrity of what we already have17:39
SpamapSmtreinish: n/m17:39
mtreinishSpamapS: heh, ok17:39
SpamapSmtreinish: I'm tuning in my head for constant TPS, this is ONE BIG TRANSACTIOn17:39
*** sshnaidm has quit IRC17:39
anteayasmcginnis: okay thank you, please add a comment when you undo it17:39
jeblairmordred: k.  also, i'm assuming bhs is beauharnois and not, as it's usually known around here, berkeley high school.  ;)17:40
SpamapSmtreinish: ok yeah those three should make this one transaction go a lot faster.17:40
anteayasmcginnis: and then please add a topic to next week's infra agenda and we can discuss it there17:40
anteayasmcginnis: are you comfortable with that?17:40
zarosigned8bit: how about helping with some gerrit upstream reviews and testing? https://gerrit-review.googlesource.com/#/c/72470/17:40
mordredjeblair: I'd kind of like to get that verified :)17:40
*** bdemers has quit IRC17:40
SpamapSsince we'll get a ton more buffer hits on the read17:40
mtreinishSpamapS: ok, cool17:40
SpamapS(of course then we'll max out write capability of the instance)17:40
jeblairanteaya: more nodes in canada!17:41
anteayajeblair: yay!17:41
notmorganjeblair: yay nodes in canada17:41
smcginnisanteaya: I will leave a comment when I remove it. I don't think I can attend the infra meeting next week. If someone else would like to discuss, that's probably a good thing.17:41
notmorganjeblair: actually yay nodes!17:41
*** Sukhdev has quit IRC17:41
anteayacanada needs more nodes17:41
SpamapSmtreinish: +1 on the instructions17:41
*** kzaitsev_mb has quit IRC17:41
*** Sukhdev has joined #openstack-infra17:41
anteayasmcginnis: okay if someone who understands the motivation for this would like to attend the infra meeting, please have them add a topic with their name and we will listen and do our best to find a way forward17:41
anteayasmcginnis: thanks for understanding17:42
jeblairanteaya, mordred, notmorgan: wow, the OVH datacenter is an above-the-fold item in the beauharnois wiki page: https://en.wikipedia.org/wiki/Beauharnois,_Quebec17:42
anteayaha ha ha17:42
mtreinishSpamapS: ok, cool. Now lets pester mordred :)17:42
nibalizermordred: are the keys in place for 24992817:42
*** hashar has quit IRC17:42
mordredjeblair: nice!17:42
anteayajeblair: I have to go therer17:42
mordrednibalizer: yes17:42
jeblairanteaya: population: 12,011 people; 360,000 servers.17:43
*** BobBall is now known as BobBall_AWOL17:43
*** dramalho has joined #openstack-infra17:43
mtreinishjeblair: do 94% of the servers also speak french?17:44
jeblairmtreinish: and want to secede :)17:44
anteaya30 servers per person17:44
* anteaya packs up for Beauharnois17:44
anteayamtreinish: multilingual17:45
openstackgerritMerged openstack/diskimage-builder: Move install-types doc to user guide  https://review.openstack.org/24529517:45
anteayasmcginnis: also any ci system can put what ever they wish on their own wikipage below the template, they just have to include the template, they are not limited to it17:45
greghaynesmordred: A thought I just had - once we remove hp public cloud I think we can go ahead and merge https://review.openstack.org/#/c/177045/ without the depends-on17:45
greghaynesalthough it would be nice to test if our new clouds that take qcow2's require that compat option17:46
jeblairhttps://www.google.com/maps/@45.3083301,-73.8928948,2382m/data=!3m1!1e3?hl=en17:46
jeblairthe dc is very easy to see from satellite view.  it is a sizable portion of the area.17:46
anteayaand churches17:46
*** changbl has quit IRC17:46
signed8bitzaro: you bet!17:46
smcginnisanteaya: Oh, I thought you were referring to a different page. Looking at your diff now, that was not me.17:46
anteaya5 churches per person in quebec17:46
anteayasmcginnis: okay thank you17:47
anteayasmcginnis: mind if I ask which page you thought I was talking about?17:47
smcginnisanteaya: Though I would still say it's useful information to capture.17:47
smcginnisanteaya: I had added a column to this one: https://wiki.openstack.org/wiki/Documentation/VendorDrivers#Vendor_Driver_Documentation17:48
*** dramalho has quit IRC17:48
smcginnisanteaya: But we have decided to migrate the data off of there so we aren't duplicating this same info so many times.17:48
*** otter768 has joined #openstack-infra17:48
anteayasmcginnis: ah okay I have no idea who looks after that wikipage17:49
anteayasmcginnis: good call17:49
*** flwang1 has quit IRC17:49
anteayaand whoever tbp is has undone their wikipage edit although I still don't know who they are17:49
anteayaso it is going to be tough to accomodate their needs since they haven't yet spoken up17:50
*** Swami has joined #openstack-infra17:50
anteayasmcginnis: always nice to chat with you though17:50
anteayasmcginnis: thanks :)17:50
smcginnisanteaya: Same! Thanks for the head up.17:50
anteayasmcginnis: :)17:50
SpamapSjeblair: hah, I imagine leader election is a contentious subject in that datacenter.17:51
*** tlian has joined #openstack-infra17:51
anteayasmcginnis: and yes there is lots of useful information that cis can provide at their wish, which is why they can add anything they like below the wikipage17:51
openstackgerritAdam Coldrick proposed openstack-infra/storyboard-webclient: Replace "New Story" button in the top bar with a dropdown menu  https://review.openstack.org/23197217:52
anteayas/below the wikipage/below the template on their wikipage17:52
openstackgerritDavanum Srinivas (dims) proposed openstack/requirements: Updated from generate-constraints  https://review.openstack.org/24959017:52
smcginnisanteaya: Yep, that's where I've been putting the info we had captured on the deprecated page. Good place for it I think.17:53
openstackgerritgreghaynes proposed openstack/diskimage-builder: Dont pass xattrs to tar if its unsupported  https://review.openstack.org/17755917:53
fungiSpamapS: if the retuning steps at https://etherpad.openstack.org/p/the-big-subunit2sql-migration are complete and nobody in infra-root is working on adding the custom trove config for those yet, i can add it in a moment17:54
fungimtreinish: ^17:54
openstackgerritgreghaynes proposed openstack/diskimage-builder: Dont pass xattrs to tar if its unsupported  https://review.openstack.org/17755917:54
anteayasmcginnis: agreed17:54
*** mahatic has quit IRC17:54
jeblairfungi: i am not doing that17:54
mtreinishjeblair: heh, at least your clear :)17:55
fungifirst i'm just now getting around to trying to look into whether the new constraints tarball job is breaking neutron stable tarballs17:55
*** mohankumar has quit IRC17:55
*** achanda has quit IRC17:55
SpamapSfungi: standby17:55
SpamapSfungi: we're seeing the migration suddenly go faster.. so its possible the OS level cache is doing enough with the memory to make it worth just letting it go17:56
jeblairmtreinish: heh, yeah, i just meant to indicate i'm not in the middle of working on that so from my pov, fungi can proceed without stepping on my toes.  i did not mean to imply any judgement or unwillingness to do it.  :)17:56
mtreinishjeblair: haha, I'm just giving you a hard time17:56
fungiSpamapS: that would be my preference if so! ;)17:56
SpamapSfungi: mine too17:57
fungialready plenty else for us to fix around here17:57
jeblairmtreinish: troll successful.17:57
*** olaph has quit IRC17:57
anteayaI did not realize ovh had more customers than amazon17:58
fungilifeless: Nakato: AJaeger: bmwiedemann: indeed, i think maybe zuul-cloner isn't being told to checkout the correct branch http://logs.openstack.org/17/17a9f8b67f9ad552f8ee6e1f788b2d44bb7bf044/post/neutron-branch-tarball-constraints/c08ce74/console.html#_2015-11-25_03_58_23_30417:58
fungishould have checked out stable/liberty there17:59
*** achanda has joined #openstack-infra18:00
*** olaph has joined #openstack-infra18:00
*** david-lyle has quit IRC18:00
*** shardy has quit IRC18:00
openstackgerritMonty Taylor proposed openstack/requirements: Bump upper-constraint for os-client-config  https://review.openstack.org/24995318:01
mordredjeblair, dhellmann: ^^18:01
mordredjeblair: that's half of the release request for ya18:01
openstackgerritMerged openstack-infra/shade: Only log errors in exceptions on demand  https://review.openstack.org/24986818:01
*** smarcet has joined #openstack-infra18:01
jeblairfungi: what is a branch-tarball-constraints job?18:02
fungijeblair: it's the branch-tarball job using a pip constraints list18:02
jeblairfungi: how does that affect the tarball output?18:03
*** jordanP has quit IRC18:03
smarcetfungi: morning / afternoon, i will need approval for this patch https://review.openstack.org/#/c/245394/18:03
jeblairfungi: oh, it sets different requirements?18:03
nibalizermordred: for 249929 if we ensure static-inventory with puppet then how can we make manual changes to it18:03
fungijeblair: it installs whatever pip requirements are specified by the constraints list when running sdist/bdist_wheel under tox18:03
fungijeblair: rather than whatever the latest on pypi are18:03
jeblairfungi: hrm, and that makes a difference?18:04
mordrednibalizer: static-inventory is puppet managed18:04
fungiit keeps tarball jobs from breaking if there's a new release of something on pypi which we're not compatible with (and hence haven't been able to update our constraints list)18:04
mordrednibalizer: emergency is the file we make manual changes to18:04
nibalizeroh then the disabled section of static-inventory is how we do it through system-config cool gotcha18:05
mordredyah18:05
*** ssancheztrujillo has joined #openstack-infra18:05
fungijeblair: i'm less worried about the branch-tarball job per se, and more that teh recent constraints migration for neutron jobs may be not doing the right thing for stable branch jobs in general18:05
mordredwell, I need to write some docs there18:05
*** clee_ is now known as clee18:05
*** clee has quit IRC18:05
*** clee has joined #openstack-infra18:05
*** fedexo has quit IRC18:06
mordredbut for the most part things will want to go in [disabled:children] - the only thing that goes into [disabled] is staticly defined hosts (this has to do with how the ansible inventory registers nodes)18:06
mordrednibalizer: but in brief "yes"18:06
*** ildikov has joined #openstack-infra18:06
*** e0ne has joined #openstack-infra18:07
anteayattx thingee if either of you are around, someone asked a question about how to add a project to activity.o.o and I don't know how to respond, reed and two contractors are the core reviewers and reed doesn't know the current status of the contract18:07
anteayattx thingee so if either of you know or could find out the status, I would be grateful18:08
*** kozhukalov_ has quit IRC18:08
*** _ody has joined #openstack-infra18:08
*** isaacb has quit IRC18:09
mordredanteaya: I had forgotten that it exists - and had thought we were sunsetting it18:09
*** unicell has joined #openstack-infra18:10
anteayamordred: oh okay well if we are sunsetting it, have we a date for when it is going offline?18:11
anteayaor have we discussed it at all in a meeting?18:11
anteayaor do we need to?18:11
*** otter768 has quit IRC18:13
*** achanda has quit IRC18:13
mordredanteaya: well, _we_ have never had any part in running it18:14
*** daemontool has joined #openstack-infra18:15
*** electrofelix has quit IRC18:15
mordredanteaya: it's a thing that the foundation paid some people to do18:15
*** ociuhandu has quit IRC18:15
mordredanteaya: and then mirantis wrote stackalytics which is what people actually use18:15
mordredanteaya: so eventually people mostly gave up on the opaque activity thing18:15
fungithe sunsetting was discussed in an infra metrics work session in vancouver18:16
fungiit's just taken us a while to get the stackalytics move into infra completed, which deprecating activity.o.o was dependent on18:17
anteayaah ha18:17
anteayathank you18:17
anteayaso my response to the person who wants to add a project to activity.o.o is use stackalytics18:17
anteayagot it18:17
fungi.o.o ;)18:18
anteayaha ha ha18:18
anteayaelo: hello, I have found an answer for you18:18
jeblairmordred: while we're waiting on the occ release, check out https://review.openstack.org/249513 which is fun18:18
openstackgerritKhai Do proposed openstack-infra/puppet-gerrit: Allow setting the number of threads for indexing Gerrit changes  https://review.openstack.org/24106018:18
anteayaelo: activity.o.o is going away in favour of stackalytics.o.o18:18
anteayaelo: is your project represented on stackalytics.o.o?18:19
jeblairfungi: you might like https://review.openstack.org/249513 too18:19
*** markus_z has quit IRC18:19
fungijeblair: is it possible zuul-cloner needs us to interpret ZUUL_REFNAME when there's no ZUUL_BRANCH set?18:19
mordredjeblair: likewise, https://review.openstack.org/#/c/249898/ and the ones before it would be nice to get in while you're waiting18:19
fungijeblair: i'm wondering if this is the first time we've tried to use zuul-cloner in the post pipeline for a non-default branch18:20
*** david-lyle has joined #openstack-infra18:20
lifelessfungi: oh fun with constraints18:20
lifelessfungi: it will have little actual impact today because (sigh) setup_requires18:20
openstackgerritMonty Taylor proposed openstack/requirements: Bump upper-constraint for os-client-config  https://review.openstack.org/24995318:20
jeblairfungi: maybe, i mean it's a whole other world there.  what should it do if you're updating a tag ref?18:21
lifelessfungi: but we're working on a PEP and thing to fix that upstream, so its worth going through this teething process even for setup.py invocations18:21
*** ihrachys has quit IRC18:21
fungijeblair: good point18:21
lifelessfungi: OTOH if its gnarly I think we could move that one job back to unconstrained18:21
lifelessfungi: (because the only thing should be the setup.py invocation)18:21
*** ihrachys has joined #openstack-infra18:21
fungilifeless: yeah, i think anything we run in post/release pipelines (well, anywhere besides check/gate/experimental) probably needs to not be using zuul-cloner for the moment18:22
jeblairfungi: in short multiple repo use within a post-merge pipeline is undefined behavior at this point.  it sounds like we want to try to define it.  :)18:22
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added additional pypi region hosts.  https://review.openstack.org/24996918:22
*** kozhukalov_ has joined #openstack-infra18:22
*** dmsimard is now known as dmsimard|food18:22
*** hdd has quit IRC18:22
openstackgerritIhar Hrachyshka proposed openstack-infra/project-config: Set ZUUL_BRANCH for periodic unittest bitrot jobs  https://review.openstack.org/24552518:23
ihrachysfungi: ok, so I updated the patch ^18:23
*** markmcd has quit IRC18:23
zaroanteaya: are satisfied with your testing on my gerrit test VM?  I would like to re-use it to test the updated rollback script with that data.18:23
*** achanda has joined #openstack-infra18:24
fungiihrachys: that's also not what i was suggesting. i was suggesting a fix in the script called by tox.ini. do you have an example of where that was attempted and why it didn't work?18:24
*** daemontool has quit IRC18:24
jeblairfungi: zuul-cloner is definitely designed to be run in all pipelines.18:24
fungiihrachys: i wanted to work backward from that to troubleshoot the underlying problem18:24
jeblairfungi: (i think it will perform in the same way as our other options)18:25
ihrachysfungi: which fix? you mean hardcoding ZUUL_BRANCH there for e.g. stable/liberty?18:25
*** markmcd has joined #openstack-infra18:26
anteayazaro: I haven't finished but go ahead and use the updated rollback script since that is what it makes more sense to test anyway18:26
*** hdd has joined #openstack-infra18:26
fungiihrachys: this is likely to get confusing since we're discussing a similar but not necessarily related issue with stable constraints jobs right now18:26
anteayazaro: when you are at 2.11 can you tell me so I have a chance to add some new patches of my own?18:26
*** rguillebert has quit IRC18:26
* notmorgan waves at zaro18:26
*** jcoufal has quit IRC18:27
ihrachysfungi: so speaking of possibility that tox does filter out the variables, I don't think it's the case because see all those variables passed thru at http://logs.openstack.org/periodic-stable/periodic-neutron-lbaas-python27-liberty/7c61ea9/tox/py27-1.log18:27
notmorganzaro: let me know if you're running into other issues on the 2.11 stuff so i can take a look. I couldn't find other issues but you know... edge cases and all that18:27
ihrachyslike JENKINS_HOME and such18:27
zaroanteaya: ok18:27
anteayazaro: thanks18:27
anteayazaro: how long to get it to 2.11 do you think?18:28
anteayazaro: so I am around18:28
openstackgerritMichael Krotscheck proposed openstack-infra/project-config: Added additional pypi region hosts  https://review.openstack.org/24997118:28
anteayazaro: roughly18:28
zaronotmorgan: hi, was wondering if you left a comment about that [L] thing somewhere?18:28
fungiihrachys: in the periodic jobs there's already a BRANCH provided in the environment afaik, and i was suggesting the script in the neutron-*aas repo look for that and set ZUUL_BRANCH to it before invoking zuul-cloner. though it's also possible jeblair has other/better suggestions for using zuul-cloner in periodic stable jobs18:28
notmorganyeah in the review, but i agree with fungi that it wont really be a benefit18:28
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added additional pypi region hosts  https://review.openstack.org/24996918:28
*** moravec has quit IRC18:28
zaroanteaya: i'm guessing it will be tommorow18:28
zaronotmorgan: link?18:28
krotscheckgreghaynes: ^^ additional pypi hosts as dependent patches of yours.18:28
anteayazaro: ah okay, thanks18:28
*** rguillebert has joined #openstack-infra18:28
* krotscheck figured he'd make your life a bit easier and just do the work :)18:29
notmorganhttps://review.openstack.org/#/c/249714/18:29
anteayalifeless: can you add this patch to add the cisco plugin to the requirements repo to your review stack? https://review.openstack.org/#/c/215673/518:29
ihrachysfungi: well we already have that fix in the repos18:29
ihrachysfungi: see e.g. https://review.openstack.org/#/c/243709/18:29
notmorganzaro: it just opens the door for other odd issues by short-circuting out early and might break something as a side effect18:29
jeblairfungi, ihrachys: i believe in periodic devstack jobs we set zuul_cloner if not defined (that's in the jjb job definition)18:29
ihrachysbut BRANCH is NOT set, that's why I try to set it in the periodic job definition18:30
fungijeblair: you mean ZUUL_BRANCH?18:30
jeblairfungi, ihrachys: i believe in periodic devstack jobs we set zuul_branch if not defined (that's in the jjb job definition)18:30
jeblairfungi: yes18:30
notmorganzaro: since we should hit this only once on login, it should be safe to eat the extra processing and it def. wont break anything else by continuing18:30
openstackgerritKhai Do proposed openstack-infra/system-config: Allow Gerrit to use 4 threads for indexing changes.  https://review.openstack.org/24173618:30
ihrachysjeblair: it's not devstack job, it's unittest18:30
jeblairihrachys: i know.  we have not needed this anywhere else up to this point.18:31
jeblairihrachys: because previously, only devstack jobs were multi-repo.18:31
ihrachysjeblair: right. so iiuc you mean that https://review.openstack.org/245525 is the way to go?18:32
jeblairihrachys: absolutely not18:32
ihrachysjeblair: well, I mean the prev revision18:32
ihrachysjeblair: setting BRANCH18:32
jeblairihrachys: zuul_branch should never be set explicitly, except in the job definition for a periodic job18:33
ihrachysjeblair: like that https://review.openstack.org/#/c/245525/3/jenkins/jobs/macros.yaml18:33
*** otter768 has joined #openstack-infra18:33
jeblairihrachys: those macros are used in many jobs, most of them not periodic.18:33
fungiihrachys: i see what i was missing. we export BRANCH={branch} in the branch-git-prep builder macro, but that doesn't carry over to the environment of subsequent macros18:34
ihrachysjeblair: sure. but how is it a bad thing that we pass BRANCH={branch} there?18:34
ihrachysfungi: exactly, it's per macro18:35
*** tjones has quit IRC18:35
fungiihrachys: so this is why we have separate builder macros for periodic devstack-gate jobs18:35
fungioh, wait that's right we don't any longer18:36
fungiwe use a branch override18:36
jeblairmordred, Shrews: those lgtm except I'm being pedantic on https://review.openstack.org/24989818:37
zaronotmorgan: is a replacement rewriterule for the one that's just below it in the file?18:37
zaro*is/is that18:37
notmorganno.18:37
fungiihrachys: so right now we're doing this to get the devstack-gate based jobs to pass along the correct branch information in periodic jobs http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/projects.yaml#n88918:37
notmorganzaro: it very specifically addresses dual leading slashes and has to act outside of normal rewrites, the below rules all look like exclusions (conds) or proxy [p]18:38
notmorganzaro: let me 2x check though?18:38
fungiihrachys: then the builder gets the branch-override and exports an envvar based on that18:38
notmorganzaro: no, the other rewrites are all proxy related to the app-server.18:39
notmorganzaro: so it is not a replacement of any other rule18:39
mordredjeblair, Shrews: yah. I think you're right on the comment. it throws if there was an actual error. I added another pedantic thing18:39
zaronotmorgan: thanks.18:39
notmorganzaro: np18:40
*** ddieterly has joined #openstack-infra18:40
fungiihrachys: we have some similar implementation in the bitrot jobs, but i think we'd still need a separate macro to run the jobs passing along the branch information provided in the job definition. zuul doesn't know a branch to set there because it's not a change-based pipeline18:41
*** ihrachys has quit IRC18:41
*** olaph has quit IRC18:41
fungiseems we've lost ihrachys again18:41
zaronotmorgan: i've just added  those rules on review-dev.o.o .  did not include the [L] though18:41
*** ihrachys has joined #openstack-infra18:41
* notmorgan nods18:42
notmorganthe [L] really should have no impact.18:42
*** yamahata has joined #openstack-infra18:42
*** achanda_ has joined #openstack-infra18:42
jeblairfungi: i think the long term solution is to have zuul timer events include a branch18:42
*** smarcet has quit IRC18:42
*** achanda has quit IRC18:42
notmorganzaro: i see no dual // logging in18:42
zaronotmorgan: it removes '//' from url but doesn't fix the redirects.18:42
fungithat would simplify a lot of our stable branch periodic jobs (which is most of our periodic jobs, i think)18:42
jeblairfungi: (that probably means zuul needs to query branches, but we've taken a step closer to that with the merge check stuff where it queries changes)18:43
*** achanda_ has quit IRC18:43
jeblairfungi: (i believe that was the complication that caused me to omit that in the first place)18:43
fungii guess if we can do that for branches we can do it for ref-oriented pipelines too?18:43
zaronotmorgan: which gets us to a working Gerrit but lost the feature where it redirects users back to the page they were on.18:43
*** olaph has joined #openstack-infra18:44
fungie.g. figure out which branch a given commit is in the history of (and have some sane fallback if the commit we're being asked to test is in the history of more than one branch, which is hopefully extremely rare?)18:44
jeblairfungi: that's still a separate problem -- those come with triggering information, the periodic pipelines only come with information that zuul generates.18:44
jeblairfungi: i think if the problem is that, for ref pipelines, you wnt to operate in the context of a 'branch' then it's probably best to base that off the refname.18:45
notmorganzaro: hm. i think it was doing that... but let me check18:45
fungijeblair: so for the periodics, just have a branch we specify for the job in layout.yaml and let zuul send that along?18:45
jeblairfungi: i am not certain that is the actual problem, that's just my guess from what i've read here.18:45
*** ddieterly has quit IRC18:45
notmorganah no18:45
ihrachysok what's the short term fix? I am sure I would not have time to drive the long term one, especially since I don't really know how infra works in any details18:45
notmorganhm. that might be a bigger issue with the referer being mangled.18:46
fungijeblair: for post the refname will be a branch i think, but for a tag-oriented pipeline that would get screwy for integration tests18:46
jeblairfungi: no, actually have the timer trigger generate an event for every branch, then the pipeline can ignore or act on those as it sees fit, and similarly, the jobs can as well.  both in the normal manner of matching branches.18:46
fungiwell, integration jobs18:46
notmorganzaro: let me spin my example back up and see if i can craft something useful. i am pretty far into the crazy here of mod_rewrite18:46
zaronotmorgan: your workaround is a step forward for us though :)18:46
notmorganzaro: ++ yah. i can prob do the next bit later today18:47
fungiwe can still just decide to punt and assume tags don't get integration jobs18:47
*** devl_ has joined #openstack-infra18:47
notmorganneed to go run an errand18:47
*** jffische has joined #openstack-infra18:47
jeblairfungi: note that currently in the post pipeline we examine the refname to decide where to publish docs18:47
jeblairfungi: we could just as easily do that to pass on extra data to zuul-cloner to have it clone specific things18:47
fungithough i'm not sure if that might be assumed by releases of things like neutron plugins which want to potentially install neutron in the virtualenv when building their release artifacts18:47
jeblairfungi: i'm not certain that we need zuul-cloner to do something automatically in the ref context.18:48
openstackgerritMerged openstack-infra/project-config: Add reno jobs for designate  https://review.openstack.org/24771718:48
notmorganzaro: so yeah at least 1 step forward :)18:48
notmorganyay18:48
*** ZZelle_ has joined #openstack-infra18:48
notmorganwill dig more on the redirect issue next18:48
jeblairfungi: why does a post job need to clone something else anyway?  why doesn't it pip install what it needs?18:49
jeblair(i think that is the real reason this has not come up yet)18:49
ihrachysjeblair: neutron does not publish to pypi18:49
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Add an API endpoint for boards  https://review.openstack.org/22226818:49
zaronotmorgan: cool, i've updated commented on your propsed change.18:49
fungijeblair: for the {name}-branch-tarball-constraints job, it's that it wants the upper-constraints.txt file from the corresponding branch of openstack/requirements18:50
ihrachysjeblair: there is an effort to extract common pieces into neutron-lib, publish it to pypi and make projects depend on it instead of neutron, but that's not where we are right now18:50
fungibecause that file is not in the repo being packaged18:50
zarodimtruck: check out https://review.openstack.org/#/c/249714/18:51
AJaegerfungi, we should only have -constraints jobs for liberty and master - do we have this running only there?18:51
*** jffische has left #openstack-infra18:51
*** glauco__ has quit IRC18:51
*** trown|lunch is now known as trown18:51
dimtruckoh nice!  looking!18:51
fungiAJaeger: yeah, but it broke the neutron-stable-liberty.tar.gz build because it defaulted to checking out master18:52
AJaegerunderstood, fungi18:52
jeblairfungi: that sounds really constraints-job-specific.  i'd suggest that the best way to resolve that would be to have the constraints tarball job examine refname and instruct zuul-cloner accordingly, much in the way the docs publishing jobs change their behavior based on refname.18:52
*** rbrndt has quit IRC18:52
*** aviau has joined #openstack-infra18:53
fungijeblair: yep, that is probably the safest solution18:53
dimtruckzaro: you mentioned this already but the redirect to patch view doesn't work :(18:54
fungijeblair: just hoping to avoid ending up with something too hacky, since the goal is for these constraints-based jobs to replace all the equivalent tox-based jobs (for projects following openstack/requirements anyway)18:54
openstackgerritMerged openstack-infra/shade: Stop using nova client in test_compute  https://review.openstack.org/24944318:54
jeblairfungi: i'm not in principle opposed to establishing a refname equivalency across repos in the way we do branches, but i do think that should take a lot of thought because it might end up being surprising behavior in some circumstances.18:54
dimtruckand pretty sure we had a similar solution before?  at least that's what i groked from the bug thread...18:55
fungijeblair: in the constraints jobs, we can probably just export ZUUL_BRANCH=${ZUUL_BRANCH:-$ZUUL_REFNAME} (or whatever the bash syntax for that is)18:56
*** ihrachys has quit IRC18:56
jeblairfungi: i'm really weirded out by the fact that apparently we're moving toward not being able to build our release tarballs with a simple "git clone; tox -e venv python sdist".  i guess that ship sailed when i wasn't looking though.18:56
jeblairfungi: i really don't think anything should ever set ZUUL_BRANCH.  the only exception to that is for periodic jobs, and that's only until zuul itself sets it.18:57
*** ihrachys has joined #openstack-infra18:57
*** ociuhandu has joined #openstack-infra18:57
*** FallenPegasus is now known as MarkAtwood18:57
fungijeblair: oh, so stuff it into the clonemap/z-c command line?18:57
jeblairfungi: zuul-cloner allows you to override what you need to override without setting ZUUL_ env variables18:57
jeblairfungi: yeah18:57
fungiright, that18:57
*** nadya_ has quit IRC18:57
fungican do18:57
*** e0ne has quit IRC18:58
fungiand yeah, i'm not sold on needing the constraints file from another repo to properly build an sdist or wheel, but this is apparently the sanest workaround for pip shortcomings at our scale of dependency sprawl18:58
*** sridhar_ram has joined #openstack-infra18:59
zarodimtruck: sorry if i'm redundant.18:59
dimtrucknot sure i understand how you're redundant zaro :) if anything, i'm just parroting your findings after double checking haha19:00
dimtrucki've been hacking on the rules last night as well.  i have someone else on stand-by to tag once i bang my head against the wall enough times...19:01
jeblairbandersnatch on nyj01 is synced19:01
openstackgerritMerged openstack-infra/shade: Clean up compute functional tests  https://review.openstack.org/24944419:01
fungijeblair: so from a logic perspective, if $ZUUL_BRANCH is empty then we want to pass --branch $ZUUL_REF or $ZUUL_REFNAME (both seem to be "stable/liberty" on the job in question)19:02
*** sergmelikyan has joined #openstack-infra19:02
fungicurious which is more correct19:02
fungii assume refname19:02
*** amuller_afk is now known as amuller19:03
jeblairfungi: yeah.  refname is the thing that comes from gerrit and is exclusive with branch.  zuul_ref is always present and might be 'refs/zuul/branch/Zsomething' or might be 'branch'.19:03
jeblairfungi: but since you're writing logic, i'd go with refname for clarity and extra assertiveness.19:04
zarodimtruck: i assume you're able to repro on your local env?19:04
dimtruckzaro: yes i am19:05
dimtruckboth on local and review-dev19:05
*** dizquierdo has quit IRC19:05
dimtrucki didn't remove the patch that broke openid19:05
dimtruckjust straight stable-2.1119:06
openstackgerritMerged openstack-infra/storyboard: Add an API endpoint for boards  https://review.openstack.org/22226819:06
fungijeblair: thanks for confirming19:06
*** ashleighfarnham has quit IRC19:07
AJaegerjeblair: is your patch to go to 79 OVH nodes ready to merge? Then I'll +2A...19:07
jeblairAJaeger: yep.  it even has a +1 from someone from ovh :)19:08
nibalizerexciting!19:08
openstackgerritAdam Coldrick proposed openstack-infra/storyboard: Implement complex permissions for worklists and boards  https://review.openstack.org/23547219:09
AJaegergreat, jeblair !19:09
openstackgerritMerged openstack-infra/shade: Fix resource leak in test_compute  https://review.openstack.org/24945219:10
*** Piet has quit IRC19:10
openstackgerritAnita Kuno proposed openstack-infra/system-config: Add openstack-meeting-cp to statusbot  https://review.openstack.org/24998419:10
AJaegerjeblair: your patch to print templates has now three +2s. Shall I approve it now?19:10
openstackgerritSpencer Krum proposed openstack-infra/ansible-puppet: Conditionally manage puppet config files  https://review.openstack.org/24339919:10
* AJaeger wasn't sure whether you wanted to babysit it...19:11
jeblairAJaeger: yeah, would be good to get that in before it changes too much.19:11
jeblairAJaeger: hopefully it won't timeout, but we should keep an eye on it.19:11
* AJaeger approves19:11
*** dmsimard|food is now known as dmsimard19:13
openstackgerritMerged openstack-infra/project-config: Increase OVH to 79 nodes  https://review.openstack.org/24951419:14
AJaegerfungi, jeblair: Looking at constraints, here's an open review: https://review.openstack.org/24852719:15
*** ddieterly has joined #openstack-infra19:17
*** otter768 has quit IRC19:17
*** dimtruck is now known as zz_dimtruck19:18
openstackgerritMerged openstack-infra/project-config: Print the JJB template name in devstack jobs  https://review.openstack.org/24726419:19
*** salv-orl_ has joined #openstack-infra19:19
*** otter768 has joined #openstack-infra19:19
*** signed8bit is now known as signed8bit_ZZZzz19:20
*** signed8bit_ZZZzz is now known as signed8bit19:20
*** salv-orlando has quit IRC19:22
openstackgerritJeremy Stanley proposed openstack-infra/project-config: Make zuul-git-prep-upper-constraints safe for post  https://review.openstack.org/24998619:22
fungijeblair: lifeless: Nakato: bmwiedemann: AJaeger: ^19:23
lifelessanteaya: done19:23
*** glauco has joined #openstack-infra19:23
anteayalifeless: thank you19:23
*** rbrndt has joined #openstack-infra19:24
crinklegreghaynes: i think the 8021q module needs to be loaded so vlans can work, where should that happen?19:25
AJaegerthanks ,fungi!19:25
*** rbrndt has quit IRC19:25
rcarrillocruzcrinkle: i think installing the vlan package sorts that out19:26
rcarrillocruzthis is what we have for dib_packages on east bifrost global vars:19:26
rcarrillocruzdib_packages: "openssh-server,vlan,vim,less,bridge-utils,language-pack-en,iputils-ping"19:26
rcarrillocruzyou have that as well19:26
rcarrillocruz?19:26
*** EricGonczer_ has joined #openstack-infra19:26
greghaynescrinkle: that should get loaded when we specify a vlan in eni, is the vlan package not installed?19:27
anteayafungi: do I need to wait on anything to approve 24998619:28
crinklercarrillocruz: greghaynes i have the vlan package installed19:28
*** EricGonc_ has joined #openstack-infra19:28
rcarrillocruzk, we didn't have to put anything on bifrost or any element to explicitly install 802.1q19:29
* zaro doing errands, bbl19:29
greghaynescrinkle: what is making you think it is a problem with the module loading?19:29
anteayazaro: enjoy errands19:29
*** otter768 has quit IRC19:29
crinklegreghaynes: http://paste.openstack.org/show/480027/19:30
crinklegreghaynes: modprobe 8021q fixes it, modprobe -r 8021q makes it come back19:30
crinklegreghaynes: http://paste.openstack.org/show/480028/19:31
greghaynescrinkle: OK, so we just need to add 8021q to /etc/modules19:32
*** EricGonczer_ has quit IRC19:32
rcarrillocruzcrinkle: are you ok if I piggyback the super-change of puppet-infracloud and add the utility playbooks we've been using in east? Thinking on putting them on /opt/stack/infra-cloud/playbooks or something along those lines19:33
*** flwang1 has joined #openstack-infra19:33
*** EricGonczer_ has joined #openstack-infra19:33
*** EricGonc_ has quit IRC19:34
crinklercarrillocruz: can you add it in a dependent change?19:34
rcarrillocruzthat too19:34
crinklercarrillocruz: it's pretty close to ready the way it is, if we add a bunch more things it's likely to break and be harder to review19:34
openstackgerritDavid Shrewsbury proposed openstack-infra/shade: Make delete_server() return True/False  https://review.openstack.org/24989819:35
*** sergmelikyan has quit IRC19:35
rcarrillocruzgreghaynes: we have a funny problem with glean init service. The service gets triggered on teh events of local-filesystem and starting network-interface19:35
rcarrillocruzwhich means19:35
*** glauco has quit IRC19:35
smcginnisAnyone know if the ssh gerrit query is broken or has changed?19:35
rcarrillocruzit runs for every physical device that starts on boot19:35
rcarrillocruzBUT19:35
rcarrillocruzthat won't apply for vlans19:35
rcarrillocruzsince, well, they are virtual19:35
smcginnisTrying to get a list of reviews and it's telling me 0.19:35
rcarrillocruztherefore we never get the eni vlan config file on boot19:35
fungianteaya: mostly just wanting to confirm that matched jeblair's suggestion, and seems like it did19:35
anteayasmcginnis: what command are you using19:35
smcginnisanteaya: ssh -p 29418 review.openstack.org gerrit query --format=JSON project:cinder status:open19:36
rcarrillocruzi'm trying to workaround it by putting a dummy element that just runs 'glean' at the end of the init process19:36
anteayafungi: he liked it19:36
fungismcginnis: openstack/cinder19:36
smcginnisfungi: Doh! Thanks! :)19:36
funginp19:36
rcarrillocruzwhich just reads the config drive and writes all interfaces it finds19:36
smcginnisMuch better. (Depending on how you look at it.)19:36
fungismcginnis: the only reason i spotted it is because i make that same mistake constantly19:36
smcginnisfungi: ;)19:36
*** Ala has quit IRC19:36
*** jtrovo has joined #openstack-infra19:37
Shrewsjeblair: updated 249898 with your preferred text19:37
greghaynesrcarrillocruz: Ah, we should be smarter about detecting what vlans to configure based on what phy we are being executed for19:37
smcginnisMust have looked over the syntax 10 times and didn't even register that I wasn't including that prefix.19:37
rcarrillocruzif you have other ideas, let me know...we could also maybe rethink the event trigger condition on the glean init service to cover both use cases (run on physical device AND run after everything finishes, so we get vlans created)19:37
jtrovoclarkb: can you review this one more time https://review.openstack.org/#/c/23554519:37
rcarrillocruzoh19:37
rcarrillocruzso19:37
rcarrillocruzyou thinking19:37
rcarrillocruzon putting logic on cmd.py19:37
rcarrillocruzwhere we create the vlans attahced to the physical one19:38
rcarrillocruzthat can work too...19:38
*** eharney has quit IRC19:38
anteayasmcginnis: that is why I asked to see it rather than have you describe it to me19:38
rcarrillocruzit's another approach , which i prefer to the custom element that re-runs glean again...19:38
greghaynesrcarrillocruz: the alternative would be to change the upstart script to only run once (rather than per interface) and not pass the interface arg to glean19:38
smcginnisanteaya: Hehe, exactly.19:38
greghaynesrcarrillocruz: SpamapS might know why we do the per-interface glean19:38
rcarrillocruzyeah, i suggested that to mordred19:38
*** unicell1 has joined #openstack-infra19:39
rcarrillocruzhe had concerns about hitting races, where we have services expecting network to be up and failing or smth like that19:39
greghaynesI am not crazy about going back to debuggin the upstart ordering which is why I suggested the other thing - it is really annoying to debug19:39
greghaynesah19:39
rcarrillocruzerm yeah19:39
greghaynesyea, that is possible19:39
rcarrillocruzi hate upstart19:39
rcarrillocruzjust saying19:39
rcarrillocruz:D19:39
*** tlian2 has joined #openstack-infra19:39
greghaynesheh, well it is just a lot easier to debug a bit of python we unit test19:39
rcarrillocruz++19:39
*** rbrndt has joined #openstack-infra19:39
openstackgerritAndreas Jaeger proposed openstack-infra/project-config: Print the JJB template name in some jobs  https://review.openstack.org/24999119:40
openstackgerritMerged openstack-infra/project-config: Make zuul-git-prep-upper-constraints safe for post  https://review.openstack.org/24998619:40
openstackgerritMerged openstack-infra/project-config: Make zuul-git-prep-upper-constraints safe for post  https://review.openstack.org/24998619:40
*** nadya_ has joined #openstack-infra19:40
*** unicell has quit IRC19:40
*** tjones has joined #openstack-infra19:41
*** tlian has quit IRC19:41
*** achanda has joined #openstack-infra19:42
*** sergmelikyan has joined #openstack-infra19:42
*** placko has joined #openstack-infra19:42
anteayamoar echo19:42
flwang1anteaya: hi19:44
notmorganzaro: ok back gonna look at things.19:44
*** changbl has joined #openstack-infra19:45
flwang1anteaya: my patch just merged, see https://review.openstack.org/#/c/248490/1/jenkins/jobs/zaqar.yaml19:45
flwang1anteaya: but, after retrigger the gate, seems it doesn't work19:45
flwang1anteaya: does the infra need sometime to get the latest gate script?19:45
AJaegerflwang1: it needs at least an hour...19:47
flwang1anteaya: because in the gate log, i didn't see the env variable http://logs.openstack.org/15/245415/3/check/gate-zaqarclient-dsvm-functional/77f8da7/console.html#_2015-11-25_17_55_10_40619:47
flwang1AJaeger: ah, i see, that make much sense19:47
flwang1i will trigger it again to double check, thanks19:48
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-log_processor: Switch to using the new log_processor project  https://review.openstack.org/24999319:48
anteaya flwang1 wait an hour and then trigger it19:48
anteayaotherwise it still wont' be there19:48
stevemaro/19:48
anteayahello stevemar19:48
stevemarwhere can i file a bug against the job board19:48
anteayastevemar: https://launchpad.net/openstack-org19:49
stevemaranteaya: excellente!19:49
anteayayour spanish is wonderful19:49
anteayathat Canadian keyboard needs more accents though19:49
*** achanda has quit IRC19:50
flwang1anteaya: cool, thanks19:51
stevemaranteaya: excellenté ?19:51
*** sergmelikyan has quit IRC19:52
stevemaranteaya: as always, you have all the answers, bug submitted, thanks!19:52
*** achanda has joined #openstack-infra19:53
*** alivigni has quit IRC19:53
*** achanda has quit IRC19:53
anteayastevemar: welcome19:54
*** gildub has joined #openstack-infra19:55
AJaegerjeblair: did the jjb update work or did puppet run into a timeout after the JJB template merge?19:56
*** abregman has joined #openstack-infra19:56
flwang1anteaya: AJaeger: so does that mean the Depends-On: tag doesn't work for project-config patch?19:57
flwang1i mean if i put another patch depends on a project-config patch19:57
AJaegerflwang1: it works fine in that way that a change will not merge before the dependent is in-19:57
AJaegerflwang1: it works not - meaning: You cannot update a job this way, there's no way to "test" the job change i nthe gate19:58
mordredjeblair: os-client-config 1.12.0 is out - so you can do BHS1 now19:58
*** rfolco_ has quit IRC19:58
*** abregman is now known as abregman|afk19:59
*** ssancheztrujillo has left #openstack-infra19:59
greghaynesianw: huh, so looks like the centos element hits a kernel panic on boot http://logs.openstack.org/39/204639/19/check/gate-dib-dsvm-functests-devstack-trusty/4a15846/console.html#_2015-11-25_18_48_38_88219:59
mordredjeblair: also, if you get a sec, Shrews updated https://review.openstack.org/#/c/249898/19:59
AJaegerpabelanger: do you want to review a new grabyaml change https://review.openstack.org/#/c/249513/ ?20:00
AJaegerI mean grafyaml20:00
*** korzen has quit IRC20:00
flwang1AJaeger: got it, thanks20:01
openstackgerritMerged openstack-infra/project-config: Fix gate-nova-functional-constraints  https://review.openstack.org/24852720:01
flwang1AJaeger: btw, seems now there is a problem with the pypy job, right?20:03
flwang1AJaeger: or any known issue?20:03
*** kzaitsev_mb has joined #openstack-infra20:03
flwang1AJaeger: in zaqar client, we're seeing this http://logs.openstack.org/15/245415/3/check/gate-python-zaqarclient-pypy/f11c94e/console.html#_2015-11-25_17_51_37_48020:04
AJaegerflwang1: please give me dtetails20:04
AJaegerflwang1: can't help with that one20:05
flwang1AJaeger: ok, anybody i can ask?20:07
*** salv-orl_ has quit IRC20:07
*** tjones has quit IRC20:07
*** sergmelikyan has joined #openstack-infra20:08
AJaegerflwang1: just ask here - that's an error building lxml, so others might be able to help20:08
flwang1AJaeger: ok20:08
*** sergmelikyan has quit IRC20:08
flwang1seems it's an error when build the lxml on the node20:09
*** changbl has quit IRC20:09
*** placko has quit IRC20:09
*** ihrachys has quit IRC20:10
anteayastevemar: was eating so my hands were full, not only do I not have all the answers I often feel I don't know anything of any value at all20:10
*** aysyd has quit IRC20:11
anteayastevemar: mostly I listen to people who do have the answers and do my best to remember what they said20:11
anteayastevemar: I characterize my day as chasing after a train that left the station all day long and never catching it20:11
anteayastevemar: if that helps you, then I'm grateful20:11
openstackgerritRuby Loo proposed openstack/requirements: Bump ironic-lib to 0.5.0  https://review.openstack.org/25000220:12
*** ihrachys has joined #openstack-infra20:12
*** nadya_ has quit IRC20:13
*** ihrachys has quit IRC20:14
*** ldnunes has quit IRC20:15
*** ihrachys has joined #openstack-infra20:18
stevemaranteaya: i pretty much go around with a small hose trying to put out these major fires all day long, while saying 'its all good!'20:19
anteayastevemar: your height helps the over all effect20:20
stevemaranteaya: glad you could laugh at your visual20:20
anteayaha ha ha20:21
anteayalaughing at myself is my hobby20:21
*** placko has joined #openstack-infra20:22
*** Swami has quit IRC20:22
*** changbl has joined #openstack-infra20:22
*** tjones has joined #openstack-infra20:24
*** placko has quit IRC20:24
*** placko has joined #openstack-infra20:24
openstackgerritMonty Taylor proposed openstack-infra/system-config: Use OpenStack for inventory instead of puppet  https://review.openstack.org/24993020:24
openstackgerritMonty Taylor proposed openstack-infra/system-config: Manage clouds.yaml on puppetmaster with puppet  https://review.openstack.org/24992820:24
openstackgerritMonty Taylor proposed openstack-infra/system-config: Add shade and openstack inventory to system  https://review.openstack.org/24992920:24
stevemarif you're involved in openstack, you gotta be able to laugh at yourself -- zing!20:25
mordredstevemar: THAT'S NOT FUNNY!!!20:25
stevemarmordred: really? i thought it was funny, and i'm damn hilarious20:26
mordredstevemar: :)20:26
*** placko has quit IRC20:26
* notmorgan knows stevemar is always lauging at me not with me.20:27
*** placko has joined #openstack-infra20:27
stevemarnotmorgan: for you, definitely, everyone else, never20:27
*** HeOS has joined #openstack-infra20:30
openstackgerritMerged openstack-infra/shade: Adjust conditions when enable_snat is specified  https://review.openstack.org/24993120:31
*** ddieterly has quit IRC20:34
*** _ody has quit IRC20:38
ianwgreghaynes: hmmm ... panic or just stops?20:38
*** dizquierdo has joined #openstack-infra20:38
*** sdake has quit IRC20:40
openstackgerritMerged openstack-infra/project-config: Add git haproxy dashboard  https://review.openstack.org/24951320:40
ianwgreghaynes: oh, i see now, sorry.  for some reason that bookmark in the url didn't work20:40
*** baoli has joined #openstack-infra20:40
*** dizquierdo has quit IRC20:40
*** dizquierdo has joined #openstack-infra20:41
ianwOut of memory: Kill process 115 (plymouthd) score 1 or sacrifice child <- oom :(20:41
openstackgerritColleen Murphy proposed openstack/diskimage-builder: Load the 8021q kernel module in simple-init  https://review.openstack.org/25001220:41
ianw32765 pages RAM * 4k / 1024 = 128mb?  is that right?  that seems like a very small instance20:43
*** dzamboni has quit IRC20:43
*** salv-orlando has joined #openstack-infra20:44
*** _ody has joined #openstack-infra20:46
*** ddieterly has joined #openstack-infra20:46
*** rguillebert has quit IRC20:47
mordredianw: I agree with your math20:47
*** _ody has quit IRC20:48
mordredianw: where does it come from?20:48
ianwthat is from the oom output, i'm not familiar with these boot tests so i'm wondering where *they* come from :)20:48
*** _ody has joined #openstack-infra20:49
SpamapSrcarrillocruz: don't worry about upstart. It will be gone in 5 years.20:49
SpamapSrcarrillocruz: you could do 'start on static-networking-up' .. that is when everything listed as 'auto' in /etc/network/interfaces is up20:50
*** ihrachys has quit IRC20:50
SpamapSrcarrillocruz: well, secifically anything in ifquery -a --list20:51
*** baoli_ has joined #openstack-infra20:51
SpamapSrcarrillocruz: if glean adds things to /etc/network/interfaces.d, they will be added to that list.. so if you run per-interface, and you want vlan interfaces which you just added there to get gleaned .. make sure they're marked as 'auto'20:51
openstackgerritMerged openstack-infra/system-config: Expose enable_unbound in openstack_project::server  https://review.openstack.org/24614420:53
*** baoli has quit IRC20:53
*** dprince has quit IRC20:54
*** dtardivel has quit IRC20:57
*** e0ne has joined #openstack-infra20:58
*** abregman|afk has quit IRC20:58
openstackgerritMerged openstack-infra/project-config: Add check-requirements job for evoque-dashboard  https://review.openstack.org/24988420:59
*** dizquierdo has quit IRC21:00
*** otter768 has joined #openstack-infra21:02
flwang1ianw: anteaya: can you help approve this https://review.openstack.org/#/c/248850/ ? thanks21:02
ianwgreghaynes: ok, don't worry about me.  now i see the code isn't in the tree because it's part of 204639 :)  anyway, yeah, that's defaulting to booting 128mb instances, which i'd say is too small.  might bump that up and send your change back through21:02
openstackgerritMerged openstack-infra/project-config: Fix wrong path name for training-guide translations  https://review.openstack.org/24981821:02
*** e0ne has quit IRC21:03
anteayaEmilienM: when I read you commit messages I hear your voice21:05
anteayaflwang1: the most I can commit to is reviewing a patch21:05
anteayaand I can review the patch21:05
*** spzala has quit IRC21:05
openstackgerritIan Wienand proposed openstack/diskimage-builder: Perform a booting test for our images  https://review.openstack.org/20463921:06
*** mtanino has quit IRC21:06
ianwgreghaynes: ^ that is bumped up to 512mb.  512mb should be enough for anyone21:06
flwang1anteaya: thanks :)21:06
*** arxcruz has quit IRC21:07
anteayaflwang1: why does zaqar need a ui21:07
anteayaI'm curious21:07
anteayasorry separate repo for a ui21:07
anteayafolks are creating repos like accessories and I'm not a fan21:08
flwang1anteaya: because horizon doesn't allow to support new project in horizon's tree21:08
flwang1horizon team is even thinking remove sahara and trove from horizon because they don't have the expertise to maintain the code21:08
*** tlian has joined #openstack-infra21:08
flwang1anteaya: so we have to create separate repo to maintain the code21:09
openstackgerritMerged openstack-infra/project-config: zuul: enable beaker jobs for puppet-mistral  https://review.openstack.org/24917921:09
openstackgerritJames E. Blair proposed openstack-infra/system-config: Add BHS1 pypi mirror  https://review.openstack.org/25002521:10
jeblairfungi, mordred: ^ TIL "emacs `git grep -li gra1`"21:10
flwang1anteaya: just like magnum-ui, manila-ui, etc21:11
anteayaflwang1: I'm talking to horizon folks trying to understand this21:11
flwang1anteaya: great21:12
*** tlian2 has quit IRC21:12
jeblair(though i assume there's some sort of (open-all-files-matching-git-grep "gra1") function in emacs)21:12
*** tlian has quit IRC21:15
openstackgerritMikhail S Medvedev proposed openstack-infra/ciwatch: Use zuul gerrit event listener implementation  https://review.openstack.org/24891921:16
*** dizquierdo has joined #openstack-infra21:17
* AJaeger waves good night and wishes everybody in the states a great Thanksgiving!21:19
*** ashleighfarnham has joined #openstack-infra21:20
sc68calAJaeger: night!21:20
AJaegerjeblair: http://grafana.openstack.org/dashboard/db/git-load-balancer is up and shows some "data collection errors"21:21
*** AJaeger has quit IRC21:21
*** ddieterly has quit IRC21:21
*** dims_ has joined #openstack-infra21:21
openstackgerritMerged openstack-infra/irc-meetings: Reset Neutron UPgrades to Weekly  https://review.openstack.org/24958921:22
*** dims has quit IRC21:22
*** _ody has quit IRC21:24
greghaynesianw: ah, good find. Thanks21:24
*** shashank_hegde has quit IRC21:24
*** _ody has joined #openstack-infra21:24
openstackgerritMerged openstack-infra/irc-meetings: Move cross-project meeting to -cp channel  https://review.openstack.org/24970821:26
*** dims has joined #openstack-infra21:26
*** dkranz has quit IRC21:27
*** ayoung has quit IRC21:27
*** dims_ has quit IRC21:28
mordredjeblair: :)21:28
mordredjeblair: I thnk it's M-x open-all-files-matching-git-grep21:29
openstackgerritMerged openstack-infra/project-config: Create Zaqar UI repo  https://review.openstack.org/24885021:30
anteayaflwang1: that also takes about an hour to propigate21:32
flwang1anteaya: awesome, thanks a lot, man21:34
openstackgerritMerged openstack-infra/shade: Make delete_server() return True/False  https://review.openstack.org/24989821:34
*** sergmelikyan has joined #openstack-infra21:35
openstackgerritMerged openstack-infra/irc-meetings: Updates the Nova Hyper-V Team meeting  https://review.openstack.org/24956621:35
anteayaI am female21:35
*** openstackgerrit has quit IRC21:36
*** openstackgerrit has joined #openstack-infra21:37
tonybIf I want to run 2 very trivial checks I assume it's lighter on infra resources to pass in both via envlist rather than create a second jenkins job with, IIUC, will then use a second node from nodepool?21:37
*** signed8bit has quit IRC21:38
openstackgerritMerged openstack-infra/project-config: Add python3 testing to neutron-lib  https://review.openstack.org/24942121:39
*** bswartz has quit IRC21:40
anteayatonyb: hi, what are the checks21:40
tonybanteaya: Oh it's the 2 tox env's in irc-meetings ical (which generates the html/ics and check with well just checks that I don't miss the obvious things21:42
anteayatonyb: sound light, yes, I'd offer the patch with two in one job and see what reviewers think21:44
anteayaI don't have any objection21:44
tonybanteaya: cool.  I'm happy to do them as separate but it seems resource heavy21:45
anteayalet's see what others think as a single job that does both21:45
anteayaI might be wrong21:45
anteayabut I suspect anyone who might disagree is preparing to eat turkey tomorrow21:46
anteayaso a review will give them a shot to register their objections21:46
*** kozhukalov_ has quit IRC21:47
anteayastevemar: ha ha ha21:48
anteayastevemar: the one mid-cycle week and everyone is going21:48
stevemaranteaya: :(21:48
anteayadid anyone besides me not want 27-29?21:48
*** signed8b_ has joined #openstack-infra21:49
stevemaranteaya: 27-29 was the highest vote getter21:49
stevemarfollowed by 20-2321:49
stevemar16/23 people said they could do 27-2921:49
*** signed8b_ has quit IRC21:50
anteayaokay fair21:50
anteayajust tough to promote cross project comms when they all happen the same week21:50
stevemar1 person said no dates work, so that was weird21:50
anteayaha ha ha21:50
*** jzerebecki has joined #openstack-infra21:50
anteayaI want to attend but I don't like any of your choices21:50
stevemarbasically21:50
anteayaha ha ha21:50
anteayatheir place then?21:50
stevemarthere were a few folks that responded with only option that worked21:51
stevemari wonder if they knew they could multi-select...21:51
anteayamaybe not21:51
*** jpeeler has quit IRC21:51
stevemarby a few i mean 2-321:51
anteayayeah21:51
*** thiagop has quit IRC21:51
openstackgerritgordon chung proposed openstack-infra/project-config: fix ceilometerclient gate: enable aodh  https://review.openstack.org/24985321:52
*** cbader has quit IRC21:52
*** baoli_ has quit IRC21:53
*** dims_ has joined #openstack-infra21:53
*** dims has quit IRC21:56
*** Sukhdev has quit IRC21:56
*** placko has quit IRC21:57
*** hashar has joined #openstack-infra21:58
*** kozhukalov_ has joined #openstack-infra21:59
*** shashank_hegde has joined #openstack-infra22:00
*** Sukhdev has joined #openstack-infra22:00
*** shashank_hegde has quit IRC22:00
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add missing relationship from orm models  https://review.openstack.org/25003422:00
*** shashank_hegde has joined #openstack-infra22:01
jheskethMorning22:01
jeblairmordred, mgagne: the single-job runtime on internap is right around (perhaps slightly faster than) that of rackspace.22:04
anteayamorning jhesketh22:04
anteayanice22:04
anteayayay internap22:04
jeblairyeah.  it could go down if we become our own noisy neighbors, but it's a good start.  :)22:04
openstackgerritMerged openstack-infra/irc-meetings: Add meeting_ids to several meetings that were missing them  https://review.openstack.org/24953022:05
mgagnejeblair do you have a UUID for the instance? our plan is to use aggregates to isolate the load. If you happened to land on an empty or inactive node, you probably didn't have noisy neighbors effects22:05
anteayajeblair: yay for good starts22:05
mgagnejeblair that ^22:05
mordredjeblair: excellent22:06
jeblairmgagne: 7872284f-1408-41a7-b83d-4d455faaa35122:06
anteayaso we have ovh and bluebox running and have internap in the testing stages?22:06
mordredyup22:06
anteayaso many hosts now!22:06
*** Daisy has joined #openstack-infra22:06
anteayayay!22:06
*** trown is now known as trown|outttypeww22:07
anteayayay for more hosts22:07
jeblairanteaya: we're still keeping an eye on ovh...22:07
anteayayup, I saw we just increased to 79 nodes today22:07
mgagnejeblair the node is far from being fully used =)22:08
jeblairwe're slowly ramping up so we don't have any surprises (which is what we'll do with internap too, once we get final word on the region and the go ahead to start using it)  <-- mgagne22:08
jeblairmgagne: heh :)22:08
anteayahere is a funny one, when I go to http://status.openstack.org/zuul/ I don't see any graph output for test nodes22:08
anteayajeblair: is graphite happy currently?22:08
jeblairanteaya: shift-reload22:08
mgagnejeblair I'm poking my boss right now22:08
jeblairanteaya: (and you've had that page loaded for a long time, i'm guessing)22:08
anteayathere it is22:08
anteayano just opened it22:08
anteayabut f5 found them22:09
jeblairanteaya: oh, browser must have still cached it22:09
jeblairi changed the metrics for that graph last week i think22:09
anteayaah22:10
anteayachrome it caches things22:10
jeblairfaster that way ;)22:10
anteayayay speed22:10
anteayaor accuracy22:10
anteayatake your pick22:10
*** Daisy has quit IRC22:11
*** otter768 has quit IRC22:12
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add missing relationship from orm models  https://review.openstack.org/25003422:13
*** otter768 has joined #openstack-infra22:13
mgagnejeblair ok, the plan is to create the aggregate very soon with existing hardware. the region is the right one.22:13
mordred\o/22:14
anteayais it region one?22:15
mgagneour region is named nyj0122:15
anteayaoh what a great name for a region22:15
mgagnewell, I guess it's much better than a default value22:16
jeblairi confess to being slightly confused by the name nyj.... is there also an nyy?22:16
openstackgerritMerged openstack/requirements: Revert "Rename semantic-version dep to semantic_version"  https://review.openstack.org/24993622:16
anteayaha ha ha22:18
*** sergmelikyan has quit IRC22:19
mgagneI think it's a contraction of new york new jersey22:19
anteayanew york jersey22:19
*** gildub has quit IRC22:20
anteayais it in the river?22:20
*** rguillebert has joined #openstack-infra22:20
*** pradk has quit IRC22:20
mgagnecould very well be New York Jets :D22:20
jeblairmordred: fqdn/puppetmaster.openstack.org.yaml hiera file looks weird22:21
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud controller node  https://review.openstack.org/20969822:21
jeblairmordred: (shows in git diff)22:21
anteayamgagne: all better than region one22:21
jeblairyeah, we have a regionone already which means we can't have another one :(22:21
jeblairmordred has some patches in progress to change our mirror naming scheme22:22
jeblairto accomodate that22:22
*** Sukhdev has quit IRC22:22
openstackgerritMerged openstack/requirements: Updated from generate-constraints  https://review.openstack.org/24959022:22
anteayayay!22:22
openstackgerritOpenStack Proposal Bot proposed openstack/diskimage-builder: Updated from global requirements  https://review.openstack.org/24759722:23
openstackgerritColleen Murphy proposed openstack-infra/system-config: Add Infra Cloud compute node definition  https://review.openstack.org/21398022:24
pabelangerAwesome to see more grafana dashboards coming online!22:24
jeblairpabelanger: i even got that one _mostly_ right :)22:25
*** sabeen1 has joined #openstack-infra22:25
*** tjones has quit IRC22:25
pabelangerjeblair: Going to start work on adding templating support.  Some pretty neat stuff you can do with it22:25
jeblairi've started rsyncing dfw -> bhs122:25
*** Swami has joined #openstack-infra22:26
jeblairpabelanger: yeah, i wonder if that would have made the git page easier -- lots of copy pasta  (i'm not complaining yaml copy pasta is better than gui clicky copy pasta) [i like my copy pasta al dente]22:26
pabelangerjeblair: Ya, I think it is a good start. But having the ability to toggle which server you want to graph will put the page over the top!22:27
*** amuller has quit IRC22:27
openstackgerritMerged openstack-infra/system-config: Add BHS1 pypi mirror  https://review.openstack.org/25002522:28
openstackgerritJames E. Blair proposed openstack-infra/project-config: Fix legends on some graphs on git dashboard  https://review.openstack.org/25004222:28
*** sabeen3 has joined #openstack-infra22:29
*** sabeen1 has quit IRC22:29
*** jaypipes has quit IRC22:30
anteayajeblair: do you copy and then toss it on the wall before you pasta to make sure it is al dente?22:30
openstackgerritJames E. Blair proposed openstack-infra/project-config: Fix some broken graphs on git dashboard  https://review.openstack.org/25004322:31
jeblairthere, that should fix all the problems i see on that page22:31
jeblairnow i will make brunswick stew22:32
mordredmmm22:32
anteayaI don't know brunswick stew22:33
*** aeng has quit IRC22:33
*** aeng has joined #openstack-infra22:33
mgagnejeblair: we will soon (later tonight) be able to segregate the load. Would you like to be able to test performance with noisy neighbors?22:33
*** david-lyle has quit IRC22:34
anteayalooks like stew with pork?22:34
* anteaya wonders if she is missing something22:34
EmilienManteaya: lol thanks22:35
EmilienManteaya: I like commit messages22:35
jeblairmgagne: i think my experience with ovh suggests that the next step should just be to start pointing a production workload at it by gradually increasing the amount we use22:35
Clintmmm.. brunswick stew22:35
*** otter768 has quit IRC22:35
anteayaEmilienM: I couldn't help it, I don't hear most people's voices when I read their commit message but I heard yours22:36
anteayaEmilienM: and thanks I like reading good commit messages22:36
jeblairmgagne: we can track runtimes and see if it becomes a problem in practice22:36
anteayalooks like you can make brunswick stew with squirrel22:37
jeblairmgagne: (we can simulate the worst-case, but ultimately that's less interesting than the average case that we get with a varied production load)22:37
anteayaanybody tried that?22:37
mgagnejeblair ok, will check with our team22:37
*** ssancheztrujillo has joined #openstack-infra22:37
jeblairanteaya: that's the most traditional meat; it's hard to find out here so i usually sub rabbit22:38
anteayaI can see the necessity of that22:40
*** peristeri has quit IRC22:40
anteayajeblair: do you snare the rabbit yourself?22:40
anteayacould be tricky in oakland22:40
*** moravec has joined #openstack-infra22:42
*** tjones has joined #openstack-infra22:45
*** moravec has quit IRC22:50
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add missing relationship from orm models  https://review.openstack.org/25003422:51
nibalizeranyone have an opinion on https://review.openstack.org/#/c/249009/1 ? i know clark wanted to backup less instead of moving backups22:53
*** dizquierdo has quit IRC22:53
nibalizerclarkb: i think that was your preference anyways22:54
*** kozhukalov_ has quit IRC22:54
*** otter768 has joined #openstack-infra22:56
anteayanibalizer: I haven't seen him all day23:03
anteayanibalizer: he mentioned he had to cook turkey23:04
nibalizerya23:04
nibalizerwell if ask fills up again someone can just fix it23:05
nibalizernot the biggest of deals23:05
anteayaI have no opinion23:05
anteayanibalizer: what is the current status of ask23:05
anteayais it asking?23:05
nibalizeranteaya: yes it is23:05
nibalizerbut per http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=15623:05
nibalizer(thanks greghaynes for diagnosis)23:05
nibalizerwe're low on disk23:05
anteayahmmmm23:06
*** david-lyle has joined #openstack-infra23:07
mgagnejeblair can you make sure you are using the "right" account? there is 2 accounts, one for internal instances and one for nodepool23:07
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Renamed pypi.<>.openstack.org to mirror.<>.openstack.org  https://review.openstack.org/23875423:07
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Clean up references to old pypi mirror urls.  https://review.openstack.org/23875523:07
anteayanibalizer: well when can you be around to ask again?23:08
anteayanibalizer: would be great if you were willing to own it so it doesn't get forgotten23:08
anteayanibalizer: I don't know what to say23:09
*** erlon has quit IRC23:09
nibalizeranteaya: i'm sure we'll get consensus before too long23:09
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Add NodeJS and NPM to single-use-slave  https://review.openstack.org/24216123:09
openstackgerritMichael Krotscheck proposed openstack-infra/system-config: Added NPM Mirror to Infra  https://review.openstack.org/23561723:09
anteayamordred: are you around and do you have an opinion?23:09
anteayajhesketh: same ^^23:09
*** kzaitsev_mb has quit IRC23:09
nibalizerits not an emergency :)23:09
*** kozhukalov_ has joined #openstack-infra23:09
anteayanibalizer: hopefully before ask fills up again23:09
anteayanibalizer: yeah would be great if it didn't become one though23:09
anteayarate of disk usage is steady23:10
anteayawouldn't be nice to fill up again over the weekend23:10
*** kzaitsev_mb has joined #openstack-infra23:10
*** sdake has joined #openstack-infra23:10
*** gildub has joined #openstack-infra23:13
*** shashank_hegde has quit IRC23:13
*** hdd has quit IRC23:14
jheskethanteaya: are we talking about https://review.openstack.org/#/c/249009/1 ?23:15
anteayajhesketh: yes23:15
anteayajhesketh: clarkb wants to backup less rather than moving backups23:16
jheskethI don't see any harm in moving to a cinder volume, but if clarkb has an objection there's likely a good reason23:16
anteayanibalizer: can you note clarkb's preference as a comment on the patch so folks can reply to it?23:16
*** med_ has quit IRC23:17
*** kozhukalov_ has quit IRC23:17
*** akshai has quit IRC23:17
nibalizeranteaya: done23:20
*** akshai has joined #openstack-infra23:20
anteayanibalizer: thank you23:21
openstackgerritMathieu Gagné proposed openstack-infra/jeepyb: Don't create new groups for system ones  https://review.openstack.org/24812623:21
openstackgerritJoshua Hesketh proposed openstack-infra/puppet-pgsql_backup: Backup folder should not be readable by other  https://review.openstack.org/25006023:23
openstackgerritK Jonathan Harker proposed openstack-infra/puppet-log_processor: Switch to using the new log_processor project  https://review.openstack.org/24999323:23
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Update app-catalog commands  https://review.openstack.org/25006123:24
*** david-lyle has quit IRC23:24
*** akshai has quit IRC23:25
*** hashar has quit IRC23:25
*** med_ has joined #openstack-infra23:25
mgagnewow, didn't take much time to get this project split from puppet module :O23:25
mgagnethanks to people that worked on that one23:25
nibalizerya thats long overdue :)23:26
*** Qiming has joined #openstack-infra23:26
mgagne+123:26
mgagnehttps://github.com/openstack-infra/log_processor/blob/master/.gitreview :D23:26
clarkbjhesketh its not a cinder volumr itd an ephemeral drive23:27
openstackgerritMathieu Gagné proposed openstack-infra/log_processor: Update project name in .gitreview  https://review.openstack.org/25006223:27
jheskethclarkb: see, knew you had a good reason ;-)23:27
mgagneoh =( https://review.openstack.org/#/c/248868/ =)23:29
*** otter768 has quit IRC23:29
openstackgerritJoshua Hesketh proposed openstack-infra/system-config: Reduce the number of backups for ask.o.o  https://review.openstack.org/25006423:30
jheskethclarkb, anteaya, nibalizer: ^23:31
*** Qiming has quit IRC23:31
*** Qiming_ has joined #openstack-infra23:31
jhesketh(also this: https://review.openstack.org/250060)23:31
nibalizerjhesketh: +2 and im abandoinging the other 24900923:31
*** ashleighfarnham has quit IRC23:32
openstackgerritMatthew Treinish proposed openstack-infra/subunit2sql: Add seperate page for v1.0.0 release notes  https://review.openstack.org/25006623:33
*** mmedvede has quit IRC23:36
anteayajhesketh: thank you23:36
anteayanibalizer: can you add clarkb's clarification when you abandon 249009?23:37
nibalizersure23:39
*** mmedvede has joined #openstack-infra23:40
*** sabeen3 has quit IRC23:42
*** ZZelle_ has quit IRC23:42
anteayathanks23:42
openstackgerritChristopher Aedo proposed openstack-infra/puppet-apps_site: Update app-catalog commands  https://review.openstack.org/25006123:49
openstackgerritRamy Asselin proposed openstack-infra/elastic-recheck: Make uncategorized_fails.py configurable  https://review.openstack.org/25007223:50
jheskethclarkb, sdague: either of you around and have a few moments to discuss swift log indexes (no worries if not)23:50
clarkbjhesketh I am doing turkey day and hokiday prep so not really around23:54
jheskethyep, figured as such :-)23:54
* jhesketh is also jealous23:54
openstackgerritSachi King proposed openstack-infra/project-config: Tarball-constraints should only run on liberty+  https://review.openstack.org/25007323:54
jheskethclarkb: have a happy thanksgiving :-)23:54
*** Qiming_ has quit IRC23:55
*** sfinucan has quit IRC23:55
*** devl_ has quit IRC23:55
*** tjones has quit IRC23:55
Nakatofungi: AJaeger: In addition to your patch fungi, tarball-constraints shouldn't have even ran on the kilo branch.  ^23:56
*** alexsyip has joined #openstack-infra23:58
*** xyang1 has quit IRC23:59
*** davideagnello has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!