Wednesday, 2019-04-24

*** dosaboy has quit IRC00:08
*** dosaboy has joined #openstack-cinder00:09
*** hoonetorg has quit IRC01:01
*** mriedem has quit IRC01:03
*** tejdeep has quit IRC01:07
*** yedongcan has joined #openstack-cinder01:11
*** baojg has joined #openstack-cinder01:12
*** markvoelker has joined #openstack-cinder01:13
*** baojg has quit IRC01:15
*** hoonetorg has joined #openstack-cinder01:15
*** whoami-rajat has joined #openstack-cinder01:16
*** enriquetaso has joined #openstack-cinder01:20
*** baojg has joined #openstack-cinder01:30
*** mvkr has quit IRC01:32
openstackgerrityenai proposed openstack/cinder master: test
openstackgerrityenai proposed openstack/cinder master: Correct the exception msg of ImageUnacceptable
*** gmann_afk is now known as gmann02:16
*** mvkr has joined #openstack-cinder02:21
*** jdillaman has quit IRC02:54
*** tejdeep has joined #openstack-cinder02:57
*** psachin has joined #openstack-cinder03:02
*** whoami-rajat has quit IRC03:35
*** udesale has joined #openstack-cinder03:58
*** whoami-rajat has joined #openstack-cinder04:06
*** tejdeep has quit IRC04:34
*** tejdeep has joined #openstack-cinder04:41
*** tejdeep has quit IRC05:01
*** tejdeep has joined #openstack-cinder05:02
*** sapd1_ has quit IRC05:03
*** e0ne has joined #openstack-cinder05:10
*** markvoelker has quit IRC05:10
*** sapd1 has joined #openstack-cinder05:14
*** e0ne has quit IRC05:17
*** sapd1 has quit IRC05:29
*** sapd1 has joined #openstack-cinder05:37
*** yedongcan has quit IRC05:39
*** mvkr has quit IRC05:40
*** sapd1 has quit IRC05:42
*** Luzi has joined #openstack-cinder05:50
*** mvkr has joined #openstack-cinder05:53
openstackgerritRajat Dhasmana proposed openstack/cinder master: update "auth_url" in document
*** sapd1 has joined #openstack-cinder05:59
openstackgerritpengyuesheng proposed openstack/cinder master: The volume groups return message add project ID
*** sapd1 has quit IRC06:21
*** boxiang has quit IRC06:21
*** boxiang has joined #openstack-cinder06:21
*** pcaruana has joined #openstack-cinder06:24
*** sahid has joined #openstack-cinder06:27
*** sapd1 has joined #openstack-cinder06:27
*** baojg has quit IRC06:28
*** baojg has joined #openstack-cinder06:37
*** Luzi has quit IRC06:50
*** Luzi has joined #openstack-cinder06:57
*** egonzalez has quit IRC07:03
*** egonzalez has joined #openstack-cinder07:04
*** rcernin has quit IRC07:05
openstackgerritMerged openstack/cinder master: Remove docs for --allow-multiattach
*** markvoelker has joined #openstack-cinder07:12
*** sapd1 has quit IRC07:13
*** sapd1 has joined #openstack-cinder07:14
*** enriquetaso has quit IRC07:21
*** tejdeep has quit IRC07:25
*** luksky has joined #openstack-cinder07:26
*** tejdeep has joined #openstack-cinder07:27
*** phasespace has joined #openstack-cinder07:28
*** tejdeep has quit IRC07:36
*** luksky has quit IRC07:42
*** helenafm has joined #openstack-cinder08:04
*** luksky has joined #openstack-cinder08:16
*** e0ne has joined #openstack-cinder09:20
*** sdinescu has quit IRC09:22
*** tkajinam has quit IRC09:52
*** xyang has quit IRC09:54
*** dirk has quit IRC09:54
*** gmann has quit IRC09:54
*** dirk has joined #openstack-cinder09:54
*** gmann has joined #openstack-cinder09:56
*** xyang has joined #openstack-cinder09:57
*** rtjure has joined #openstack-cinder10:18
openstackgerritGorka Eguileor proposed openstack/cinderlib master: Fake unused packages
openstackgerritRajat Dhasmana proposed openstack/cinder master: WIP: Add migrations for default volume type
*** sdinescu has joined #openstack-cinder10:37
*** gmann has quit IRC10:44
*** carloss has joined #openstack-cinder10:56
*** udesale has quit IRC11:16
*** rtjure has quit IRC11:28
*** baojg has quit IRC11:30
*** baojg has joined #openstack-cinder11:31
*** baojg has quit IRC11:35
*** helenafm has quit IRC11:45
*** abishop has joined #openstack-cinder11:55
*** markvoelker has quit IRC12:08
*** udesale has joined #openstack-cinder12:29
*** thgcorrea has joined #openstack-cinder12:37
*** eharney has quit IRC12:50
hemnaso it seems that even if I have nova setup to use ceph as ephemeral storage, the placement api can't find a provider as my host doesn't have enough local disk space12:52
hemnaservice placement] found 0 providers with available 80 DISK_GB12:52
*** jdillaman has joined #openstack-cinder12:53
*** irclogbot_1 has quit IRC12:55
*** irclogbot_0 has joined #openstack-cinder12:55
*** jistr is now known as jistr|afk12:56
*** altlogbot_2 has quit IRC12:57
*** altlogbot_2 has joined #openstack-cinder12:57
*** ybunker has joined #openstack-cinder13:29
*** lseki has joined #openstack-cinder13:32
*** sapd1_x has joined #openstack-cinder13:34
*** seba has joined #openstack-cinder13:38
*** enriquetaso has joined #openstack-cinder13:46
*** eharney has joined #openstack-cinder13:48
*** KeithMnemonic has joined #openstack-cinder13:49
*** gmann has joined #openstack-cinder13:51
*** mriedem has joined #openstack-cinder13:53
*** helenafm has joined #openstack-cinder14:04
*** udesale has quit IRC14:06
*** udesale has joined #openstack-cinder14:07
*** udesale has quit IRC14:08
*** udesale has joined #openstack-cinder14:08
*** phasespace has quit IRC14:26
jungleboyjhemna:  Hmmm.  I don't know that we are providing any data to the placement service.  Or are they doing their own thing to get that data from us?14:26
jungleboyjMorning by the way.14:27
smcginnisWe are not. That was the one thing we identified that we could do with placement.14:31
*** itlinux has quit IRC14:35
jungleboyjsmcginnis:  Ok.  That was what I thought.  I was going to chat with cdent but I haven't seen him online.  Should try e-mail I guess.14:35
smcginnisjungleboyj: I think he said he was going to come to the US a little early and visit family before the Summit.14:36
jungleboyjOk.  Let me see what I can figure out there.14:37
whoami-rajathemna: Hey14:45
hemnafigured out my nova issues w/ help from lyarwood14:46
hemnadevstack evidently ignores nova.conf and uses nova-cpu.conf now14:47
hemnaso nova never saw my ceph setup in [libvirt]14:47
whoami-rajathemna: can you take another look at [1], since we discussed your concern at meeting last time.14:47
jungleboyjhemna: Yikes, that is a big change.14:47
hemnawhoami-rajat: sure14:48
whoami-rajathemna: thanks14:48
*** luksky has quit IRC14:49
*** Luzi has quit IRC14:50
*** tejdeep has joined #openstack-cinder14:50
jungleboyjhemna: So does it appear that placement has basic support for Cinder/Ceph without any of our intervention?14:56
hemnajungleboyj: well, if nova is configured correctly, the placement API sees the space available from ceph correctly14:58
hemnathis wasn't necessarily a cinder issue14:58
hemnabut my devstack node was low on disk space and couldn't fire up 80GB vms14:58
jungleboyjhemna:  Ok, that is good.  But that is just a statement for Ceph Ephemeral?14:58
hemnaso I updated nova.conf to point to ceph for ephemeral and it was still failing14:59
hemnauntil I discovered nova-cpu.conf vs. nova.conf14:59
hemnajungleboyj: yah it was for ceph ephemeral14:59
jungleboyjhemna:  Ok, so that still leave the discussion of getting similar support in place for Cinder storage.14:59
*** lemko has joined #openstack-cinder15:00
hemnafor placement api?15:00
hemnahow is the scheduler going to use the placement api?15:00
jungleboyjhemna:  Correct.15:00
hemnareplacing the scheduler ?15:00
jungleboyjNot the scheduler.15:00
jungleboyjThe placement API being able to use data from Cinder to make placement choices.15:00
* jungleboyj could not be making sense.15:01
hemnahrmm, for nova vms?15:01
*** dave-mccowan has joined #openstack-cinder15:01
hemnaafaik the placement api is for scheduling/placing/locating nova vms15:01
smcginniss/nova vms/resources/15:02
jungleboyjRight, for instance, if placement needs to place a VM and they are doing a boot from volume then I would assume the placement API would need to know what Cinder storage is available in that area.15:02
jungleboyjJust for example.15:02
hemnahow does cinder know what that 'area' is ?15:02
jungleboyjBy AZ?15:02
smcginnisI think that was the idea.15:03
hemnaheh, do we really even support AZ's very well ?15:03
jungleboyjhemna:  Better than we used to.  :-)15:03
hemnalike....a driver/volume type in 1 AZ vs. not in another15:03
smcginnisNot a lot of detail in
hemnain order to support that I think we'd need to be able to associate a volume type against an AZ15:04
hemnato associate a particular backend living in AZ1 vs. not in AZ215:04
jungleboyjhemna:  We have added support for that in Stein I believe.15:05
*** pcaruana has quit IRC15:06
hemnaah ok that's great15:06
*** dave-mccowan has quit IRC15:07
*** amotoki_ is now known as amotoki15:08
hemnawhoami-rajat: so the spec wasn't updated after our discussion15:11
hemnaso I'm not sure what I should do :)15:11
hemnaI thought we agreed on a vtype name of _default_ or something like that15:11
eharney_default_ was suggested15:12
eharneyIMO just going with "default" is better15:13
whoami-rajathemna: i'm not sure, eharney said it was least of our priority as of now15:13
whoami-rajatsince the name is just a constant value to be used during db migration, we can finalize on it anytime, so IMO we can focus on other details (if needs to be updated) in the spec.15:16
jungleboyjeharney:  I thought we wanted to stay away from default as it was more likely that that name might have been previously used.15:16
eharneyjungleboyj: it doesn't matter if it's previously used if it doesn't have extra specs associated w/ it15:16
smcginnisYeah, I've seen clouds use "default" as their default, so that is not a good choice.15:17
eharneyexcept, it's a fine choice, if you do this right15:17
whoami-rajati've used "default_vol_type" in the code
jungleboyjI don't want to bikeshed on this.15:18
jungleboyjI think the goal of having something that is obviously a set default is not so much for code or anything like that but for people who might be debugging the system and can clearly see that it is something we have set that is being used.15:19
jungleboyjThat is my $0.02 there but I would like to move beyond this and get start fixing the issue.15:19
eharneywell, we decided to not bikeshed on it before and it led to "we agreed on" above15:20
smcginnisAs long as we don't use a name that is obvious to cause conflicts, I'm fine with it.15:20
hemnawell, I don't want to ignore this, especially since we are still confused about it.15:20
jungleboyjRight and then you said that 'default' was sufficient.15:20
smcginnis'default' definitely is not a good name and is pretty much guaranteed to cause problems.15:21
jungleboyjThe notes from last week's meeting say that something like __DEFAULT__ should be good.15:21
hemnajungleboyj: ++15:21
eharneyanything is pretty much guaranteed to cause problems if the goal is to just avoid thinking through potential problems.15:22
hemnathat could be a problem15:22
smcginnis"thinking through potential problems" being actually addressing where we have the assumptions that there will be a volume type set?15:23
whoami-rajatso i brought up the spec to move on from the "naming conflict", guess didn't turn out so well.15:25
* jungleboyj face palms15:25
smcginnisI don't think we can approve the spec until it doesn't say we use "default" as the name. If it doesn't specify a name, it should. It's kind of a criticial piece of information.15:26
jungleboyjI don't think using a name that isn't going to conflict is an avoidance tactic, I think it is good practice to make it clear that this is something that was automatically set.15:26
jungleboyjIt also allows others to key off it with other automation if necessary.15:26
whoami-rajateharney: if 'default' doesn't seem to cause issues then using '__DEFAULT__' shouldn't either.15:26
smcginnisAnd it's not bikeshedding. It's deciding whether to build the bikeshed in the middle of your driveway or not.15:27
jungleboyjIf we need to do an upgrade check on it in the future, then we can.15:27
jungleboyjWe still need to think through the technical issues beyond it but we aren't going to get that far if we don't make this decision.15:27
jungleboyjsmcginnis:  Will it be at the beginning or end of the driveway?  ;-)15:28
*** itlinux has joined #openstack-cinder15:28
whoami-rajatsmcginnis: agreed. but i think other specifications also need focus than the name itself?15:30
eharneyanyway, if you a) reuse an existing "default" type if it's safe to use (no specs), and b) just don't migrate volumes on deployments where it's not safe to use, it works fine15:31
eharneyi guess the counter proposal is that you figure that a pre-existing volume type with underscores is less likely to exist, and if it did, some odd things might happen on that deployment15:32
smcginniseharney: So if we have a check to see if 'default' exists and it doesn't have extra specs, then it's safe to use. But if not, then we will have some clouds with the default being 'default' and other clouds with... something other name we decide to pick?15:32
eharneysmcginnis: don't you have to do that either way?15:33
smcginnisNo, we say it's '__DEFAULT__" (or whatever) and have an upgrade check that it doesn't exist and document in the release notes that it is a reserved name.15:33
smcginnisSo we have a very slight chance that an operator needs to change something, versus a 100% chance that at least a few operators will need to change things.15:34
smcginnisAnd get pissed off that the default they've documented for their customers to use is no long the default.15:34
eharneyok, so we need to get some proposal of upgrade checks in the spec15:37
eharneybecause i don't think i've even thought down that route yet15:37
whoami-rajateharney: will do.15:38
whoami-rajatThanks everyone for the inputs.15:39
jungleboyjeharney:  Yeah, I had mentioned that in the meeting last week and I think this is a perfect example for an upgrade check.15:40
jungleboyjSo, we have a way to guarantee that we won't fall down the hole you are concerned we might.15:40
eharneyi was actually leaning more toward it being ok for a few deployments to fall in a hole where we just don't retype the volumes, and make this a more seamless process15:40
eharneymore of a best-effort transition15:41
eharneybut, both work15:41
*** itlinux has quit IRC15:41
whoami-rajateharney:  smcginnis  so if the deployer has created __DEFAULT__ (with no extra specs) should we pass that in upgrade checks?15:42
smcginnisI'd prefer just having a blanket "no type named __DEFAULT__" check, but I suppose we could.15:43
tejdeepCan some one help me with error in Zuul check, tempest-full is complaining about POST_FAILURE tempest results looks fine
smcginnisPOST_FAILURE indicates it happened after tempest completed.15:44
jungleboyjwhoami-rajat:  I would handle that as a warning but maybe not fail the check.15:44
eharneyif we're going w/ the underscore name, i'm not sure there's as much value in pursuing the "reuse the existing type" route15:44
whoami-rajatsmcginnis: that would most probably make the check code easier but the user needs to delete the __DEFAULT__ type he has for the migration to create it again.15:44
eharneythat sounds like a lot of work that would help, like, one deployment somewhere15:44
smcginnistejdeep: This is athe actual failure:
whoami-rajatokay, so the check will fail anyway with the rare case of __DEFAULT__ type existing.15:45
smcginnistejdeep: There were some network issues in our infra cloud providers last week that were causing problems. It could just be a side effect from that.15:45
*** pcaruana has joined #openstack-cinder15:46
tejdeepsmcginnis: Thanks for the update, i didnt get this error in my previous run. should i run recheck again?15:50
smcginnistejdeep: Yeah, I think that was just some weird gate glitch. I think a recheck will likely pass.15:50
jungleboyjGood god.  Lunch at the PTG is 2 hours?15:53
rosmaitajungleboyj: i think because there is no lunch, we have to leave the venue15:54
jungleboyjUgh, I thought that for the PTG they did have lunch.  Just no lunch for the Summit.15:54
rosmaitai could be confused15:54
smcginnisBusiness idea: stop at Costco and get a bunch of bread, peanut butter, and jelly. Set up a sandwich stand in the Summit hallway.15:55
*** itlinux has joined #openstack-cinder15:56
jungleboyjsmcginnis: ++15:58
jungleboyjYes, there is lunch at the PTG.15:58
jungleboyjHmmm, Could I hang a 'Lenovo' sign on that?15:58
*** tejdeep has quit IRC15:59
jungleboyjAnd Ciner of course.15:59
openstackgerritRajat Dhasmana proposed openstack/cinder-specs master: Untyped vol to default vol type
*** helenafm has quit IRC16:00
whoami-rajat^^ no more name conflict :D16:00
*** _erlon_ has joined #openstack-cinder16:03
*** kambiz has quit IRC16:07
*** kambiz has joined #openstack-cinder16:08
*** altlogbot_2 has quit IRC16:09
*** altlogbot_3 has joined #openstack-cinder16:11
*** ianychoi has quit IRC16:14
*** ianychoi has joined #openstack-cinder16:15
*** ybunker has quit IRC16:15
*** ybunker has joined #openstack-cinder16:31
*** itlinux has quit IRC16:33
smcginnisFrom Clark regarding the official Summit party:16:33
smcginnis is the summit pa    rty signup site. password is "denver" and for the phone number you can just put in 555-555-555516:33
smcginnisI hadn't seen anything about that anywhere, but registered just for funsies.16:34
*** itlinux has joined #openstack-cinder16:37
*** altlogbot_3 has quit IRC16:43
*** sapd1_x has quit IRC16:43
*** altlogbot_3 has joined #openstack-cinder16:43
*** itlinux has quit IRC16:44
*** itlinux has joined #openstack-cinder16:50
*** altlogbot_3 has quit IRC16:53
*** tejdeep has joined #openstack-cinder16:53
*** psachin has quit IRC16:53
*** altlogbot_2 has joined #openstack-cinder16:53
*** sahid has quit IRC16:54
*** itlinux has quit IRC16:56
*** luksky has joined #openstack-cinder16:59
*** e0ne has quit IRC17:02
jungleboyjsmcginnis:  I have seen e-mail about it go out if you attended their previous parties and I think it is linked to the Summit Schedule.17:04
jungleboyjI am registered as well.17:04
*** abishop is now known as abishop|afk17:05
jungleboyjSo we can maybe go as a group after the Cinder Dinner.17:05
*** henriqueof has joined #openstack-cinder17:06
*** tejdeep has quit IRC17:07
*** itlinux has joined #openstack-cinder17:10
whoami-rajatsmcginnis: seems like downgrading isn't allowed in the current db migration implementation?17:18
*** markvoelker has joined #openstack-cinder17:22
jungleboyjwhoami-rajat:  COrrect.17:25
*** markvoelker has quit IRC17:26
hemnabe careful, don't move a chair in the PTG!17:29
*** phasespace has joined #openstack-cinder17:30
whoami-rajatjungleboyj:  saw something written related to downgrading stuff, guess needs to correct it.17:31
whoami-rajatjungleboyj: here it is
jungleboyjhemna:  What?17:33
hemnaif you move a chair, or powerstrip at the PTG the labor unions will fine the Foundation.17:34
hemnait's in the email that was just sent out17:34
smcginnisThat's going to be interesting.17:34
jungleboyjI just saw that.  That is stupid.17:34
hemnagotta lovem17:34
smcginnisI hope they have those fines built in to the budget, because it's almost a guarantee.17:34
jungleboyjCan we do it if we are high first?17:34
jungleboyjYeah, that sounds like a money maker.  Especially when they say right there that they don't have enough power strips.17:35
hemnayup, I noticed that too17:35
smcginniswhoami-rajat: We used to support migration downgrades, but there was an OpenStack-wide effort to drop support for it because it could cause some pretty serious issues.17:35
* jungleboyj sees a disaster in the making.17:36
smcginnisI'm not counting on these box lunches either.17:36
smcginnisIf it's like Berlin, I'd rather order Jimmy Johns or something.17:36
jungleboyjsmcginnis:  Yeah.17:36
*** markvoelker has joined #openstack-cinder17:37
jungleboyjI miss the glory days.17:37
hemnaI guess we could cascade 43 power strips to their unmovable strips17:38
whoami-rajatsmcginnis: i see. alembic seems to support that [1] by mentioning the revision17:38
hemnamaybe we should clarify if we are allowed to plug into the power strips...17:38
* jungleboyj can hear Kendall W sighing now.17:39
whoami-rajatsmcginnis: so our 'version' param is unstable if we're migrated to the latest version (which is always?)17:39
smcginniswhoami-rajat: Hmm, I don't think we will want to officially support that once we switch.17:39
smcginnisIt was kind of useful for development where you were OK if you messed up your DB, but dangerous for operators.17:39
smcginnisHere's the spec:
jungleboyjsmcginnis:  ++17:40
jungleboyjWe used to allow the downgrades but no longer.  So, if there is documentation that says it is supported it should be fixed.17:41
*** eharney_ has joined #openstack-cinder17:42
whoami-rajatsmcginnis:  hmm, the current implementation of alembic in openstack-congress allows it, and i'm not sure about other projects that have moved to it (i think glance) but that is for future discussion17:43
jungleboyjOh, you are talking Alembic.17:43
jungleboyjI was thinking SQLAlchemy17:43
whoami-rajatcurrently i faced many issues using the version parameter and don't think it works correctly as described. should we still expose it to the users?17:44
whoami-rajatjungleboyj: ^^this one is for sqlalchemy17:44
*** eharney has quit IRC17:44
*** eharney_ is now known as eharney17:45
jungleboyjOk, for SQLAlchemy the downgrade is not supported.17:45
whoami-rajatjungleboyj: yep but is there still use for ``cinder-manage db sync [VERSION]`` option, IMO it doesn't seem to have any benefit for deployers.17:47
jungleboyjGood question. I suppose it could be used to upgrade to a certain point and no further.  I don't know that people really use that.17:48
*** udesale has quit IRC17:49
hemnaif you only want to upgrade to a certain point17:49
hemnawithout doing all of it17:50
hemnamaybe they want to migrate some large data to another archive table before continuing?17:50
whoami-rajati'm not sure if that is a common usecase in deployment to custom migrate.17:54
whoami-rajatalso specifying any lower version than current migrations throws error "downgrade is not defined in the script", it seems like the error is suggesting user to add a downgrade script.17:54
whoami-rajatfor that migration*17:55
*** tejdeep has joined #openstack-cinder18:00
*** abishop|afk is now known as abishop18:27
*** henriqueof has quit IRC18:31
*** abishop has quit IRC18:31
*** e0ne has joined #openstack-cinder18:48
*** itlinux has quit IRC18:50
*** itlinux has joined #openstack-cinder18:52
*** itlinux has quit IRC18:54
*** openstackgerrit has quit IRC18:57
*** itlinux has joined #openstack-cinder18:58
*** itlinux has quit IRC19:11
*** ybunker has quit IRC19:17
*** _erlon_ has quit IRC19:23
*** eharney has quit IRC19:23
*** e0ne has quit IRC19:32
*** itlinux has joined #openstack-cinder19:33
*** e0ne has joined #openstack-cinder19:35
*** dave-mccowan has joined #openstack-cinder19:56
*** thgcorrea has quit IRC20:05
*** e0ne has quit IRC20:08
*** abishop has joined #openstack-cinder20:13
*** gnufied has quit IRC20:18
*** lemko has quit IRC20:19
*** pcaruana has quit IRC20:39
*** eharney has joined #openstack-cinder20:52
*** tejdeep has quit IRC21:06
*** henriqueof has joined #openstack-cinder21:09
*** itlinux has quit IRC21:16
*** abishop has quit IRC21:19
*** tejdeep has joined #openstack-cinder21:19
*** itlinux has joined #openstack-cinder21:23
*** itlinux has quit IRC21:24
*** whoami-rajat has quit IRC21:25
*** enriquetaso has quit IRC21:26
*** itlinux has joined #openstack-cinder21:28
*** itlinux has quit IRC21:41
*** itlinux has joined #openstack-cinder21:48
*** imacdonn has quit IRC22:01
*** imacdonn has joined #openstack-cinder22:02
*** itlinux has quit IRC22:03
*** itlinux has joined #openstack-cinder22:09
*** itlinux has quit IRC22:15
*** itlinux has joined #openstack-cinder22:25
*** luksky has quit IRC22:31
*** tkajinam has joined #openstack-cinder22:53
*** rcernin has joined #openstack-cinder22:54
*** mvkr has quit IRC23:12
*** mchlumsky has quit IRC23:23
*** itlinux has quit IRC23:33
*** itlinux has joined #openstack-cinder23:36
*** lseki has quit IRC23:40
*** carloss has quit IRC23:45
*** itlinux has quit IRC23:50

Generated by 2.15.3 by Marius Gedminas - find it at!