Sphere shape cast causes assert in GetWorldSpaceSurfaceNormal

[SamCZ]

Hi, Im trying to get my ray casts to work and I use sphere cast for my character controller and when it toches the ground I get nans or inf because by screen gets black, also often I get assert crash.

Could you please somebody look at my casting and tell me if this is correct ?

RayCast

bool Physics::RayCast(
	PhysicsContext* physicsContext,
	const Vector3& origin,
	const Vector3& direction,
	RayCastResult& result,
	float maxDistance,
	LayerMask layerMask
)
{
	JPH::RRayCast rayCast(CastVector3(origin), CastVector3(glm::normalize(direction) * maxDistance));
	JPH::RayCastSettings rayCastSettings;
	
	JPH::ClosestHitCollisionCollector<JPH::CastRayCollector> collector;
	collector.ResetEarlyOutFraction(maxDistance);

	physicsContext->JoltPhysicsSystem.GetNarrowPhaseQuery().CastRay(rayCast, rayCastSettings, collector, {}, MaskObjectLayerFilter(layerMask));

	if (collector.HadHit())
	{
		JPH::RayCastResult res = collector.mHit;

		JPH::Vec3 surfaceNormal = collector.GetContext()->GetWorldSpaceSurfaceNormal(res.mSubShapeID2,  rayCast.GetPointOnRay(res.mFraction));

		result.Normal = CastVector3(surfaceNormal);

		if (glm::dot(result.Normal, glm::normalize(direction)) > 0)
		{
			result.Normal = -result.Normal;
		}

		result.Point = CastVector3(rayCast.GetPointOnRay(res.mFraction));
		result.Distance = res.mFraction;

		result.HitEntity = reinterpret_cast<FEntity*>(physicsContext->JoltPhysicsSystem.GetBodyInterface().GetUserData(res.mBodyID));
		result.HitCollider = reinterpret_cast<CComponent*>(physicsContext->JoltPhysicsSystem.GetBodyInterface().GetShape(res.mBodyID)->GetUserData());
		return true;
	}

	return false;
}

Sphere cast

bool Physics::SphereCast(
	PhysicsContext* physicsContext,
	const Vector3& origin,
	float radius,
	const Vector3& direction,
	RayCastResult& result,
	float maxDistance,
	LayerMask layerMask
)
{
	JPH::SphereShapeSettings sphereShapeSettings(radius);

	JPH::RShapeCast shapeCast(sphereShapeSettings.Create().Get(), CastVector3(Vector3(1.0)), JPH::Mat44::sTranslation(CastVector3(origin)), CastVector3(glm::normalize(direction) * maxDistance));
	JPH::ShapeCastSettings rayCastSettings;
	//rayCastSettings.mReturnDeepestPoint = true;
	//rayCastSettings.mUseShrunkenShapeAndConvexRadius = true;
	//rayCastSettings.mReturnDeepestPoint = false;

	JPH::RVec3Arg inBaseOffset(CastVector3(Vector3(0.0f)));
	JPH::ClosestHitCollisionCollector<JPH::CastShapeCollector> collector;
	collector.ResetEarlyOutFraction(maxDistance);


	physicsContext->JoltPhysicsSystem.GetNarrowPhaseQuery().CastShape(shapeCast, rayCastSettings, inBaseOffset, collector, {}, MaskObjectLayerFilter(layerMask));

	if (collector.HadHit())
	{
		JPH::ShapeCastResult shapeCastResult = collector.mHit;

		JPH::ShapeRefC shape = physicsContext->JoltPhysicsSystem.GetBodyInterface().GetShape(shapeCastResult.mBodyID2);

		JPH::Vec3 contactNormal = -shapeCastResult.mPenetrationAxis.NormalizedOr(JPH::Vec3::sZero());
		JPH::Vec3 surfaceNormal = collector.GetContext()->GetWorldSpaceSurfaceNormal(shapeCastResult.mSubShapeID2, shapeCastResult.mContactPointOn2);

		if (contactNormal.Dot(surfaceNormal) < 0.0f)
			surfaceNormal = -surfaceNormal; // Flip surface normal if we're hitting a back face
		if (contactNormal.Dot(JPH::Vec3(0, 1, 0)) > surfaceNormal.Dot(JPH::Vec3(0, 1, 0)))
			surfaceNormal = contactNormal; // Replace surface normal with contact normal if the contact normal is pointing more upwards*/

		result.Point = CastVector3(shapeCastResult.mContactPointOn2);
		result.Normal = CastVector3(surfaceNormal);

		result.Distance = Math::Abs(shapeCastResult.mPenetrationDepth);
		result.HitEntity = reinterpret_cast<FEntity*>(physicsContext->JoltPhysicsSystem.GetBodyInterface().GetUserData(shapeCastResult.mBodyID2));
		result.HitCollider = reinterpret_cast<CComponent*>(physicsContext->JoltPhysicsSystem.GetBodyInterface().GetShape(shapeCastResult.mBodyID2)->GetUserData());
		return true;
	}

	return false;
}

I tried to change surface normal vs contact normal and I get the same result

The assert happens in (JPH::TransformedShape::GetInverseCenterOfMassTransform() const+616)

[jrouwe]

The reason it's crashing is because you're using collector.GetContext(). This value is only valid in the AddHit callbacks and in your case is pointing to freed memory. Use collector.mHit.mBodyID. I've added an extra remark there now.

This code is also not correct:

	JPH::RRayCast rayCast(CastVector3(origin), CastVector3(glm::normalize(direction) * maxDistance));
	JPH::RayCastSettings rayCastSettings;
	
	JPH::ClosestHitCollisionCollector<JPH::CastRayCollector> collector;
	collector.ResetEarlyOutFraction(maxDistance);

You should EITHER specify the length in rayCast.mDirection OR do ResetEarlyOutFraction(maxDistance), not both. Now your ray is maxDistance * maxDistance long.