Unity box collider not quite working properly on the diagonal

That's quite a mouthful for a title - but something that has chewed up many hours trying to resolve. It'll probably take forever and a day to explain exactly how we ended up at this, suffice to say that after getting our web-based map editor successfully creating maps for our Unity 3D app, we're now busy coding behaviours and game-logic; and it's not exactly been plain sailing!

The long and short of it is, we're using Unity's inbuilt Raycasting to calculate line-of-sight and other obstacle detection routines. The idea is that you select a playing piece, select a destination, and the code will work out if you have an un-obscured path to the destination (if you have, the character walks into the target square, if not, a wakka-wakka-oops sound plays, to let you know that your instruction was rejected as invalid).

But Unity's built in Raycaster is not without it's problems.
Sure, it works fine if you draw a ray from a point and it collides with a large face in your box collider:

Here, we've tried to make our character walk through a door. We draw a ray from the source to the destination squares and then walk through the hit-objects collection, to see if any objects have been tagged as either "wall" or "door" (if we have, we can't walk in the selected direction).

This works just fine. But when we approach our walls from an angle, things start to go a bit screwy. As we approached this opening, we had a free square immediately in front. The squares diagonally above and below this one were free, but partially obscured by a wall:

The Unity RaycastAll correctly returned the wall as an obstacle that couldn't be walked through. But when we selected a target square on the other diagonal, something peculiar happened:

(the direction of the ray has been added to this screenshot)

This time, we can clearly see the ray passing through our box collidor.... but the RaycastAll function returned zero collision objects.

Where things got really weird was when we reversed this ray/vector, and tried to walk back to the square we'd just come from:

This time, the Unity RaycastAll function said that there was a wall in the way! So the wall it didn't recognise when attempting to travel in one direction has suddenly become noticeable, when approached from the other side!

We speculated that this was perhaps because the first time, our ray entered the "side" of the box collider and passed through the "front" of it - whereas when approached from the other side, we're hitting the "front" of the box collider before exiting out of the side.

This theory seemed to hold up because if we increased the width of the box colliders (so that they extended slightly beyond the actual edges of the object) the raycast detect function seemed to work again

Except (although not immediately obvious in the screenshot above) the ray is still entering the "side" of the box collider. It's almost as if the box collider is not a box at all, but a "cross-shaped" intersection of two planes - and by approaching diagonally from the side, if the angle is just right, the ray does not intersect the plane passing down the centre of the box - make it a little bit wider, and suddenly the ray passes through it.

Making over-sized box colliders feels a bit "hacky".
There's just something not right about it.
But it sort-of works for now, so we're going to leave it at that.

We'd much rather understand why it is happening, and take care of it "properly" rather than fudge a way around it. It's like the suggestion that each object should be disabled and re-enabled after re-sizing or re-positioning, because this too can affect the way box colliders work.

We've already done that (disabling then re-enabling the objects, as well as their respective box colliders after positioning them onscreen) but it make no difference.

The hack sort-of works. But we can't help but feel that this issue is going to come back and bite us on the arse sometime in the near future!