When is it time to Rebuild?

If the engine doesn't smoke, doesn't leak or consume oil, doesn't make funny noises, holds good compression in all cylinders, holds good oil pressure, and runs well, it has plenty of life left in it yet regardless of mileage.

If it ain't broke, don't fix it.
If it starts smoking all the time, not just when you first start it, you might have to rebuild the heads, or have a slight chance of ring issues.
Excessive smoking from the crankcase is a sure sign of ring issues, but is hard to notice with the PCV system hooked up.
If you ever see your dipstick not fully seated, push it back into place, then see it unseated again after driving, that is a sign.

The biggies for knowing when the bottom end is out are:
Low oil pressure, or oil pressure that drops when the RPMs rise.
Compression testing that shows more than a 20% variance in results from one cylinder to the next, both wet and dry tests.
Bearing noises