What prevents an engine from being able to rev past a certain point can be any of several factors. Of the many design criteria of an engine that affects how high it can spin up, the most limiting factor is what determines its redline. Below are some of the most common limiting factors.

But before getting into that it's worth mentioning a few interesting points. Most people imagine redline as being a limit past which the engine may rip itself apart. This is sometimes, though not always, true. What is more common is that the engine is physically capable of spinning faster without any damage, but past redline the torque has fallen off so far that it's just not making power anymore. In other words, redline is often just the optimal shift point for maximum acceleration. You could rev higher but there's no point to it.

To put it differently, very few engines are still developing increasing power when they reach redline. In most engines, the slope of the power curve is already negative before reaching redline. For an engine like this, redline is just the optimum shift point for maximum acceleration.

Now for some of the limiting factors:

The valve train is a common limitation. Past a certain RPM, the valves will float. This means the engine is spinning so fast that before the spring snaps the valve closed, it's already being pushed open again by the cam, so the valve never actually closes. This limit can be increased by using stronger valve springs and lighter valves. But if the springs are too strong it can cause cam wear and power loss. This limit can also be increased by having more valves per cylinder, because then each valve doesn't need as much lift so there is less motion.

The intake is another common limitation. But this is not strictly an RPM limitation, but rather a power limitation. The amount of power the engine develops is directly related to air flow. But if you think of an intake as a snorkel through which the engine breathes, an intake of any given size and geometry can only flow so much air before the intake becomes a restriction. You can increase the volumetric efficiency of the intake, but this means at low RPM the air is at a lower velocity, which can impair low end torque. The ideal is a variable intake geometry to maintain the right balance between velocity and flow. But this is rarely used due to the engineering cost and complexity.

The ignition system is another common limitation. As the engine revs higher, the spark plugs fire more rapidly. This means there is less time in between plug firings for the coil to recharge, which means at some point the spark is going to get weaker. This makes misfires more likely as the RPM climb.

Another common limitation is the speed of the pistons. Every piston travels twice its stroke length for each crankshaft revolution. Thus, for any given RPM, the longer the stroke the faster the pistons are moving. Each piston is constantly being decelerated, stopped, reversed and accelerated. Since kinetic energy is proportional to velocity squared, the energy losses are tiny at idle, but climb rapidly with increasing RPM. Second, faster piston speeds create more heat and friction with the cylinder wall. Lighter pistons can be used. Also, one can shorten the stroke and increase the bore, while maintaining the same overall displacement. The shorter stroke slows down piston speeds which is good. The shorter stroke also means the piston rod connects closer to the center of the crankshaft, providing a shorter moment arm for the force the piston exerts, which is bad. But this is overcome by the fact that the larger bore provides for more mixture and a larger force exerted by the piston. Theoretically, with a larger force but shorter moment arm, torque should be unaffected. Thus the net effect is being able to rev higher without any loss of torque.