There is nothing wrong with letting him play. He wont hurt anything as long as he doesn't force anything.
Hell I was 3 years old in 1950 and I took my granddads keys for his 1947 Cadillac off the kitchen table, put them in the ignition, pushed the starter button, stood up on the seat, pulled...