I vote for an intake manifold leak. If it was bad valve seals/guides, you would get the puff of smoke on every start up. But maybe the lifters and pushrods are splashing
oil up around the intake manifold gasket. So a warm restart might be sucking the
oil into the chambers.
Tighten the intake manifold to the proper torque using the correct sequence. If that does not fix the problem you might want to do a compression test (or leak down) to rule out valve stems/guides. Easy test before you take off the intake manifold and replace the gaskets.