File handlers and subclasses all accept a delay keyword (docs):
class logging.FileHandler(filename, mode='a', encoding=None, delay=False, errors=None)
Returns a new instance of the FileHandler class. The specified file is opened and used as the stream for logging. If mode is not specified, 'a' is used. If encoding is not None, it is used to open the file with that encoding. If delay is true, then file opening is deferred until the first call to emit(). By default, the file grows indefinitely. If errors is specified, it’s used to determine how encoding errors are handled.
However, this option is not plumbed through to basicConfig, so it will always take the default value False in that case. Instead of using the basicConfig convenience, you'll want to configure logging explicitly in order to use this feature.