How to limit log file size in python
Lose basicConfig()
and use RotatingFileHandler:
import logging
from logging.handlers import RotatingFileHandler
log_formatter = logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')
logFile = 'C:\\Temp\\log'
my_handler = RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024,
backupCount=2, encoding=None, delay=0)
my_handler.setFormatter(log_formatter)
my_handler.setLevel(logging.INFO)
app_log = logging.getLogger('root')
app_log.setLevel(logging.INFO)
app_log.addHandler(my_handler)
while True:
app_log.info("data")
To use BasicConfig and RotatingFileHandler, add RotatingFileHandler as Handler in BasicConfig.
main.py:
import logging
rfh = logging.handlers.RotatingFileHandler(
filename='foo.log',
mode='a',
maxBytes=5*1024*1024,
backupCount=2,
encoding=None,
delay=0
)
logging.basicConfig(
level=logging.DEBUG,
format="%(asctime)s %(name)-25s %(levelname)-8s %(message)s",
datefmt="%y-%m-%d %H:%M:%S",
handlers=[
rfh
]
)
logger = logging.getLogger('main')
logger.debug("test")
other.py
import logging
class Other():
def __init(self):
self.logger = logging.getLogger('other')
self.logger.info("test2")
"test" will be written into foo.log with the tag 'main'
"test2" will be written into foo.log with the tag 'other'
When you use logging.basicConfig with a file, the log is attached with a file handler to handle writing to the file. afterwards you created another file handler to the same file with logging.handlers.RotatingFileHandler
Now, once a rotate is needed, RotatingFileHandler is trying to remove the old file but it can't becuase there is an open file handler
this can be seen if you look directly at the log file handlers -
import logging
from logging.handlers import RotatingFileHandler
log_name = 'c:\\log.log'
logging.basicConfig(filename=log_name)
log = logging.getLogger()
handler = RotatingFileHandler(log_name, maxBytes=1024, backupCount=1)
log.addHandler(handler)
[<logging.FileHandler object at 0x02AB9B50>, <logging.handlers.RotatingFileHandler object at 0x02AC1D90>]