Why the column order is changing while appending pandas dataframes?

Starting from version 0.23.0, you can prevent the append() method to sort the final appended DataFrame. In your case:

all_data = all_data.append(df, sort=False)

Try this .....

all_data = all_data.append(df)[df.columns.tolist()]