LM358N single supply does not reach GND
LM358 datasheet says it is only able to sink a couple tens of µA when the output is close to 0V (see page 6, output current/sink).
If you check the internal schematics (page 13) this is quite obvious (Q13 can't pull the output below 1 Vbe). This isn't a true rail to rail output. It can only go to 0V if the load is something like a resistor connected to ground, in this case the opamp does not have to sink any current, so it works.
In your schematic, when the output is at 0V, the opamp output will have to sink 1.65V/10kOhm = 165µA which is too much.
You need a true rail to rail opamp, or larger feedback resistor values (which will increase noise due to bias current).
Note: LM358 was introduced in 1972. It is still produced because it works fine in its application domain, and it is very cheap. It's a timeless classic. However, real rail-to-rail opamps that actually work well are a much more recent development. Don't expect it to compare to a 50c modern RRIO opamp... also 50 cents is very expensive compared to the price of LM358...
The LM358 is NOT a rail to rail op amp.
Reference to the schematic shows that there is a limited current source that allows you to get close to the negative rail, but the datasheet shows that you can only get as close as about 200 mV.
To get rail-to-rail you need a FET op-amp.
Read this for more information, it shows the current limitations as you approach the positive and negative rail.
Select a rail-to-rail op-amp such as the MCP6411 which can get you to within a couple of mV of the rails.