Self-esteem is not something you have, it is something you nurture.
Self-esteem or self-worth is not a constant; once it's high it does not necessarily stay high. You need to feed it every day. For example one day you walk past a pizza place and you think, "My diet has been excellent recently", so you have a pizza - or two. Maybe also a beer? Why not? Other time you do not work as you planned.
What is wrong, postponing a task for a day?
These things alone don't take much from your thoughts about yourself, maybe tiny 0.1%. However if you continue to do things that are not good for they add up. Then after a while your self-esteem is ruined.
But good news! This works the other way as well. Always make decisions according to what you're really think is right, work on your goals as you planned, stand up to something that requires courage from you and go outside your comfort zone.
All of these actions will raise your self-esteem.