Is padding really needed while making buttons didn't seem to make much difference
It all depends on your design. Padding is used to add spaces around an objects to keep it organized. In other terms, It simply adds spaces between two or more objects.
