Why 'worse than weeds'?

SONNET 94
Shakespeare

They that have power to hurt, and will do none,
That do not do the thing they most do show,
Who, moving others, are themselves as stone,
Unmoved, cold, and to temptation slow:
They rightly do inherit heaven's graces,
And husband nature's riches from expense;
They are lords and owners of their faces,
Others, but stewards of their excellence.
The summer's flower is to the summer sweet,
Though to itself only live and die,
But if that flower with base infection meet,
The basest weed outbraves his dignity:
For sweetest things turn sourest by their deeds;
Lilies that fester smell far worse than weeds.