Butter an 8-inch baking dish (or line with parchment paper) and set aside.
In a medium saucepan over medium heat, heat the butter and peanut butter until the mixtures comes to a boil. Remove from the heat.
Add the salt, lavender and vanilla extract, then stir in the sugars until smooth and no lumps remain.
Pour the fudge mixture into the prepared pan and smooth the top. Press a piece of plastic wrap directly on the fudge and refrigerate until set, at least 1 hour.
Cut into squares and serve. Fudge can be stored at cool room temperature in an airtight container.