Background noise, white noise, or EMI, can come from a lot of places. The most common cause is from the electronics within the computer itself. There are several possible solutions to this.
Option 1: Try turning down any boost/mic gain. The lower your gain the less of this noise you will hear. You can compensate by moving the mic closer to (but not in front of) your mouth and ensuring that the mic icon on the tip of the boom just before the pop filter is facing your mouth.
Option 2: Get a USB adapter like ours. This will usually solve the problem by removing the mic input from the onboard audio.
Option 3: Get a dedicated DAC. If you want to really begin enhancing your audio experience, these can be expensive, but may improve both your mic and headphones.