All Game Boy consoles had built in noise to their system. The original speakers played this hum/hiss noise, it was just so low you could not really hear it, but put your ear up to the speaker and you will.
One of the side effects of this is when you add an amplifier and a modern speaker capable of better frequency response, you get the noise amplified too.
There are several things you can do depending on the system.
NOTE: The noise is much more present when there is no game inserted. The noise reduces once games are running, even in silent times.
For all Game Boy consoles putting Isopropyl Alcohol on the volume wheel, pushing down with one finger on the wheel and rotating it up and down the volume scale 20 times helps clean the dirt inside the volume wheel.
Another must is to clean the power switches. De-solder the metal lid, remove the plastic power slider, and put IPA in the power switch area, Clean with gently scraping using tweezers, and clean with cotton wool bud.
For the Game Boy Original increasing the capacitance on the main power pins helps a lot. Add a large capacitance (220uF or more) to the main input (top 2 pins of the 4 pin power board wires) or just upgrade to a CleanPower board (which has the added bulk capacitance built in).
For the Game Boy Color, add a 480uF or larger capacitor in place of the existing power capacitor at the bottom right of the board.
Finally, soaking the entire board (excluding PCB) in Isopropyl Alcohol for 10 minutes then scrubbing with a toothbrush and letting air dry for an hour really helps remove a lot of noise.
Removing the noise
I am always working on improving the consoles and products, and I have created a mod called the PureSilence mod. This is currently available for the GBA, and built into the CleanAmp products sold after August 2020. The PureSilence removes all noise from the speaker when the volume wheel is turned down, so you will get pure silence when playing.
I will continue to improve the hardware so eventually all the noise is removed even with the volume up.
Where does the noise come from?
Many people online say "the amp is noisy", even I said that until I reverse engineered the PCB a little more.
The noise comes from different sources on each board. If we removed all noise introduced from dirty PCBs, contacts, volume wheels and insufficient capacitance, there are still several sources of noise.
For the Game Boy Advance I have scoped the issue down to two main things. Firstly the inductor and the switching regulator of the GBA adds some 100Mhz harmonics to the circuit. This adds audio hiss. I plan to fix this by making a CleanPower for the GBA. Secondly and more noticeably the main and true reason for the hum noise on the GBA is the fact the CPU outputs an approximate 8.8khz PWM signal for the encoded audio. The audio is encoded by pulsing out PWM signal at around 8.8kHz. Its duty cycle high period determines the volume of the audio at that point in time.
The PWM is fed through a resistor divider (volume wheel) then into the audio amp. The amp then converts the DC to an AC audio signal at the right levels for the speaker and headphones. However, the PCB as a whole does not fully filter out this 8.8kHz and we are left with a small constant 8.8kHz ripple.
The solution is to filter this out before it reaches the amp, as all the amp is doing is amplifying the ripple. I removed the amp from the circuit and powered it directly with a frequency generator, and when it is fed a clean signal with no noise it does not amplify the noise. So the noise is not due to the amp, but the noise from the signal of the CPU needing filtering better.
When I get time I will create a hardware mod that will fully remove this noise, either directly on the PCB pre-amp input, or make a dynamic filtering circuit to clean it up pre-amp output.