I would put forward an argument about IQ tests. In theory, if you took IQ tests throughout your life, the number should remain the same. However, it does not take into account that later on you have had more practice, and know what to expect.
In genetics, certain alleles are not always passed on. For example, having parents with ginger hair does not necessarily mean the child will have ginger hair. Obviously, this leads to a good debate!
If it was indeed environmental, I could argue that intelligence is determined in the "sensitive period", outlined by Bowlby, a psychologist. If this is the point where children make the primary attachment, it could be feasible that intelligence is also defined at this point.
An interesting example (to some) is of me and my identical twin brother. It seems to be that he is more academic, but less creative than me. Maybe there is a limited "capacity" in our minds for both intelligence and creativity. Some people may have a smaller capacity (through genetics) than others, but environmental changes affect how we choose to go on, early in life.
It would certainly explain the difference between me and Joe.