Your dress code will depend on your job. There are some general things to keep in mind, though.
In a push to reduce energy costs, and for survival's sake in places without AC, Japanese society is big on Cool Biz. This is a variation of business-wear that features short sleeves and cooler fabrics. It is acceptable in the hotter months of the year, but is put aside for the cooler ones. Clothes are usually clearly marked "Cool Biz" on the tag as a marketing tactic, so it should be easy to spot in the stores. This stuff is pretty much a necessity for people that need to wear nicer clothes even during the summer. You can pick some up at departments stores or places like Aoki and Aoyama.
For most placements, piercings and dyed hair are forbidden. The rules can be a little relaxed for ALTs - especially if they come already having dyed hair and piercings - but that is more of an exception than a rule. Likewise, tattoos should be reasonably concealed and you should avoid getting any new ink that would show at work. To do so may not get you fired, but it would be disrespectful to your coworkers.
On the other hand, as far as you can't see it at work and it doesn't get in the way of work, have at it!
(Keep in mind, though, most onsen and public baths will forbid entrance of people with tattoos.)
Elementary school is usually very relaxed about dress codes. Keep a suit or the equivalent around for ceremonies, and otherwise casual, non-revealing clothes should be fine. They may prefer you to wear long pants, but this isn't always the case. A good track outfit will fit right in at elementary school.
Junior High School
Each JHS is probably a little different, but non-flashy and non-revealing casual clothes are usually okay for the typical work day. JHS has many ceremonies, though, for which you will need a suit and probably some Cool Biz attire. Most teachers will probably dress in business casual without the tie, but quickly change into track uniforms whenever possible. You can follow in their steps if you're worried, but most of us just come to work in casual clothes and stay that way (maybe with a suit in our locker just in case).
High school male ALTs tend to go with business casual minus the tie, opting for Cool Biz when appropriate. This basically means a button down or sweater and dress slacks for me. Most female ALTs seem to get by with a conservative casual.
Many people tend to misjudge dress codes. If you aren't sure what's casual and what's business casual, go for dressier instead of more casual. With that said, ALTs have a lot more liberty with their dress code than other teachers. The best thing you can do to prepare, though, is to ask your predecessor for specifics about what they found to be okay and not-okay at work.
Some people say that Japanese people dress up just to go get groceries, but that's not really the case in Yamanashi. As long as your outfit isn't bordering on illegal (or highly offensive), whatever you usually wear on your own time should be just fine for when you're off the clock. Remember that most of the prefecture has 28 to 38 degree summers and -8 to 10 degree winters (depending on how elevated your area is) - so cool clothes and warm clothes are necessary.
One last thing! Japan loves festivals. They happen all year round! For some festivals people like to wear jinbei (a.k.a. hippari). If someone suggests you wear one, try it out!
Summer festivals are popular places to wear yukata. If you're looking to buy a yukata or want to ask if it's appropriate for an event, as around in the community! Plenty of people should be able to help you out.
Some foreigners worry about how it looks for them to wear traditional clothes, and if it's rude. It's not! Anyone who notices will probably just think it's cute, and cool that foreigners are interested in Japanese culture (other than Tokyo and izakaya, of course).