The historical symbol of Annecy, its construction extended over four centuries (from the 12th to the 16th). The counts of Geneva made it their home from the 13th century. The town acquired the Castle in 1953 and restored it with help from "Historical Monuments" which made it possible to set up a museum inside.
You can get combo tickets for the Château and Palais de l'Isle (http://musees.agglo-annecy.fr/), honestly inside the castle there is not so much to see. The Tower and Perrière Lodge house the Alpine Lakes Regional Laboratory which is interesting. The city also uses the Castle for a modern art exhibition which can feel strange in the middle of a medieval place.
It is not recommended at all for people who have problems walking or push chairs, it is not far from the old town but really up hill! What I prefer from there is the amazing view but you have to pay the entrance fee to go inside the Castle court.
It could be good to visit the museum on a rainy day except you will miss the view, so can be perfect to hide from the warmth of summer days. From there, if you feel like it, you can also walk up to the Basilique de la Visitation (15 min walk) which also have an amazing view.
The Legend of the Castle...
At the begining of the IX century, the emperor Lothaire (under Charlemagne empire) was living in Annecy Castle for a few years. He was married to a princess called Tiedtberge, their earlier years of their wedding was perfect...but soon he was really interested in the company of the Princess's friends. Tiedtberge was jalous and could not stand her husband's constant infidelities so she was giving him hard time. Lothaire wanted his freedom back, he could have killed her but decided to leave her in jail in the darkness of the fortress (Tour de la Reine). He told others that she was crazy. Tiedtberge was bored and getting sick in this cold, dark and humid place, she would have gone crazy without the help of a chaplain...The chaplain who wanted to reconfort her with the religion and set her free arranged her escape with a ladder. Two religious took her to the Talloires Abbaye, the unfaithful husband was furious but never heard from her again....