"Srinagar is a great idea but it would have been better if there wasn't an election looming in the state," Tharoor tweeted.
Assembly elections are due in Jammu and Kashmir this year-end.
"Will be in Srinagar on Diwali, 23rd October & will spend the day with our sisters & brothers affected by the unfortunate flood," Modi tweeted.
Congress had on October 13 removed Tharoor as the party's spokesperson after the party's Kerala unit lodged a complaint against him for praising Modi.
The Kerala PCC leadership had demanded "appropriate action" against the Lok Sabha MP from Thiruvananthapuram for his "pro-Modi stands", including unilaterally accepting the PM's invitation to be a brand ambassador of Swachh Bharat campaign.
The report of the Kerala unit had held that Tharoor's adulatory statements about Modi had hurt Congress workers in Kerala, who had worked sincerely and tirelessly for his victory from Thiruvananthapuram Lok Sabha seat.