A few days before the terrorist attacks on the Boston Marathon, I visited the memorial to the victims of the September 11 attacks at the site once occupied by the World Trade Center in New York. With construction still underway on the five towers of the replacement WTC, the streets are filled with scaffolding and construction workers.

Opposite the main memorial is the fire station with a bronze bas-relief tribute to the 343 firefighters who died in the attacks. A man stood beside the memorial wall, loudly declaiming in a rap-style beat: “Remember! Remember what used to be here! People forget! You just remember what used to be HERE!”

Inside the memorial square, the sites of the two stricken buildings are marked with two large reflective pools that mark the footprints of the stricken buildings. The water spills out of sight “into a void”, an elderly guide explained to his group. He was “a family member or a survivor”, according to the memorial’s description of its tours. Water, he explained, had a cleansing quality according to long tradition both in Western culture and in “Eastern religions” like Buddhism, Shinto and Hinduism.

And Islam, I thought to myself. It wasn’t surprising that Islam didn’t rate a mention, given the identity of the men who had ploughed the hijacked planes into the towers. But their victims had included many Muslims as well, their names among those commemorated in the brass plaques that surrounded the twin fountains. Sarah Khan. Abdu Ali Malahi. Mohammad Salahuddin Chowdury.

These names have not placated those who believed that plans to build an Islamic community centre, dubbed “the Ground Zero mosque”, in the vicinity of the memorial would be an insult to those who had died. To judge by the number of halal food carts and Arabic signage in the area, Muslims are part of the human mosaic of the neighbourhood. But the plans for the complex, initially titled “Cordoba House” and now known simply by its street location, as Park51, were greeted as an arrogant gesture of conquest linked to the carnage that had taken place nearby. Although I had not witnessed any anti-Muslim sentiment at the memorial, I was too conscious of my Muslim identity to ask for directions to Park51, feeling like a German tourist asking the way to the Anne Frank museum. I hedged, asking my way to Park Street without specifying my destination.

“What location on Park Street?” Oh, god. “Number 45?” “What’s at number 45?” Err …

“I think people are starting to forget about us.”

In the gift shop outside the memorial, I scolded both myself for my cowardice and the opponents of Park51 for their bigotry. Why should the presence of a community centre and prayer space a few blocks away be considered transgressive just because, like over a billion people around the globe, we belong to the religious faith bin Laden and his murderous accomplices had claimed to represent? Then I was distracted by the ethical question of whether plush toy dogs wearing 9/11 remembrance T-shirts were a tacky form of commemoration or whether I was guilty of intellectual snobbery.

While the 9/11 memorial had been far more crowded than I’d expected, the Park51 complex was far more deserted. On a weekday afternoon only a handful of men were in its public space. The organisation behind the complex has undergone an internal split, resulting in a loss of key personnel and funds. In the long term, the plan is still to demolish the existing 19th-century building to make way for the larger complex, but to judge by the amount of renovation work underway, this isn’t going to happen any time soon. I asked Ali, a young Egyptian man who had arrived in New York some years after the 2001 attacks, whether Park51 was still attracting hostility.

“Yes — but I think people are starting to forget about us,” he said with a wan smile. Perhaps the internal conflict and slow progress on the complex had made them a smaller target, or perhaps the scaremongers had just moved on to other territory. Pamela Geller, one of the key figures in campaign against the “Ground Zero mosque”, won a court case allowing her to run advertisements in the subway showing the World Trade Center exploding into flames alongside quotes from the Koran. As The Huffington Post reports, after the first batch of advertisements were vandalised, Geller’s organisation responded by doubling the number of ads to run in its next campaign last December.

There were no other women at Park51 during my visit, but the men showed no discomfort at my uncovered hair and showed me their women’s prayer space, pointing out that it was alongside the men’s and that the women had a clear view of the imam. They presented me with several copies of the Koran as well as leaflets about various Islamic community services for Muslims in the neighbourhood. I had regarded the area as a diverse and basically safe space, but remembering Geller’s campaign I discreetly turned the transparent bag containing the gifts from Park51 to conceal its contents before going back out to the street.

I thought of the signs on the lawns at the 9/11 memorial telling visitors “not to walk on the lawns while the grass is taking root”. The grass is still taking root in the ways that we remember September 11. And for some, Muslim feet are regarded as far too disruptive.